OpenZFS on root for Slackware
Introduction
First off, I want to thank all of the dedicated Slackware hackers that have helped me along the way. Since I can't remember exactly where or when any particular person has helped me over a hump with their on-line snippet of wisdom, I will have to just say thanks to you all.
This is a hands-on, messy way of getting the job done. But, since my goal was to experiment and learn, I hope you can find something useful in the following.
Some background
I started out with Slackware on an unused older Pentium III back in 2000 and something. The goal was to create a server/router/gateway for my home network. I also wanted to play with a web server, mail server, etc. It was all just a hobby. Over time and with each upgrade of Slackware, I added to the configuration. I added a software mirrored raid, experimented with file systems, etc. So, having a raid1 root file system with reiserFS, gave me a taste of initrd and booting troubles. More than once after a hard drive crash or a complacent kernel update, I found myself in rescue mode after booting from the Slackware CD trying to reassemble my raid.
Also, before each Slackware Upgrade to the new version, I usually took the opportunity to upgrade the hardware. This way, I had a non production box to play around with before placing it in the server room for everyone to use.
And, since Slackware 14.2 was getting a little dated—and I was getting tired of custom compiling all of the latest versions of the software—I figured it was time to do another hardware upgrade and move on to the next version of Slackware. I usually never mess with slackware-current. My server needs to be stable. Even though I like to experiment, it is a working machine that everyone in the house uses and needs everyday.
But, the time seemed right to get started. It was winter and Slackware 15.0 was almost ready, right?—so I figured I had time to play. And, true to my "let us play around with the hard stuff" philosophy, I started reading about BtrFS. But, the more I read, the more I started to get a little undecided. Then, I found some articles on ZFS and was immediately drawn to it. I can't exactly put my finger on it, but I decided ZFS was what I was going to use for my new server root file system.
Then reality set in! ZFS isn't in the kernel tree! But, since I like to experiment with things like this—and I felt like I had some time to do so—I decided to go ahead and take the plunge.
For the next undetermined amount of time, I searched and read. And read and searched, and searched and searched, and read some more. Finally, after getting as much background information as I could, I downloaded a current slackware iso sometime in February 2021, and got to work. The unfortunate part of this is that I didn't document the process very well.
Way back in the beginning, when I started configuring my server, I had wanted to keep a log, so that I could remember what I did or how I configured something. I have a lot of custom slackbuild scripts that I have used in the past. But, the step-by-step documentation just wasn't there. I try to add a lot of comments to my scripts, but after a while it gets hard to replicate exactly what you did without step-by-step procedure.
Since I never seem to throw anything away—I have a large bash history—the following is what I have pieced together to document the process I used.
-
install slackware-current. This is the stage that you wish you had a way to have all dependent packages install for say "development". I am such a minimalist. I hate waste. But, it is just easier sometimes to install everything. (Just to let the reader know that, since I was planning on using this install for a rescue environment, I didn't install everything. I wanted to make it fit in an 8GB partition.)
-
download the zfs-on-linux package from slackbuilds.org. All I did to make this slackbuild work for version 2.1.1 is change the version in the slackbuild script and get the source tarball from the OpenZFS github releases page. See End Notes (The final install package is huge! So I did make some modifications to the script to move the "testing" stuff to its own installer package.)
-
install the modules into the kernel. (and initrd) This is where I started tweaking the initrd stuff. I first added zfs to the module list and rebooted. I wanted to make sure the modules would load at boot before they were required. (Later I continued to tweak the initrd tree)
-
create the zfs pool(s) and datasets. Now came the hard part for me. I had to decide how I was going to actually use zfs in my new system. I definitely wanted to leverage the snapshot feature. So, I planned out how I was going to set up my pools. I don't plan on exporting and importing my pools between multiple servers. But, you never know. So, I decided to prefix my pools with my hostname.
Also, deciding on options proved to take some time researching as well. I chose the following scheme based on much snooping around on the internet:
Create unimatrix0-zroot zpool
# zpool create -o ashift=13 -O acltype=posixacl -O canmount=off \
-O compression=zstd -O dnodesize=auto -O normalization=formD \
-O relatime=on -O xattr=sa -m none unimatrix0-zroot /dev/sda6
# zfs create -o mountpoint=none unimatrix0-zroot/ROOT
# zfs create -o mountpoint=/ -o canmount=noauto unimatrix0-zroot/ROOT/slackware.current
I used a little shell script to create some of my datasets. Not sure I got it right. As I was looking through the history file, I notice a lot of manual tweaking. Anyway, adjust to your liking.
#!/bin/bash
MAIN_ZPOOL_DATASET=unimatrix0-zpool/ROOT/slackware.current
for i in {usr,var,var/lib};
do
zfs create -o canmount=off $MAIN_ZPOOL_DATASET/$i
done
for i in {home,opt,root,usr/local,usr/src,var/cache,var/log, \
var/spool,var/www,var/lib/clamav,var/lib/mysql,var/lib/pgsql};
do
zfs create -o canmount=on $MAIN_ZPOOL_DATASET/$i
done
Here are a couple of the options I changed on the above datasets. I wanted to set the recordsize for the MySQL database on the /var/lib/mysql dataset to the recommended 16K. And, I changed the compression to zstd-10 on the /usr/src mount. Here are the commands to do that:
# zfs set recordsize=16K unimatrix0-zpool/ROOT/slackware.current/var/lib/mysql
# zfs set compression=zstd-10 unimatrix0-zpool/ROOT/slackware.current/usr/src
My final setup looks like this after tweaking:
We are not finished yet. I just put this here to give an idea of what a bunch of datasets look like with inheritance and different options. Make sure you get this right. It is hard to fix later after you install Slackware. 1
NAME RECSIZE MOUNTPOINT MOUNTED RDONLY CANMOUNT USED RATIO COMPRESS
unimatrix0-zpool 128K none no on off 41.0G 1.82x zstd
unimatrix0-zpool/ROOT 128K none no off off 27.4G 2.01x zstd
unimatrix0-zpool/ROOT/slackware.current 128K / yes off noauto 27.4G 2.01x zstd
unimatrix0-zpool/ROOT/slackware.current/home 128K /home yes off on 3.39G 1.37x zstd
unimatrix0-zpool/ROOT/slackware.current/opt 128K /opt yes off on 448K 1.00x zstd
unimatrix0-zpool/ROOT/slackware.current/root 128K /root yes off on 988M 1.38x zstd
unimatrix0-zpool/ROOT/slackware.current/usr 128K /usr no off off 1.95G 2.00x zstd
unimatrix0-zpool/ROOT/slackware.current/usr/local 128K /usr/local yes off on 436M 1.06x zstd
unimatrix0-zpool/ROOT/slackware.current/usr/src 128K /usr/src yes off on 1.52G 2.45x zstd-10
unimatrix0-zpool/ROOT/slackware.current/var 128K /var no off off 3.82G 1.63x zstd
unimatrix0-zpool/ROOT/slackware.current/var/cache 128K /var/cache yes off on 438M 1.21x zstd
unimatrix0-zpool/ROOT/slackware.current/var/lib 128K /var/lib no off off 344M 1.88x zstd
unimatrix0-zpool/ROOT/slackware.current/var/lib/clamav 128K /var/lib/clamav yes off on 416K 1.00x zstd
unimatrix0-zpool/ROOT/slackware.current/var/lib/mysql 16K /var/lib/mysql yes off on 334M 1.82x zstd
unimatrix0-zpool/ROOT/slackware.current/var/lib/pgsql 128K /var/lib/pgsql yes off on 8.68M 4.35x zstd
unimatrix0-zpool/ROOT/slackware.current/var/log 128K /var/log yes off on 122M 4.66x zstd
unimatrix0-zpool/ROOT/slackware.current/var/spool 128K /var/spool yes off on 867M 1.44x zstd
unimatrix0-zpool/ROOT/slackware.current/var/www 128K /var/www yes off on 2.09G 1.60x zstd
Prepare the system for booting into zfs-root
Okay. Now you have a bunch of datasets in the root pool without any data.
What I chose to do is mount the zfs pool under /mnt_zfs of the current booted rescue system, then rsync a copy to the mounted zpool. I achieved this by setting the mount property of the zpool before mounting it.
# zfs set mountpoint=/mnt_zfs unimatrix0-zpool/ROOT/slackware.current
# zfs mount unimatrix0-zpool/ROOT/slackware.current
# rsync -avxHAXPS / /mnt_zfs
Then I modified the /boot/initrd-tree-zfs/init to mount zfs. Since I didn't use the legacy mount options, I added a couple of zfs commands. The following diff is how I modified the mount command in the init file from initrd-tree-zfs:
190a191,199
> # setup ZFS here
> if [ "${ROOTFS}" == "zfs" ] # ZFS on root
> then # ZFS on root
> /sbin/zpool import -N -d /dev/disk/by-id ${ROOTDEV%%/*} # ZFS on root
> /sbin/zfs set mountpoint=/ ${ROOTDEV} # ZFS on root, jsr added
> /sbin/zfs set readonly=on ${ROOTDEV%%/${ROOTDEV##*/}} # ZFS on root, jsr added
> mount -t $ROOTFS $ROOTDEV /mnt # ZFS on root, jsr modified
> fi # ZFS on root
>
325c334,337
< mount -o ro${ROOTFLAGS:+,$ROOTFLAGS} -t $ROOTFS $ROOTDEV /mnt
---
> if [ "${ROOTFS}" != "zfs" ] # ZFS on root
> then # ZFS on root
> mount -o ro${ROOTFLAGS:+,$ROOTFLAGS} -t $ROOTFS $ROOTDEV /mnt # ZFS on root
> fi # ZFS on root
Setup a mirror for the root zpool
Also, since I first created this pool without a mirror, I decided to add a partition as a mirror to this zpool like so:
# zpool attach unimatrix0-zpool /dev/sda6 /dev/sdb3
Since I executed this command after I rsync'd the data, the new device immediately began to resilver.
I should mention that before I started the first slackware install, I partitioned my two hard drives with mirroring in mind. So, the partitions I wanted to mirror were the same size.
As a side note, you will notice that I am using zfs in partitions instead of full disks. This appears to deviate from the "zfs way". But, I chose to use partitions instead of full disks so that I have more flexibility. Namely, so that I can have my rescue partition and EFI partition on the same disk as my zfs root. Also, I could play around with more than one data pool without using extra disks.
Rebooting into the new ZFS root file system
First you need to prepare your initrd for booting with the zfs kernel module. Here are a few of the things I did.
- mkinird.conf
Here are the relevant variables I modifed in the mkinitrd.conf file before running the mkinitrd command.
SOURCE_TREE="/boot/initrd-tree-zfs"
OUTPUT_IMAGE="/boot/initrd-zfs.gz"
MODULE_LIST="sym53c8xx:ext4:e1000:zfs:9p:9pnet_virtio:fat:vfat"
ROOTDEV="unimatrix0-zpool/ROOT/slackware.current"
ROOTFS="zfs"
- Then I prepared the initrd.gz file. First compile the latest OpenZFS if not done already and install it in the local system and the initrd folder like so:
#!/bin/bash
set -e
ZFS_VERSION=2.1.1
NEW_KERN=${NEW_KERN:-5.14.11}
cd slackbuilds/zfs-on-linux/
KERN=$NEW_KERN zfs-on-linux.SlackBuild
installpkg /tmp/zfs-on-linux-${ZFS_VERSION}_$NEW_KERN-x86_64-1_jsr.txz
depmod $NEW_KERN
ROOT=/boot/initrd-tree-zfs/ upgradepkg /tmp/zfs-on-linux/zfs-on-linux-${ZFS_VERSION}_$NEW_KERN-x86_64-1_jsr.txz
mkinitrd -k $NEW_KERN -F
cp /boot/initrd-zfs.gz /boot/efi/EFI/Slackware/initrd-zfs.gz
cp /boot/vmlinuz-generic-$NEW_KERN /boot/efi/EFI/Slackware/vmlinuz
Then, if you are brave enough, reboot and see what happens. You still do have that rescue install somewhere on the hard disk right?
Also, be aware that, if you change zfs versions and upgrade your pools to use the new features, make sure you update to the same newer version on your rescue partition too! Or, you may find that you can't mount the zfs file system in rescue mode.
Create unimatrix0-zdata zpool
I created a mirrored data pool from the start here.
# zpool create -o ashift=13 -O acltype=posixacl -O canmount=off \
-O compression=zstd -O dnodesize=auto -O normalization=formD \
-O relatime=on -O xattr=sa -m none \
unimatrix0-zdata /dev/sda4 /dev/sdb5
#
# zfs create -o canmount=off -o mountpoint=none unimatrix0-zdata/DATA
# zfs create -o mountpoint=/data -o canmount=on unimatrix0-zdata/DATA/data
# zfs create -o mountpoint=/data/books -o canmount=on -o recordsize=512K unimatrix0-zdata/DATA/data/books
# zfs inherit -r mountpoint unimatrix0-zdata/DATA/data/books
# zfs create unimatrix0-zdata/DATA/data/elasticsearch
# zfs create unimatrix0-zdata/DATA/data/embyserver
# zfs create unimatrix0-zdata/DATA/data/homeassistant
# zfs create unimatrix0-zdata/DATA/data/nextcloud
# zfs create unimatrix0-zdata/DATA/data/portainer
# zfs create -o recordsize=512K -o compression=zstd-10 unimatrix0-zdata/DATA/data/samba
# zfs create -o mountpoint=none -o canmount=on unimatrix0-zdata/DATA/docker
# zfs create -o mountpoint=/var/lib/docker -o canmount=on unimatrix0-zdata/DATA/docker/docker
Customize to your heart's content. Unfortunately I found it hard to get this stuff right the first time. And as time has passed, I find myself adding or tweaking recordsize, compression, or atime settings.
I put the zfs inherit command in above, because—as I was looking through my bash history—I did specify a mountpoint sometimes when I probably shouldn't have. When you do this, the dataset doesn't get inherited from the /data mountpoint. This probably doesn't matter on datasets that are mounted after the root file system is mounted and running. But, it seems like a good idea for subdirectories that are actually inherited to have the proper mount inheritance.
NAME RECSIZE MOUNTPOINT MOUNTED RDONLY CANMOUNT USED RATIO COMPRESS
unimatrix0-zdata 128K none no off off 263G 1.06x zstd
unimatrix0-zdata/DATA 128K none no off off 253G 1.06x zstd
unimatrix0-zdata/DATA/data 128K /data yes off on 43.5G 1.27x zstd
unimatrix0-zdata/DATA/data/books 512K /data/books yes off on 309M 1.16x zstd
unimatrix0-zdata/DATA/data/elasticsearch 128K /data/elasticsearch yes off on 2.14G 1.09x zstd
unimatrix0-zdata/DATA/data/embyserver 128K /data/embyserver yes off on 358M 1.58x zstd
unimatrix0-zdata/DATA/data/homeassistant 128K /data/homeassistant yes off on 21.6M 3.79x zstd
unimatrix0-zdata/DATA/data/nextcloud 128K /data/nextcloud yes off on 8.08G 1.27x zstd
unimatrix0-zdata/DATA/data/portainer 128K /data/portainer yes off on 2.81M 2.68x zstd
unimatrix0-zdata/DATA/data/samba 512K /data/samba yes off on 24.3G 1.12x zstd-10
unimatrix0-zdata/DATA/docker 128K none no off on 4.77G 2.17x zstd
unimatrix0-zdata/DATA/docker/docker 128K /var/lib/docker yes off on 4.77G 2.17x zstd
You will notice that I have not been using the legacy mount option. Instead I use /etc/rc.d/rc.zfs and have modified /etc/rc.d/rc.6 and /etc/rc.d/rc.S to mount and unmount zfs files systems correctly. I also modified the /etc/rc.d/rc.zfs init script so that stop does not unmount anything. Hence no need for restart and condrestart. The "unmount all zfs file systems" just looks too scary to me. The init scripts seem to unmount everything fine at shutdown.
Here are the patch files for that:
rc.zfs.patch
81a82
> echo "This init script Does No Unmounting . . ."
83,87d83
<
< echo "Unmounting ZFS filesystems"
< "$ZFS" umount -a
<
< rm -f "$LOCKFILE"
110,119d105
< restart)
< stop
< start
< ;;
< condrestart)
< if [ -f "$LOCKFILE" ]; then
< stop
< start
< fi
< ;;
121c107
< echo $"Usage: $0 {start|stop|status|restart|condrestart}"
---
> echo $"Usage: $0 {start|stop|status}"
rc.S.patch
279c279,286
< /sbin/mount -w -v -n -o remount /
---
> ZFSROOT=$(/sbin/mount | grep "on / type zfs" | cut -d' ' -f1)
> if [ -z "$ZFSROOT" ]; then
> /sbin/mount -w -v -n -o remount /
> else
> echo "----Looks like zfs is mounted on root!----"
> /sbin/zfs set readonly=off ${ZFSROOT%%/${ZFSROOT##*/}}
> # the rest of the datasets are mounted below
> fi
365a373,379
> fi
>
> # zfs
> rm -f /var/lock/zfs/zfs
> echo "Mounting zfs datasets now:"
> if [ -x /etc/rc.d/rc.zfs ]; then
> /etc/rc.d/rc.zfs start
rc.6.patch
249c249,255
< /bin/mount -v -n -o remount,ro /
---
> ZFSROOT=$(/sbin/mount | grep "on / type zfs" | cut -d' ' -f1)
> if [ -z "$ZFSROOT" ]; then
> /bin/mount -v -n -o remount,ro /
> else
> echo "----Setting zfs ROOT to readonly----"
> /sbin/zfs set readonly=on ${ZFSROOT%%/${ZFSROOT##*/}}
> fi
End Notes
Also, since the Python 3.10 update in slackware-current, zfs-on-linux didn't want to compile. This is another reason being on the bleeding edge can sometimes come back and bite you. Thankfully, I didn't have to enter my rescue system to fix this. (If I would have rebooted before finding a way to recompile the zfs kernel modules, I would have had a bad day.)
From snooping around on the OpenZFS github page, I found patch #12073 that had some merit. So, I applied this patch to the current zfs-2.1.1.tar.gz, and had to rerun automake. Here is what I added to my slackbuild script:
patch -p1 < $CWD/12073.diff
# added this because of the python 3.10 >= 3.4 is false !!!
aclocal
automake --add-missing
autoconf -f
Conclusion
I hope the process documented here will help someone. If you have any question, please let me know and I will try and answer them as best I can. Also, since this is my first time playing around with ZFS, I would appreciate any comments and feedback on where I could improve.
Thanks,
Jeff
1 It is possible to tweak a few things later if you didn't get it right the first time. Say you want to change the recordsize for the /opt dataset.
- You have to move all of the data to a different zpool like:
mv /opt/* unimatrix0-zdata/DATA/data/tmp/opt
- Make the change you want to the dataset
zfs set recordsize=1M compression=zstd-10 unimatrix0-zpool/ROOT/slackware.current/opt
- move all the data back to /opt.
mv unimatrix0-zdata/DATA/data/tmp/opt/* /opt
- I have also had some issues when creating datasets. Sometimes when you check mountpoints with:
zfs get mountpoint unimatrix0-zpool/ROOT/slackware.current/opt
The mountpoint isn't inherited but "local". If instead you want to have it inherit from the root "unimatrix0-zpool/ROOT/slackware.current", use the following to fix it:
zfs inherit -r mountpoint unimatrix0-zpool/ROOT/slackware.current/opt
Add comment