After being bit by FreeBSD loader half a decade ago, I started preferring GNU GRUB as my bootloader for booting all the operating systems, I ever wrote or decided to try out. Recently I started using ZFS extensively, initially for just my /home, and later for / (root). It is great except for one issue that I needed a UFS2 as / volume for booting. Not a big deal, just created a tiny UFS2 partition and got started…

This worked great for me until yesterday, when due to abnormal power cut which my box experienced, and caused my UFS2 / to literally wipe off after I fscked it :(. Fortunately there wasn’t much stuff on it, except /etc, /boot, /lib{,exec}, and /{s,}bin just enough to boostrap my FreeBSD box and mount my ZFS root which has /usr, /var, /tmp etc. volumes. I was able to reinstall my root, and got back to old setup in just an hour. But to avoid this for future, I decided to give a shot to ZFS-only setup again. The normal way to have a ZFS only system requires having gptzfsboot (for GPT drives, ofcourse) as boot loader, zfsloader as loader (which does some magic so you can set vfs.root.mountfrom as a ZFS volume without getting any root device not found error), but since I use GRUB2, I can’t use gptzfsboot, but I can use zfsloader although that requires a bit of configuration, like creating a loader.conf(5) etc. Anyways, I decided to go zfsloader way, and started /boot/zfsloader from GRUB2 command-line:

grub> kfreebsd (hd0,7)/boot/zfsloader

This presented me with a loader menu, so instead of continuing with loader, I came back to loader prompt. Then I did lsdev and it displayed ZFS volumes. And then I did lsmod to verify if it added zfs.ko, which it didn’t, but I found something else there. A module entry for /boot/zfs/zpool.cache, which revealed the whole magic zfsloader was doing.

So I rebooted and decided to replicate this with GRUB2 taking inspiration from a similar situation I was in few months ago when I was trying to boot dragonflybsd kernel to boot from encrypted HAMMER volume, and needed an initrd to mount root from. Following is the final GRUB2 configuration which worked for me, with changes in bold:

menuentry "FreeBSD 9.0-RC3" {
        set root=(hd0,7)
        kfreebsd /boot/kernel/kernel
        kfreebsd_loadenv /boot/device.hints
        kfreebsd_module /boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
        kfreebsd_module_elf /boot/kernel/opensolaris.ko
        kfreebsd_module_elf /boot/kernel/zfs.ko
        kfreebsd_module_elf /boot/kernel/coretemp.ko
        kfreebsd_module_elf /boot/kernel/geom_journal.ko
        kfreebsd_module_elf /boot/kernel/sound.ko
        kfreebsd_module_elf /boot/kernel/snd_hda.ko
        kfreebsd_module_elf /boot/kernel/uhid.ko
        kfreebsd_module_elf /boot/kernel/ukbd.ko
        kfreebsd_module_elf /boot/kernel/ums.ko
        kfreebsd_module_elf /boot/modules/nvidia.ko
#       set kFreeBSD.vfs.root.mountfrom=ufs:ad14p7
        set kFreeBSD.vfs.root.mountfrom=zfs:zroot
	set kFreeBSD.vfs.zfs.prefetch_disable=1
	set kFreeBSD.vfs.zfs.txg.timeout=5
	set kFreeBSD.maxvnodes=250000
	set kFreeBSD.zfs.txg.write_limit_override=805306368
}

In the above configuration (hd0,7) is my UFS2 partition which hosts /boot, which I’ll soon rm -rf once I upgrade my GRUB to 1.99 (which has support for ZFS), ufs:ad14p7 was my old UFS2 root, and zroot is the my ZFS volume I’m using as /.

chateau.d.if!abbe:~ % mount |grep -v zfs
devfs on /dev (devfs, local, multilabel)
devfs on /var/named/dev (devfs, local, multilabel)
/dev/ad14p7 on /boot (ufs, local, read-only)
chateau.d.if!abbe:~ % zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
home                        147G   538G  79.6G  /home
home/abbe                  65.3G   538G  65.3G  /home/abbe
home/tscache                  2G   540G    16K  -
zroot                      18.0G   178G   351M  legacy
zroot/tmp                   124K   178G   124K  /tmp
zroot/usr                  15.7G   178G  6.66G  /usr
zroot/usr/ports            8.67G   178G   244M  /usr/ports
zroot/usr/ports/distfiles  6.55G   178G  6.55G  /usr/ports/distfiles
zroot/usr/ports/packages   1.88G   178G  1.88G  /usr/ports/packages
zroot/usr/src               351M   178G   351M  /usr/src
zroot/var                  1.60G   178G   502M  /var
zroot/var/crash            31.5K   178G  31.5K  /var/crash
zroot/var/db                895M   178G   846M  /var/db
zroot/var/db/pkg           49.6M   178G  49.6M  /var/db/pkg
zroot/var/empty              31K   178G    31K  /var/empty
zroot/var/log              11.2M   178G  11.2M  /var/log
zroot/var/mail               54K   178G    54K  /var/mail
zroot/var/run              89.5K   178G  89.5K  /var/run
zroot/var/squid             231M  11.8G   231M  /var/squid
zroot/var/tmp               663K   178G   663K  /var/tmp

I hope it doesn’t happen again. Fingers crossed….;)