Freebsd-zfs-on-root
From FoppaWiki
Yes it is possible to very easily install FreeBSD on zfs, actually it's the easiest install of freebsd I've done. Running FreeBSD on zfs alone can be achieved in several ways, but Martin Matuška made an iso ready to boot from, and by using a zfsinstall script it only takes a matter of minutes to get it up and running. My task was to maximize the use of my harddisks on a FreeBSD installed server, and get to learn abit about the zfs file system. Specifically I had a server with 6 x 700mb sata disk I needed for backup server and installing FreeBSD on a raidz would give great use of the harddisks and give me abit of fault tolerance. What follows is my experience completing that, and the examples will only show 4 harddisks, but thats because I was testing and making the wiki on vmware setup.
Contents |
Getting the image
Download the special edition at http://mfsbsd.vx.sk/ - the 8.1-RELEASE has a bug which prevents FreeBSD to boot on anything else than the first harddisk on this setup, which kinda defeats the purpose here, because systems breaks if the first harddisk breaks. There is a patch to correct this, but everything worked perfectly when I tried the 8.2-RC1-amd64 special edition
Installing FreeBSD-zfs-on-root
After booting from the image, it actually tells you what to do. First you mount the cdrom:
mount_cd9660 /dev/acd0 /cdrom
Then we proceed to use the zfsinstall script. The example in the motd and on his homepage is for a single disk setup. You need a little extra when you go mirror or raidz, and mine would be:
zfsinstall -d da0 -d da1 -d da2 -d da3 -t /cdrom/8.2-RC1-amd64.tar.xz -r raidz -p tank -s 2G
Last thing to do is setting a root password, and make some very basic additions to /etc/rc.conf so we can reboot to our new system:
chroot /mnt echo 'hostname="zfsbsd"' >> /etc/rc.conf echo 'keymap="danish.iso"' >> /etc/rc.conf echo 'ifconfig_em0="DHCP"' >> /etc/rc.conf echo 'sshd_enable="YES"' >> /etc/rc.conf reboot -r now
Checking installation
After a successful boot, we can check how everything looks:
zfsbsd# df -h Filesystem Size Used Avail Capacity Mounted on tank/root 23G 331M 23G 1% / devfs 1.0K 1.0K 0B 100% /dev tank/root/tmp 23G 38K 23G 0% /tmp tank/root/var 23G 238K 23G 0% /var
zfsbsd# zpool status
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0p3 ONLINE 0 0 0
da1p3 ONLINE 0 0 0
da2p3 ONLINE 0 0 0
da3p3 ONLINE 0 0 0
errors: No known data errors
zfsbsd# gpart show
=> 34 20971453 da0 GPT (10G)
34 128 1 freebsd-boot (64K)
162 4194304 2 freebsd-swap (2.0G)
4194466 16777021 3 freebsd-zfs (8.0G)
=> 34 20971453 da1 GPT (10G)
34 128 1 freebsd-boot (64K)
162 4194304 2 freebsd-swap (2.0G)
4194466 16777021 3 freebsd-zfs (8.0G)
=> 34 20971453 da2 GPT (10G)
34 128 1 freebsd-boot (64K)
162 4194304 2 freebsd-swap (2.0G)
4194466 16777021 3 freebsd-zfs (8.0G)
=> 34 20971453 da3 GPT (10G)
34 128 1 freebsd-boot (64K)
162 4194304 2 freebsd-swap (2.0G)
4194466 16777021 3 freebsd-zfs (8.0G)
Replacing a disk
Ok the next question was, what will all this be good for if a harddisk breaks! As it turns out, its no big deal! As we can see above all the harddisks have the same layout with swap and boot next to the zfs slice, which means we can replace any of them. Basically we just have replace the broken disk and then partition the disk like the others.
note: rebooting the machine at this stage should not be a problem either, but device names can alter if a disk is missing, and that might make the identification of the disks a little harder, but not impossible
So based on the numbers above and assuming our disk got da4 we create a new gpt and slice it up. Replace da4 with whatever device name your new disk gets:
gpart create -s GPT /dev/da4 gpart add -b 34 -s 128 -t freebsd-boot -l da4-boot /dev/da4 gpart add -b 162 -s 4194304 -t freebsd-swap -l da4-swap /dev/da4 gpart add -b 4194466 -s 16777021 -t freebsd-zfs -l da4-data /dev/da4
Now if it was da2 which broke, we replace it in the zpool and wait for a resilver:
zpool replace tank da2p3 da4p3
If the new harddisk got the same device name it would just be:
zpool replace tank da2p3
We must be able to make the new disk bootable also, the zpool replace command reminded me of this:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da4
Other
Best practice for replacing disk on FreeBSD:
1. zpool offline adX
2. atacontrol list (to find the ataX device number)
3. atacontrol detach ataX
4. dmesg (verify the detach worked)
5. Physically remove the disk (must be in a hot-swap enclosure)
6. Physically insert the new disk
7. atacontrol attach ataX
8. dmesg (to determine what the adX drive number is; on my systems
the adX drive number remains static/does not change)
9. zpool online pool adX
10. zpool replace pool adX
11. zpool status (watch until finished)
This guy did not use the whole disk for root raid : http://microsux.dk/?p=54
