CentOS 7 and ZFS
I have used ZFS before on FreeBSD but never on any Linux variant.
So it became time to play around with ZFS on Linux, but as you can read below, there still are some mayor issues.
As I just downloaded the latest CentOS 7 Minimal CD I used CentOS as my first try.
After the minimal install and updating all packages on my little VM we were ready to go.
ZFS is not part of CentOS so you will need to enable extra repository’s to install ZFS.
First we enable EPEL and add the zfs repo:
1 2 | sudo yum install epel-release sudo yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm |
Then install ZFS:
1 | sudo install kernel-devel zfs |
This is where the first issue came up, ZFS did not seem to have installed correctly giving me not much more than an:
1 | Failed to load ZFS module stack. |
It seemed that ZFS was not build correctly because of the kernel update that was installed when I updated the system for the first time. After numerous attempts to fix ZFS I decided to just remove all the ZFS packages and old kernel and start over.
An tip online on gmane also mentioned to remove all modules, so sure lets go:
1 2 3 | find /lib/modules/$(uname -r)/extra -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f find /lib/modules/$(uname -r)/weak-updates/ -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f |
Then I removed my old kernel, headers and all ZFS packages and finally an reboot:
1 2 3 4 | sudo yum erase kernel-3.10.0-229.el7.x86_64 sudo yum erase kernel-headers kernel-devel kernel-tools sudo yum erase zfs zfs-dkms libzfs2 spl spl-dkms dkmsc reboot |
After the reboot it was time to install everything again:
1 2 3 | sudo yum localinstall --nogpgcheck https://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm sudo yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm sudo yum install kernel-devel zfs |
And finally ZFS worked like it should, no more error messages.
I have 4 small 5GB drives added to my VM to test with,
sdb, sdc, sdd and sde
So lets create an pool called data01 in raidz, as the drives are blank and unformatted you will need to force the creation of an new GPT.
1 | zpool create data01 raidz sdb sdc sdd sde -f |
See if all went well:
1 2 3 4 5 6 7 8 9 10 11 12 13 | zpool status pool: data01 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM data01 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 |
That looks fine, now lets see what zdb says:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | zdb
data01:
version: 5000
name: 'data01'
state: 0
txg: 141
pool_guid: 11371218379924896498
errata: 0
hostname: 'gluster01'
vdev_children: 2
vdev_tree:
type: 'root'
id: 0
guid: 11371218379924896498
create_txg: 4
children[0]:
type: 'raidz'
id: 0
guid: 832996839702454674
nparity: 1
metaslab_array: 34
metaslab_shift: 27
ashift: 9
asize: 16060514304
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 2105630605786225987
path: '/dev/sdb1'
whole_disk: 1
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 6145009721102371365
path: '/dev/sdc1'
whole_disk: 1
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 14635352678309772165
path: '/dev/sdd1'
whole_disk: 1
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 1544288362510023523
path: '/dev/sde1'
whole_disk: 1
metaslab_array: 112
metaslab_shift: 25
ashift: 9
asize: 5353504768
is_log: 0
create_txg: 134
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data |
That all looks fine so we should have an 15GB /data01 partition:
1 2 3 4 5 6 7 8 9 | df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 8.5G 1.1G 7.5G 13% / devtmpfs 481M 0 481M 0% /dev tmpfs 490M 0 490M 0% /dev/shm tmpfs 490M 6.7M 484M 2% /run tmpfs 490M 0 490M 0% /sys/fs/cgroup /dev/sda1 497M 119M 378M 24% /boot data01 15G 0 15G 0% /data01 |
Perfect, that all works fine, lets see if duplication works:
I will create a 1GB file just filled with zero’s to see if this works.
1 2 3 4 5 6 7 8 9 10 11 | dd if=/dev/zero of=/data01/zerofile1 bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.45455 s, 241 MB/s ls -lh /data01/ total 1.0G -rw-r--r--. 1 root root 1.0G Jul 28 22:42 zerofile1 df -h | grep data01 data01 15G 1.0G 14G 7% /data01 |
Okay we have a single file on /data01 that uses 1GB of space, lets copy the file a few times and see what happens, if everything works like it should the usage of the drive should not increase:
1 2 3 4 5 6 7 8 9 10 11 12 | i=2; while [ $i -lt 7 ]; do cp /data01/zerofile1 /data01/zerofile$i; i=$[i+1]; done ls -lh /data01/ total 1.0G -rw-r--r--. 1 root root 1.0G Jul 28 22:42 zerofile1 -rw-r--r--. 1 root root 1.0G Jul 28 22:53 zerofile2 -rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile3 -rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile4 -rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile5 -rw-r--r--. 1 root root 1.0G Jul 28 22:54 zerofile6 df -h | grep data01 data01 15G 1.0G 14G 7% /data01 |
Yep we have 6 of the same files that are each 1GB in size but still 1GB of total space used, looks good thus far, lets clean up and the drive should be empty again:
1 2 3 4 5 | rm -rf zerofile* ls -lh /data01/ total 0 df -h | grep data01 data01 15G 1.0G 14G 7% /data01 |
Thats weird, I removed the files but still the drive has 1GB of space in use.
But if you unmount the drive and mount the drive again the disk space is free again:
1 2 3 4 5 | cd / umount /data01/ zfs mount data01 df -h | grep data data01 15G 0 15G 0% /data01 |
This seems to be an bug, creating an sub drive with xattr=sa and modify the drop_caches seems to ‘fix’ the issue
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | zfs create -o xattr=sa data01/fs echo 3 > /proc/sys/vm/drop_caches dd if=/dev/zero of=/data01/fs/testfile bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 1.98644 s, 541 MB/s # df -h | grep data data01/fs 15G 1.0G 14G 7% /data01/fs # rm /data01/fs/testfile rm: remove regular file ‘/data01/fs/testfile’? y # df -h | grep data data01/fs 15G 128K 15G 1% /data01/fs |
So it seems ZFS on CentOS 7 still has some issues, next up in test is Debian.