ZFS on Linux on Tomato

Discussion in 'Tomato Firmware' started by lancethepants, May 18, 2019.

  1. lancethepants

    lancethepants Network Guru Member

    So this is one of those things I do for fun and academic reasons, not because it's practical.

    Was playing around and managed to get zfs to compile for Tomato ARM. I use zfs other places, and wanted to see if a router could run it. Turns out it can, and it's not quite as terrible as you would think. Actually a bit better than I would have thought. I tested on my R7000 with 4tb hdd & usb3 over samba and got...
    write: 6.5MB/s - 10MB/s
    read: 23MB/s

    Not spectacular numbers, but then again I had no idea what to expect, or whether it would even run.
    I haven't tested too much, but it feels like ext4 isn't that much faster than this, at least that how it feels. Maybe someone out there will give this a whirl and come up with some more solid numbers. Makes me want to overclock my R7000 and see how much it can handle. Would be cool to see tomato run on beefier devices too like ASUS RT-AX88U.

    I plugged in an external drive I have that already uses zfs for backups, and I was surprised that it mounted without issue. Through samba I was able to view my photos and they loaded very quickly, even raws. I loaded a mkv movie on, and was able to stream it to my pc.

    It would be interesting to experiment more with & without checksums & encryption, or different types of checksums and encryptions, and see what performance hits/benefits you get.

    Speaking of encryption, that is a nifty feature with this release. You can encrypt the whole pool, or just an individual file system natively with zfs. It is also really nice to create and destroy file systems on the fly. You don't have to worry about pre-partitioning your disk. Need /mmc, or /opt, or something under /mnt? Just create and mount it.

    If you use upx to compress the zfs & zpool binaries (about the only binaries you'll need) and put them with the kernel modules, they take up about ~6MBs space, easily fitting into ARM jffs. If you want the full suite of things, you can extract to /opt. I compiled for that prefix.

    I'm not going to get into too much detail of how to try this, there's plenty of zfs howtos already.

    Load your kernel modules in this order.
    insmod spl.ko
    insmod icp.ko
    insmod zavl.ko
    insmod zlua.ko
    insmod znvpair.ko
    insmod zunicode.ko
    insmod zcommon.ko
    insmod zfs.ko
    and then you're good to go. zfs and zpool userspace tools should work.
    If you don't already have usb enabled in the gui, you may have a few more kernel modules to load beforehand.

    Maybe a couple useful commands to get you going

    Create new pool (-m is mountpoint)
    zpool create \
    -m /mnt/tank \
    tank \

    Create new pool with whole pool encryption
    zpool create \
    -m /mnt/tank \
    -O encryption=aes-256-gcm \
    -O keyformat=passphrase \
    tank \

    or create single filesystem with encryption
    zfs create \
    -o encryption=on \
    -o keyformat=passphrase \

    other useful commands
    zpool get all
    zfs get all
    zfs get checksum tank
    zfs set checksum=on/off tank
    zpool import tank
    zpool import tank -l (prompt for encryption password)
    zpool export tank
    zfs mount -a
    zfs mount -a -l (prompt for encryption password)
    zfs set mount=/mmc tank/somefilesystem
    zvols currently don't work due to that code using kernel api's not found in, and me just commenting those lines out to get it to compile. They seem to claim compatibility back to 2.6.32, so I've submitted bug report with that and other apis not present in old kernels.
    Last edited: May 29, 2019
    AndreDVJ, Riddlah and rs232 like this.
  2. rs232

    rs232 Network Guru Member

    Lance, you really add value to this forum! :)

    Nonetheless I would discourage people from using ZFS (apart for fun like Lance does), which is an enterprise filesystem, on your home devices. It requires tons (GB and GB) of memory to perform well. It was actually designed for SAN environment so it will not help you with your USB storage, opposite it's likely you'll be disappointed.

    I had ZFS running on my NAS for a couple of years I assigned to the VM something lile 16GB of RAM where the storage was 4x3TB sata in a RAID5. It was extremely slow over and complained it didn't have enough RAM. I spent weeks of my life looking into this and concluded that it's just not worthy for home environment, not even a large NAS. All the headaches are just not worthy to benefit from de-duplication and little more.

    Close the parenthesis I'm having wonderful results with XFS, so wondering if that can be used in tomato instead?
  3. lancethepants

    lancethepants Network Guru Member

    I initially thought that memory would be the biggest issue, and it certainly does like it. The kernel modules take a few megabytes, but the real ram usage happens when doing file transfers. It does seem to know to stay within its constraints because I've never seen it max out completely, or go over 200MB of the 250MB in the router. Funny routers have all this ram, but we hardly touch any of it.

    If you were doing deduplication on a small nas, that definitely could have hampered things. With my core i5 & 32GB ram I don't even think dedup is worth it for the marginal benefit in a home environment. Space is cheap and I can't imagine I'd have all that much duplication. RAIDZ (Raid5) is computationally expensive as well for a low-powered device to compute all that parity. I strictly stick to mirrored vdevs. Sure, you don't get as much space as RaidZ, but it sure saved my butt when two disks failed in a single week. A situation where RAIDZ would have lost me all my data. I was lucky to two disk were on different vdevs. I now keep an extra copy on external storage (the one I'm playing with on the router) because of that.

    XFS might be a possibility.
  4. lancethepants

    lancethepants Network Guru Member

    So I got zvols to work. It requires a couple linux api functions to be backported to our kernel. A very minimal and non-intrusive kernel back-port. The zfs team may rewrite the zvol code to not require it, but it's kind of up in the air right now.

    For zvols to work it also requires a new set of kernel modules I've made available.
    I've also compiled and uploaded the latest version, 0.8.0, now that it has been released.
    rs232 likes this.
  5. rgnldo

    rgnldo Networkin' Nut Member

    ARM Routers with poor processing and memory are limited. I configured a 500GB SSD in ext4 without journaling and got 60mb/s in Samba, with wireless.
  6. lancethepants

    lancethepants Network Guru Member

    Yes, zfs will perform very poorly in comparison with other file systems on tomato. I'm surprised it would run at all.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice