Speeding up the SAMBA by 30%

Discussion in 'Tomato Firmware' started by ryzhov_al, Nov 5, 2012.

  1. ryzhov_al

    ryzhov_al Addicted to LI Member

    Eric Sauvageau discovered a very easy way to accelerate file transfer:
    • find use sendfile = yes in SAMBA config,
    • replace "yes" to "no",
    • restart SMBD.
    My tests gives a 30% speed up while copying files from RT-N66's USB disk to Win x64 PC. Looks like a sendfile() syscall too heavy for our routers or\and it's uClibc implementation too poor. May we include this to Tomato firmware? If you wish, of course.

    Eric, again and again: thank you.
  2. kthaddock

    kthaddock Network Guru Member


    Thank you for posting all godies here :)

  3. PBandJ

    PBandJ Addicted to LI Member

    That's very interesting. I'm not using SMB with my Tomato router. I'll have to experiment with that on my NAS (DNS-323).
  4. ryzhov_al

    ryzhov_al Addicted to LI Member

    Please note that sendfile() is a kernel specific and implemented for every filesystem separately. For example, with kernel 3.0 + uClibc 0.9.32 + EXT4 turning this option off will degrade SAMBA perfomance.

    We stuck here with Broadcom's SDK on kernel forever. So we have to take any advantages as we can.
  5. koitsu

    koitsu Network Guru Member

    The sendfile() syscall on Linux has been historically a pain in the butt -- meaning they break it all the time, fix it all the time, break it again, etc.. This looks to me like the kernel, sadly, may have broken sendfile() support (or at least non-optimal). It wouldn't have anything to do with uClibc -- it would either be a problem with the kernel, or a problem with the Samba version provided on the router.

    It could also be a problem with the sendfile() implementation for the filesystem you're using -- you didn't state what filesystem your "USB disk" is using. For example, if it's NTFS, try using ext2 or ext3 instead and see if there's any gain or loss. But keep reading...

    However, before advocating this change, more testing needs to be done. Your tests were done, quote, from RT-N66's USB disk to Win x64 PC. Please remove the USB stack from the testing procedure -- we need to be testing/benchmarking using a memory/RAM disk as the source, because there are also known performance oddities with the USB layer on these routers (USB support on SOC is quite bad compared to a dedicated IC).

    I've written a fairly extensive "tuning document" for FreeBSD and how to get Samba to have excellent I/O performance there, so I imagine I could write something similar for TomatoUSB. Linux is a completely different beast though, and I do not use Samba on my RT-N16 router. The most notable performance improvement in Samba comes from enabling use of AIO, which to my knowledge is disabled by default. The values to pick for AIO adjustments are specific, and I should note VERY CLEARLY that what you'll find on the Internet is often 100% wrong, so don't go using values you read online. I can provide further reference info if you want.
  6. ryzhov_al

    ryzhov_al Addicted to LI Member


    For example, if it's NTFS, try using ext2 or ext3 instead and see if there's any gain or loss. But keep reading...

    Later. Now I'm showing differences on real task, where both weird USB controller and heavy user-space program fights for (weak) CPU resources.

    @All: just checked. Switching off sendfile() in VSFTPD gives a 33% perfomance boost. All you have to do is:
    • find vsftpd config,
    • add a use_sendfile=NO string there,
    • restart vsftpd.
  7. koitsu

    koitsu Network Guru Member

    Your comment about fighting over CPU slice time makes sense -- yep, sounds like sendfile() isn't optimised for the CPU in question, or is just downright busted (badly designed) on Funny how kernels/features are added to things with power-hungry desktop and server CPUs in mind -- the embedded world is very, very different.

    For those wondering: there's also no harm in disabling sendfile(). In fact, there's actually good reasons to disable it (for example on FreeBSD sendfile() would cause major problems when used with ZFS; the issue had to get fixed in both the sendfile() syscall code as well as within ZFS; and sendfile() is known to sometimes behave oddly when used against files that are stored on NFS).

    Long story short: we should definitely see about disabling this universally, in Samba as well as any other userland daemons we have in TomatoUSB.

    As usual, thanks for the great insights, ryzhov_al!
  8. RMerlin

    RMerlin Network Guru Member

    But to be fair, no user shares files from their RAM disk. What they care about is the performance from their USB disk to their client. Anything else is irrelevant :)

    I wish we had some skilled kernel developers around. If the sendfile() implementation in 2.6.22 is actually broken, backporting a newer version might be an even better solution. Or, it could be that sendfile() is simply not optimized for scenarios where you have a fairly weak CPU like these routers do.
    koitsu likes this.
  9. RMerlin

    RMerlin Network Guru Member

    Interesting. From what I found, the sendfile() implementation got scrapped in 2.6.23, to be replaced with a new implementation based on splice() (which is basically a superset of sendfile()). I couldn't find anything pointing out at the performance implication of that switch unfortunately.
  10. Mangix

    Mangix Networkin' Nut Member

    is it possible to update the kernel in tomato? I have a wndr3400 running dd-wrt with version of the linux kernel running...
  11. koitsu

    koitsu Network Guru Member

    It is possible to update the kernel, but quite literally everything has to be tested thoroughly. I don't mean "yeah the program runs", I'm serious when I say things have to be tested extensively. The other problem with updating the kernel is that the wireless driver, as I've mentioned in other threads, is a binary blob. There's no source code to it. The chance of it breaking when changing kernels (either downgrading or upgrading) is extremely high, and how the breakage manifests itself can vary (sometimes it might be obvious like a missing symbol, other times the driver might load but not function correctly due to internal kernel ABI semantics changing).

    If you're worried about how old the Linux kernel is on these routers, I strongly recommend you use OpenWRT. The developers of that firmware are much more "in tune" with keeping things up to date (if possible) and trying to get some headway with vendors like Broadcom (to do away with the binary blob wifi driver and give source instead).
  12. RMerlin

    RMerlin Network Guru Member

    Updating the kernel is probably out of the question for both Asuswrt(-Merlin) or Tomato. Aside from the binary-only drivers, there are a lot of patches that were accumulated over time on the current kernel - many of which by Broadcom themselves. The kernel used by Asuswrt (for example) has bits of code added to support their boards. With the RT-AC66U, they added support for the NAND technology used by that specific router. And I had to implement the AC66U JFFS support to the kernel myself. So basically, it's already a highly customized kernel, mixed with backported patches from upstream. Nobody could sort out what needs to be up-ported or dropped if they were to move to a newer kernel - only Broadcom actually knows and understands what was added in there.

    So IMHO, the only way toward a newer kernel would be for Broadcom to release a newer SDK that is based on one Ralink has done the transition to the 3.x kernel on their newer products, so it will most likely happen eventually. There's a good chance it will be only for newer chips however, I doubt they would go back to migrate existing platforms to the new one.

    How does DD-WRT deals with it? They barely do. Despite having two fulltime devs working on DD-WRT as their main job (I'm assuming eko is also doing this fulltime, I might be wrong), you can still see the various issues cropping up related to the kernel/driver issues. ext3 access causing kernel panics on the RT-N66U, for example. That issue has been on their ticket system for over a year now I think. Or they couldn't support 64 KB NVRAM on the RT-N66U until someone actually updated the CFE, because it would have required more kernel patching. And you can see how complicated still it is - they have builds with 2.4 kernels, 2.6 kernels, and maybe two or three different drivers, based on which router you are going to run.

    I wish a kernel upgrade would be a simple drop-in-place type of thing, but it's much, much more complicated than that. Currently, the best (and possibly only) way to deal with this is to backport cherry-picked bits when appropriate.
    The Master and Python46 like this.
  13. ryzhov_al

    ryzhov_al Addicted to LI Member

    I can:) Asus kernel is a vanilla + LinuxMIPS +
    Broadcom's hardware support. The (almost) rest job done by Leonid Lisovsky (lly), Vladislav Grishenko (theMIROn) and Fedor Kozhevnikov.

    You may see how all it's patches applied to vanilla kernel here.
    BTW, sendfile() problem is solved, try to "grep" this bunch of patches for {sendfile,splice} keywords.
  14. RMerlin

    RMerlin Network Guru Member

    That probably doesn't include the MTD and NAND patches Asus applied for the AC66U, or all the IPv6 patches/backport Asus did a few months ago tho.

    I'll have to take a look to see if the sendfile() patch actually helps performance or not.
  15. RMerlin

    RMerlin Network Guru Member

    I checked some of the sendfile/splice patches in that list, and they are already applied to Asus's kernel, so they won't help us with our current performance issues - they're already applied.

    Looking at this insanely long list of patches, it's even more obvious to me than ever that changing Tomato/Asuswrt's kernel would be a very difficult, bug-prone affair. The current kernel is way too customized.
    koitsu likes this.
  16. mstombs

    mstombs Network Guru Member

    You have to be careful with the reported kernel version in Tomatousb, the reported string is fixed because of the drivers with version specific kernel modules, but as mentioned lots of backports from later kernels. dd-wrt has commercial links via Buffalo for example, so reasonable to assume they have access to custom driver builds, but I always suspect they just bump the reported kernel versions in kernel and driver to make it harder for others to re-use the Broadcom binary blobs in same way we can do with those from Linksys or Asus. Yes I know dd-wrt has great svn/trac system for all the 'open' source for huge number of different products, but I was never able to get anything compilable out of it!

    I really like the look of this, in OpenWRT style it seems you start with official Linux kernel release and apply all the patches. Fedor used to filter all the good stuff from WL500g project into tomatousb, I wonder if its time for a big reunion? IIRC WL500g also using newer uclibc and toolchain.
  17. quihong

    quihong Networkin' Nut Member

    Has this optimization option been added to the latest (Shibby) TomatoUSB builds?

    If not, is it just a matter of adding
    "use sendfile = yes"
    in the "Samba Custom Configuration" box under "USB and NAS"->"File Sharing" screen?

    Or do I have to provide a complete smb.conf file in the samba custom config box. Thanks
  18. quihong

    quihong Networkin' Nut Member

    I meant:

    "use sendfile = no"
  19. volgera

    volgera Networkin' Nut Member

    Yes you can. After that, if you cat smb.conf in /tmp/etc/ you'll be able to see your added line there.
    quihong likes this.
  20. quihong

    quihong Networkin' Nut Member

    Thanks @volgera!

    That worked (changed confirmed in /tmp/etc/smb.conf).

    I'll have to give it a quick test this weekend to see how much of a difference it makes.
  21. leandroong

    leandroong LI Guru Member

    ZTE ZXV10 H618B
    Tomato Firmware 1.28.0000 MIPSR1-107 K26 USB BTgui-VPN
    Thanks !!! It boosted like 88% on my samba transfer speed.
  22. lilstone87

    lilstone87 Serious Server Member

    Well I did this, and it seems it increased write speed to my connected ntfs drive from 3MB's to 3.3MB's. So around about a 11% increase, I know I need to get another drive, and not use ntfs as tomato's driver for ntfs sucks in "write to" speed. I like tomato firmware alot, but miss Asus build for its driver for my N66U, as I can avg between 14-16MB's writing to my ntfs drive.
  23. merclin

    merclin Reformed Router Member

    on Openwrt Breaking Barrier r39402,
    test done with a usb2 drive

    sendfile yes :
    smbd 60%cpu
    usbstorage 10%cpu
    total : 70%cpu

    sendfile no:
    smbd 45%cpu
    ksofirqd 15%cpu
    total : 60%cpu

    if cpu is your bottleneck , this seems to help.

    (In my case, cpu was ok. Passing from 70% to 60% gave no visible transfer speed improvements.)
  24. FYI I've got tp-link wr-842nd 400MHz and I'm getting little over 6MiB/s samba downlaod with openwrt bb 14.07 final. Disabling sendfile degrades perf a little to little under 6MiB/s. The external disk drive is surely faster than that (250GB USB2 HDD).

    writing to share:
    $ dd if=/dev/zero bs=1MiB count=100 of=/mnt/NAS/testfile conv=sync
    CPU: 1% usr 25% sys 0% nic 44% idle 9% io 0% irq 19% sirq

    reading from share:
    $ pv < /mnt/NAS/testfile > /dev/null
    CPU: 0% usr 14% sys 0% nic 54% idle 0% io 0% irq 30% sirq

    doing write on router itself:
    # time dd if=/dev/zero of=testfile bs=1048576 count=100 conv=sync
    100+0 records in
    100+0 records out
    real 0m 4.48s
    user 0m 0.00s
    sys 0m 2.36s
    I calculate this means around 20.8MiB/s actual possible write speed

    doing read from disk on router itself:
    # time dd if=testfile of=/dev/null bs=1048576
    100+0 records in
    100+0 records out
    real 0m 5.25s
    user 0m 0.00s
    sys 0m 2.13s
    I calculate this means around 18.3MiB/s actual possible read speed

    looks like samba is the bottleneck but CPU is not maed out.

    update: I may have a networking issue at the moment, so it's possible that CPU is not maxed out because network is slow.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice