Tomato WAN to LAN throughput below par on Asus RT-N16 - future bottlenecks

Discussion in 'Tomato Firmware' started by jaybee, Jan 29, 2013.

  1. jaybee

    jaybee Serious Server Member

    - posted the below on tomato usb forums then read that posting on linksys info would be better as more active community -

    Hi all. Let me be the first to say I have previously ran Tomato and had a great experience with it. This is why I recently purchased an Asus RT-N16 in order to install and use Tomato on it. Unfortunately I did not see documented anywhere the fact that Tomato causes such a poor WAN to LAN throughput. I even read of one person somewhere saying the RT-N16 was capable of over 250mb/s WAN to LAN on Tomato. This is not the case in my testing.
    Why does this matter?
    Internet home connection speeds are getting faster. I want to buy a router that will last for a few years with my configurations. A fit and forget if you will, at least for some amount of time. Tomato allows this as it is known for being so stable. Currently Virgin Media in the UK have connection speeds of 120mb available. This will in time rise most likely to 200mb as this has already been trialled. People do not want to use the Virgin Media superhub since it is so buggy. They want their own router. I have tested using various firmwares and these are the results.

    Asus RT-N16 router throughput results

    *I notced that on they tested this router as having about 140 mb/s in both directions. I do not understand why this is so low, as even on the first Asus firmware available for this device I see speeds of over 200 mb/s. Perhaps the internal hardware changed, but I am only aware of 1 hardware version so far.*

    Asus official firmwares:

    Stock Asus Firmware out of the box version
    WAN to LAN: 228 mb/s
    LAN to WAN: 240 mb/s

    Asus Firmware updated to latest (featuring new WRT skinned GUI)
    WAN to LAN: 300 mb/s
    LAN to WAN: 341 mb/s

    Tomato builds:

    Tomato-K26USB-1.28.9054MIPSR2-beta-vpn3.6 (latest official that I could find)
    WAN to LAN: 115 mb/s
    LAN to WAN: 109 mb/s

    Shibbytomato-K26USB-1.28.RT-MIPSR2-105-VPN (latest Shibby VPN build I could see)
    WAN to LAN: 109 mb/s
    LAN to WAN: 103 mb/s

    tomato-K26-1.28.7501.2MIPSR2Toastman-RT-Std (latest standard Toastman build I could see)
    WAN to LAN: 110 mb/s
    LAN to WAN: 111 mb/s

    tomato-K26USB-1.28.7501.2MIPSR2Toastman-RT-VPN (latest Toastman VPN build I could see)
    WAN to LAN: 110 mb/s
    LAN to WAN: 111 mb/s

    I also tested DDWRT via an old mini build:

    DDWRT_mini_v24-sp2_12-18-09 (used as an Intermediate build before going to Tomato)
    WAN to LAN: 113 mb/s
    LAN to WAN: 149 mb/s

    Notice that the stock Asus firmware is much faster. You would expect this to be faster, but it is massively faster. On a 120mb Virgin UK connection then most Tomato builds would bottleneck your downstream speed. This makes Tomato unusable going forward on this and many other routers.
    I then found in a search some information about FAST-NAT. It has been said that since a certain build (sorry I am still trying to find the thread on build number) Fast-nat has been disabled because it is known to cause issues with some features of Tomato and this needs to be fixed with future developments but will be difficult.

    The Asus RT-N16 uses FAST-NAT in order to provide very fast throughput. But essentially, on Tomato there seems to be a file located at:


    The file is called: ip_conntrack_fastnat

    The file has a value of simply "1" inside it, which sets it to be ON or enabled. Setting to "0" turns it off.
    I looked up this file in the latest official 1.28 build 9054 and as promised, this feature has been removed since the file does not even exist. However, it does exist (and is set to ON) in both the Shibby and Toastman builds I tested above which still come out with very slow throughput results. Why might this be?

    If I go back to an official 1.27 build as below before when it was removed, it gives much better throughput:

    WAN to LAN: 210 mb/s
    LAN to WAN: 223 mb/s

    Is the fact that the file is present in the latest Toastman and Shibby builds meaningless? i.e. The file is there and set to ON but some other over ride is not enabling it properly? Is this a particular issue with Asus RT-N16 and fast-nat on Tomato latest builds with it still set to on?
    If this is the case, and the only way to currently get fast throughput is using older builds, I have three questions:

    1: What risks are there running an older build using fast-nat? It was not made particularly clear in other threads what exactly this can break. Somewhere I think I read of QOS or bandwidth monitoring issues?

    2: Are other routers with fast throughput crippled by Tomato if they use fast-nat to get such speeds normally?

    3: What about routers that DO NOT appear to use fast-nat or are not documented to, but supposedly have fast throughput. Is it possible to run a router on Tomato with 200 mb/s+ WAN to LAN throughput and if so which one and which build?
  2. koitsu

    koitsu Network Guru Member

    Simply put, you're asking why the Linux kernel option ip_conntrack_fastnat was removed from later TomatoUSB firmwares. This ""file"" is actually a kernel tunable, as in a way to turn something on and off. These are scattered throughout /proc. (I can see you don't have much familiarity with Linux, which is perfectly fine -- so I'm educating you :) ).

    The reason it was removed is because it's known to break things under certain configurations. I can point you to all the threads on the topic, but you need to read them completely and not skim them.

    This is an old changelog in the Toastman firmware, but it mentions the fastnat issue right at the top: Builds/RT (MIPSR2 - newer routers like RT-N16, E3000,3500L etc) - NORMAL BRANCH/READ THIS CHANGELOG FIRST.txt

    As far as the kind of throughput you're wanting -- you might be able to get that using an RT-N66U, but generally speaking you're going into speeds that warrant a real router/product from a real company, not a residential product. You should be talking to companies like Juniper and Sonicwall. Expect to pay a pretty penny -- but you will get the speed (and upgrade/capacity) you want.

    The reality of the situation: not a lot of ISPs here in the States have that kind of speed, at least not at an affordable price. Verizon FiOS is only available in very specific states on the east coast and in very specific cities/regions, and costs quite a lot. Cable providers (Comcast, Cox, Time Warner, Charter, etc.) do not tend to offer anything that high -- Comcast is the one exception, and practically no one has it due to its cost.

    I'll point out that we've been down this road before in the past. Back when the WRT54G (the very original) was released. ISPs 4-5 years later began increasing their speeds, and suddenly the router CPU became the bottleneck. People began complaining, ISPs began insisting customers not use routers, blah blah. It took another 2-3 years for the hardware to catch up. This situation will perpetually happen unless the consumer purchases something decent/high-end and stops expecting consumer-grade SOC-based products to push so much traffic. These consumer-grade routers use very slow CPUs (they're absolutely nothing like what your desktop has in it, or even a desktop made 7-8 years ago); only local LAN traffic is handled predominantly by hardware offloading.

    The other part to this is the fact that the Linux kernel being run on these routers is old (8-10 years if I remember right). It has to remain that way because of the use of binary blob wireless drivers provided by Broadcom. Meaning: the wireless drivers are not source code that can be compiled or fixed or adjusted. Broadcom provided Linksys/Asus/etc. a raw binary blob and said "here you go". Changing kernel versions results in this driver breaking. Broadcom is one of the worst companies in this regard, and is very anti-open-source (search for "Broadcom" here). Broadcom is also the manufacturer of the SOC itself, meaning all the features/etc. are on a single chip manufactured by one company and that company alone.

    That's the reality of the situation.
  3. jaybee

    jaybee Serious Server Member

    That's a very informative and helpful reply Koitsu and I appreciate it. I apologize for not replying sooner but I have had problems accessing this forum from the UK, not sure if it was just me. I like this reply in your post: "I can see you don't have much familiarity with Linux, which is perfectly fine -- so I'm educating you" which made me lol. Quite brutal, but I like honesty and direct to-the-point statements like that. :) I do have some basic Linux familiarity and do administer Solaris and some Linux distros, but yes, I am by no means an expert and can merely "get by" for most basic things. :)

    I will read through those threads you listed. Thank you.

    With regard to the suggestion to use the RT-N66U, I think unfortunately I have read before on here or some other forums, that people got similar speeds on that when using Tomato. Presumably it also used some means of FAST-NAT to reach said speeds. I think you are right, the reality of the situation is that it is not looking good for the time being for people in the UK with consumer grade equipment.

    I would love some input as to whether you recommend giving the RT-N16 a go on the older 1.27 build WITH fast-nat enabled. The features I use would only really require VPN. I don't use QOS. I will go and read those threads in detail to see what it does break. I might come back and post a "list" of things known to break. When I read around on this before, i could only see that QOS broke....
  4. jaybee

    jaybee Serious Server Member

    OK I just read those threads and the threads they linked to within them. I can see only two things that it clearly breaks with Fast-Nat enabled:

    1: QOS
    2: Access restrictions (i.e. Http url blocking)

    There is nothing else mentioned. You mention yourself it causes an ip tables problem, but actually translating this through to the features that are affected and break only shows the above two mentioned.

    It is also not clear to me if the above two mentioned issues are problematic in the builds BEFORE 1.28. (They were removed/disabled from 1.28 onwards).

    Even if they are broken in builds pre dating 1.28, I can live without access restrictions and QOS I think.
  5. sandspike

    sandspike LI Guru Member

    Trying to revive an old thread: I think this limitation may have smacked me in the face as well. I have AT&T Uverse Gigapower 300Mb/s Up and Down. If I bridge their modem/gateway over to my Netgear WNR3500L, running Shibby Tomato, I only get 100Mb/s, but with their router 320Mb/s. Is their a build with Fast-NAT enabled, removing QOS and URL filtering? I have to have the full speed, but I hate their router, and want my tomato back.
  6. chrcoluk

    chrcoluk Reformed Router Member

    looks like I am also affected.

    the router is struggling to push approx 70mbit of traffic from wan to lan on my ac66.

    if I start a steam download the cpu is so saturated than I cannot browse the router lan pages, I have slow response on ssh, and web browsing goes to a crawl, I dont remember it been this bad before but regardless the cpu is been maxed out on the router.

    In the modern internet routers (and their firmwares) need to push 100mbit+, and the ac66 easily does it on the merlin/asus firmware's.

    I suggest prioritising FASTNAT over QoS and web url filtering, so basically its enabled by default, and make a seperate build for those 2 features or allow people to turn it off to access those features, the path to simply remove it seems a very lazy approach.

    I do like shibby tomato but I think this issue may make me move back to merlin's firmwares.

    merlin's are ok but the traffic monitoring isnt as good, the ui loads much slower in the browser and the VPN implentation also isnt as good as tomato, also ipv6 is more broken on merlin, which is why I moved over to tomato, but I cannot afford now to go out and buy a faster router, meaning my choice is to swap firmware again :(

    unless fastnat can be put back into shibby.
  7. mstombs

    mstombs Network Guru Member

    FastNAT is an opensource Linux kernel option, it skips some tracking steps for better throughput. Broadcom have a closed source binary kernel module patch called "ctf" "cut through forwarding" that somehow optimizes throughput between Broadcom wireless and switch drivers, bypassing the standard Linux code. Asuswrt uses the Broadcom module. Bypassing the kernel will break Tomato QOS, port forwarding, IP accounting etc, so little interest in patching it back in as an option - but it could be done. It's great we have Merlin firmwares to fall back to! I can get more than 200Mb/s download wired via a Tomato E3000 in AP mode and an N66 with asuswrt-merlin, Tomato could easily break 100Mb/s barrier - are you testing with wireless?

    See the graphs in this early review of an AC66 and N66

    The HW NAT has a dramatic effect 500%?

    Clearly real throughput when using wireless encryption and ppp wan connection are not used for advertising!

    What we need for Tomato is an opensource fast-path implementation
    Last edited: Apr 10, 2015
  8. chrcoluk

    chrcoluk Reformed Router Member

    I have just reinstalled merlin asuswrt so I can do a comparison, although still configuring it.
  9. chrcoluk

    chrcoluk Reformed Router Member

    just done a quick test, surprisingly exact same issue on merlin's asuswrt

    various people have said the ac66 can handle much higher than 70mbit wan to lan so any ideas?

    its the same problem, cpu saturated by software interrupts.

    disabling ip traffic monitoring improves it very slightly but nothing large, CTF is enabled in the gui.
  10. chrcoluk

    chrcoluk Reformed Router Member

    ok this is on the sysinfo page

    HW acceleration Disabled - incompatible with:

    and yes it is actually blank at the end :(
  11. RMerlin

    RMerlin Network Guru Member

    This usually means that you disabled a feature that's incompatible with it, but failed to reboot the router to clear it, so the router never had a chance to load and initialize the CTF module. Check again after a reboot.
  12. chrcoluk

    chrcoluk Reformed Router Member

    ok I had to reboot after disabling ip traffic monitoring to get it enabled.

    with it enabled cpu still maxes out :confused: at just 70mbit throughput but its sys cpu usage now instead of interrupts and it stays reasonably responsive whilst under full speeds. so is improved but not in the way I expected.

    per ip monitoring is useful but I will live without it for now.

    also didnt notice your response earlier merlin thanks, you were correct, reboot cleared it.

    also to mstombs I appreciate the reply but no way it breaks all that. I found another post which said it just breaks 3 things.

    web url filtering
    per ip monitoring

    plus asuswrt works fine with it on and port forwarding.

    I dont see why shibby cant do the same as merlin, have it auto disable when a conflicting feature is enabled but allow it to be enabled at other times.
  13. mstombs

    mstombs Network Guru Member


    for comments from tomatousb re incorporating bcm ctf.

    It has the potential to break anything in routing or iptables configured filtering (maybe access restrictions more likely than port forwarding?)

    I've also seen router limiting with high sirq, as far as I can tell the Broadcom Ethernet and Wifi drivers do their stuff under software interrupts so this category just means the router busy.

    For the AC66 to limit at 70Mbps - are you using PPPoE WAN type?
  14. chrcoluk

    chrcoluk Reformed Router Member

    yes is pppoe, has anyone tested routers with pppoe and CTF off?
  15. chrcoluk

    chrcoluk Reformed Router Member

    well download just finished and is 0% idle still, mostly used by sys.


    Mem: 68360K used, 171204K free, 0K shrd, 9232K buff, 30068K cached
    CPU: 4.3% usr 95.2% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.3% sirq
    Load average: 1.13 1.13 1.09 3/57 1939
    1 0 admin R 2488 1.0 0 99.6 /sbin/init
    1431 699 admin R 1436 0.6 0 0.2 top
    686 1 admin S 5060 2.1 0 0.0 minidlna -f /etc/minidlna.conf -R
    690 686 admin S 5060 2.1 0 0.0 minidlna -f /etc/minidlna.conf -R
    691 690 admin S N 5060 2.1 0 0.0 minidlna -f /etc/minidlna.conf -R
    322 1 admin S 4116 1.7 0 0.0 httpd -s -p 8443
    682 1 admin S 3300 1.3 0 0.0 smbd -D -s /etc/smb.conf
    329 1 admin S 2476 1.0 0 0.0 watchdog
    307 1 admin S 2476 1.0 0 0.0 /sbin/wanduck
    777 1 admin S 2476 1.0 0 0.0 ntp
    785 1 admin S 2476 1.0 0 0.0 disk_monitor
    373 329 admin S 2476 1.0 0 0.0 ots
    559 1 admin S 2476 1.0 0 0.0 usbled
    316 1 admin S 2476 1.0 0 0.0 wpsaide
    161 1 admin S 2468 1.0 0 0.0 console
    681 1 admin S 2376 0.9 0 0.0 nmbd -D -s /etc/smb.conf
    699 698 admin S 2192 0.9 0 0.0 /opt/bin/bash
    564 1 admin S 2160 0.9 0 0.0 u2ec
    565 564 admin S 2160 0.9 0 0.0 u2ec
    566 565 admin S 2160 0.9 0 0.0 u2ec
  16. chrcoluk

    chrcoluk Reformed Router Member

    I think its related to ipv6, I am on one of my isp's broken endpoints where ipv6 doesnt work and any service command seems to fail and the log shows is waiting for radvd to start, so asuswrt-merlin seems to have a flaw whereby if radvd has an issue it simply waits forever with init hogging cpu. so it does seem ctf has helped a lot and the sys issue is unrelated but caused by the ipv6 problem.
  17. chrcoluk

    chrcoluk Reformed Router Member

    after fixing the radvd issue (disabled auto start, and run it manually as needed).

    this is the cpu usage with CTF :)

    Mem: 67020K used, 172544K free, 0K shrd, 8924K buff, 28184K cached
    CPU: 0.9% usr 0.1% sys 0.0% nic 97.4% idle 0.0% io 0.1% irq 1.1% sirq
    Load average: 0.00 0.00 0.00 3/64 1888

    seems to me not using CTF is obselete on the modern internet unless the router has a really beefy cpu.

    thanks to everyone who replied.
  18. mstombs

    mstombs Network Guru Member

    What firmware uses radvd? ipv6 ra handled by dnsmasq now.

    ppp connection won't benefit from ctf - the Ethernet frame has to be re-constructed in the kernel - it doesn't just appear in the switch driver, so is bound to benchmark less than simple wired comms without ctf - see the graphs on the .ru site reviews I linked above.

    Pure ipv6 doesn't have the nat issue and should be amenable to HW acceleration, but if tunnelled over ipv4 clearly requires extra processing.

    ctf and simple Ethernet comm benchmarks definitely essential for router marketing these days:- linking back to the first posts these speeds are possible these days:-

  19. chrcoluk

    chrcoluk Reformed Router Member

    the 374_43 fork of merlin still uses radvd, is maintained by john.

    and I Can assure you there is a massive difference with ctf, the software interrupts has plummeted from 95%+ to under 10%. (with 70mbit throughput).

    until a fix is found radvd handles mtu properly for ipv6 whilst dnsmasq doesnt (at least it doesnt on shibby), my bug report regarding ipv6 mtu remains unanswered currently I filed it months ago.

    so you not confused I am dual stack, much stuff I use such as steam still uses ipv4.
    Last edited: Apr 11, 2015
  20. RMerlin

    RMerlin Network Guru Member

    Broadcom added CTF support for PPPoE starting with SDK 5.110. CTF mostly deals at the Netfilter level, so PPPoE packets do benefit from it - altho you will still have some additional overhead versus straight Ethernet.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice