Hardware NAT, CTF, FA in Tomato

Discussion in 'Tomato Firmware' started by trevorw, Feb 3, 2016.

  1. trevorw

    trevorw LI Guru Member

    Hi everyone,

    This is a topic that has been discussed in the past however I couldn't find one dedicated thread for it.
    For a while now, some router vendors (in particular Asus) has enabled on their routers (whether Broadcom or MediaTek/Ralink) the feature of 'hardware nat' or CTF (Cut-Through-Forward) and in some recent version another layer of it called FA (Flow Assistance).
    RMerlin, the man himself, has a nice thread about these <a href="http://www.snbforums.com/threads/broadcoms-hardware-acceleration.18144/"> here</>.
    Unfortunately these are not available in Tomato, only in Asuswrt (official firmware) or asuswrt-merlin or padavan. Which is ironic since, as far as I know, asuswrt is based on (an old) version of Tomato.

    As the Broadcom SDK is being used for the ARM support, I wonder if there's even a small chance for these proprietary, closed-source modules to be incorporated into Tomato?

    In my case after close to 8-9 years, of using Tomato on my routers, I'm forced to move of it as it cannot properly handle my network connection despite the hardware being capable: my ISP is using PPPoE - initially with 100mbps / 100mbps, my reliable RT-N16 worked just fine. After being upgraded to 200mbps/100mbps, I've noticed Tomato maxing out at ~140-150 mbps. After using `bcm_nat` module, it went past 200 mbps. However now that the connection has been upgraded to 300 mbps downstream, even with the module loaded it cannot go past 205-206 mbps. Switching to Asuswrt-merlin (and thus with CTF enabled) made the N16 to easily reach 300 mbps with a CPU load of ~0.25. on PPPoE nevertheless.
    Yes it's an 'old' router however with the proper firmware, it turns out it's quite capable.
    Another Asus router, N56U with Ralink SoC (I know Tomato is Broadcom only) turned out to be quite a beast using Padavan firmware.
    In both cases though, I would have much preferred Tomato.

    I know a lot of features (like QoS) will not work with CTF but then again, it's easy for folks to pick and chose.

    Would the Tomato ARM see hardware nat/CTF/FA feature or not?

    Thank you!
  2. Malakai

    Malakai Networkin' Nut Member

    CTF exists and is working pretty well on ARM (R7000 with Tomato Shibby MultiWan AIO 132). You can find it in Advanced -> Miscellaneous.
    I got almost the max of my Internet speed (around 900 Mbps) but I don't know exactly what features don't work when CTF is enabled so I disabled it.

    Don't know if it exists for MIPSEL as my RT-N16 is using quite an old build (Tomato Shibby AIO 120).

    Someone knows where I can find a list of all the features that don't work when using CTF?
  3. trevorw

    trevorw LI Guru Member

    @Malakai can you post a screen snapshot please? As I mentioned above for N16 I have the `bcm_nat` module loaded however the usage of PPPoE takes its tall and it looks like CTF is not enough. I assume in your case the 900 is without PPPoE correct?
    Have you tried a speed test without CTF? What numbers do you get?

  4. Monk E. Boy

    Monk E. Boy Network Guru Member

    As a general rule if you expect to do anything besides route packets between WAN & LAN, it's not going to work with CTF enabled on MIPS. CTF is essentially turning your router into a switch, so it can't perform any operations in CTF mode besides shove packets between one interface and another. Even iptables/iptablesv6 is iffy.

    I don't know if the situation is any different on ARM, but I don't see why it wouldn't. Turning all ports into switch ports and shoveling packets between them in switch mode doesn't exactly fit the model of a stateful firewall.

    As a general rule if your internet connection is so beyond the capabilities of your router that you need to enable CTF, you really should just get a router capable of handling your connection. Even an old used x86 box you pull out of someone's trash will be capable of shovelling packets between interfaces with full QoS, etc. enabled at GigE speeds. Embedded CPUs are good for many things, but by any reasonable performance metric they're not powerful.

    You can still use your old router running Tomato to bridge WLAN & LAN, your higher performance router just needs a couple ethernet ports and nothing more.
  5. somms

    somms Network Guru Member

    You just need to upgrade your router. I have no issue reaching 300Mb/s using Shibby's tomato on my r7000 gateway router.
    koitsu likes this.
  6. Malakai

    Malakai Networkin' Nut Member

    @trevorw My ISP uses PPPoE so the 900 Mbps is with CTF active and PPPoE on an Netgear R7000 with latest Shibby firmware. With the same router and PPPoE but without CTF I get around 300 Mbps (so the same as @somms).
    For the RT-N16, as I mentioned I use an old firmware on it and it doesn't have the "bcm_nat" module, but with PPPoE I get around 120-130 Mbps.

    What do you mean by that? The rules in iptables aren't checked anymore? So if I add my own rules to the firewall and enable CTF I should expect them not to work?

    And I agree with you on the x86 hardware part but having a router on which you put a nice firmware and tick a few things to get a lot of features is a lot easier and painless to configure. And it is also a matter of space and power consumption. But eventually I will get an x86 hardware some day and figure it out.
  7. vincom

    vincom LI Guru Member

    pfsense on a x86 is fairly easy to setup.
    u need a pc, 2 nics preferrably intel pros, some pc skills.
    read pfsense install instructions.
    then just go through the setup wizard.
    theres tons of options u can use/install after initial setup but thats upto you if u need them.
    for gb speeds it maybe cheapest solution if you already have a router u can use as a wifi ap & switch
    Malakai likes this.
  8. trevorw

    trevorw LI Guru Member

    @Monk E. Boy - CTF is clearly cheating (hence why things like bandwidth traffic and such aren't applied properly) however can you expand on what you mean by the firewall issues/iffy-ness. That's the main reason for a router and anything that compromises that is a problem for me.

    @somms As I've mentioned in my initial post, RT-N16 can sustain 300 Mbps (over PPPoE) when using Asuswrt; on Tomato with bcm_nat it tops up 200.

    @Malakai Is your provider RDS by any chance :) ? 900 is still shy of 1Gbps but good enough. I had no idea under ARM the CTF functionality is available (wonder why it hasn't been ported back to N16 - probably not enough interest/man power).
    My RT-N16 performance matches yours - on Tomato with PPPoE, 120-130 without CTF, ~200 with CTF).

    @vinacom / @Monk E. Boy Regarding x86, I used to do that a long time ago. In fact I had a Pentium II setup with OpenBSD 3.0 and the just released (back then) PF. It kicked us but also required maintenance and upgrades weren't that easy.
    Tomato provides all this and more in a really nice package - it's the best router fw I used. Sure it's not as powerful as pf sense but it fits my needs just fine. Same with ARM/MIPS vs an x86.
    If the CTF cheat is good enough, I'm fine with giving up the x86/64 power in return.

    P.S. Good point about cost though, the latest routers are way too expensive though most of the cost goes to the wifi bit which is not what I'm interested in; I just want wired routing performance, the wifi bit I can cover with APs.
  9. Malakai

    Malakai Networkin' Nut Member

    Indeed :)
  10. koitsu

    koitsu Network Guru Member

    Regarding CTF/FastNAT and iptables: it's highly suspected that there are some tables which are bypassed. There's almost certainly other parts of the networking stack (I'm intentionally being vague here because I don't have specifics) which are bypassed as well. Please remember: the entire point of CTF/FastNAT is to bypass portions of the networking stack, thus greatly relieving the amount of work the CPU does.

    The thread about all of that is here (please note the year -- it still applies today): http://tomatousb.org/forum/t-368941/fast-nat

    I urge someone to correct me if I'm wrong (very likely I am!), but my impression is that CTF/FastNAT is a kernel module (specifically a binary blob) that lacks source code, so it's very difficult for us to know specifically "where" it's injecting itself into the flow of network traffic and what it chooses to pass on to the underlying layers and what it doesn't. Is this possible to determine? I imagine it is, with a kernel that has debugging enabled + serial port for kernel gdb, and a lot of spare time, but...

    The well-established adage since day one here has been this: if you've got a network connection that really mandates extremely high throughput and bandwidth (i.e. 500mbit/s and higher), then you should be considering something other than a "generic home router". I would suggest looking into Ubiquity's products, or possibly conversing with Cisco and/or Juniper. Make a list of all the features you require, including the throughput rate you expect, and talk with those companies. Expect to pay for what you get, i.e. in the case of the latter two, the products tend to be expensive. There are people using ARM-based routers running Tomato (e.g. the R7000) and are doing 300mbit/s symmetrically (see above post from somms :)). ARM CPUs are substantially bigger workhorses than their Broadcom counterparts.
  11. trevorw

    trevorw LI Guru Member

    @koitsu Thanks for stepping in. I'm pretty clear that proper (CPU based) gigabit routing requires a LOT of power, in fact even x64 hardware like ALIX (from PC Engine) don't seem to be up to the task. How much cheating happens in the kernel module, only Broadcom knows - hopefully the firewall rules still apply (this can be tested though).

    Though I'm tempted to go the x86/64 route I keep coming back to Tomato - I've been using it for a long time and I'm quite happy with and it would be really nice to have the CTF functionality (which is available in asuswrt and in Tomato-ARM) available for the old kernels (K26).

    I've done a bit of testing these days and the latest Asuswrt-merlin fork for RT-N16 has the following modules installed:

    admin@RT-N16:/lib/modules/ lsmod
    Module                  Size  Used by    Tainted: P
    sr_mod                 16336  0
    cdrom                  41792  1 sr_mod
    cdc_ncm                 9952  0
    rndis_host              6272  0
    cdc_ether               5152  1 rndis_host
    asix                   17056  0
    usbnet                 19312  4 cdc_ncm,rndis_host,cdc_ether,asix
    usblp                  16272  0
    ohci_hcd               21520  0
    ehci_hcd               45248  0
    ufsd                  350192  0
    vfat                   11840  0
    fat                    56112  1 vfat
    ext2                   68512  1
    ext3                  137088  0
    jbd                    62112  1 ext3
    mbcache                 7280  2 ext2,ext3
    usb_storage            45632  1
    sg                     31552  0
    sd_mod                 27584  1
    scsi_wait_scan           960  0
    scsi_mod              105696  5 sr_mod,usb_storage,sg,sd_mod,scsi_wait_scan
    usbcore               149952 10 cdc_ncm,rndis_host,cdc_ether,asix,usbnet,usblp,ohci_hcd,ehci_hcd,usb_storage
    jffs2                 121424  1
    zlib_inflate           15040  1 jffs2
    zlib_deflate           21552  1 jffs2
    nf_nat_pptp             2592  0
    nf_conntrack_pptp       6000  1 nf_nat_pptp
    nf_nat_proto_gre        2128  1 nf_nat_pptp
    nf_conntrack_proto_gre     4128  1 nf_conntrack_pptp
    nf_nat_ftp              2816  0
    nf_conntrack_ftp        7968  1 nf_nat_ftp
    wl                   2902768  0
    igs                    18736  1 wl
    emf                    22880  2 wl,igs
    et                     51296  0
    ctf                    23056  0
    the same command for latest Tomato returns:

    root@RT-N16:/tmp/home/root# lsmod
    Module                  Size  Used by    Tainted: P
    ip6table_mangle          992  0
    ip6table_filter          704  0
    xt_recent               6800  2
    xt_IMQ                   736  0
    imq                     2320  0
    bcm_nat                 1856  0
    ehci_hcd               34576  0
    vfat                    9216  0
    fat                    45936  1 vfat
    ext2                   55392  1
    ext3                  113568  0
    jbd                    48288  1 ext3
    mbcache                 4528  2 ext2,ext3
    usb_storage            32064  1
    sd_mod                 21376  1
    scsi_wait_scan           384  0
    scsi_mod               75392  3 usb_storage,sd_mod,scsi_wait_scan
    usbcore               115088  3 ehci_hcd,usb_storage
    nf_nat_pptp             1440  0
    nf_conntrack_pptp       3808  1 nf_nat_pptp
    nf_nat_proto_gre        1072  1 nf_nat_pptp
    nf_conntrack_proto_gre     2464  1 nf_conntrack_pptp
    nf_nat_ftp              1568  0
    nf_conntrack_ftp        5792  1 nf_nat_ftp
    nf_nat_sip              5920  0
    nf_conntrack_sip       19008  1 nf_nat_sip
    nf_nat_h323             5504  0
    nf_conntrack_h323      37120  1 nf_nat_h323
    wl                   1781264  0
    et                     49280  0
    igs                    13680  1 wl
    emf                    17408  2 wl,igs

    Considering the two firmwares have the exact same kernel version(, would it be possible to simply move over the CTF module? Is there something else at hand?
  12. trevorw

    trevorw LI Guru Member

    P.S. Tried loading the ctf.ko from extras but it crashes the router. The one from asuswrt merlin is compiled differently and throws "unknown symbol".
  13. tomatosoup

    tomatosoup Networkin' Nut Member

    I compiled an image, based on Shibby v132 for my E4200 with CTF; loading the module also crashes the router.
  14. trevorw

    trevorw LI Guru Member

    @tomatosoup have you set `ctf_disable=1` in nvram? Looking at the sources it seems that if the module is present in the firmware, that setting will load it automatically (in fact, it looks like it has to be loaded before all the other modules).

    How long did it take to compile the image?
  15. tomatosoup

    tomatosoup Networkin' Nut Member

    @trevorw If you compile an image with CTF support, you can enable it easily with a checkbox from the GUI (Advanced - Misc. - CTF) - that's what I did.
    Assuming you have a working compile environment, compilation of an image takes ~10-15 minutes, depending on the speed of your machine. You just have to add "CTF=y" to the corresponding image in the Makefile.
  16. trevorw

    trevorw LI Guru Member

    @tomatosoup Thanks, I don't see it in the default build for RT-N16. I'll try and compile things myself for n16.
  17. Monk E. Boy

    Monk E. Boy Network Guru Member

    There is very good evidence that enabling CTF bypasses some tables. Port forwarding, for instance, is only possible with CTF enabled by bypassing the CTF module (by tagging packets to be forwarded), which then puts you right back in the not-enough-CPU camp for if you have a lot of forwarded packets.

    The linked thread from earlier discusses this in much more detail but ultimately they're making use of some CPU features to enable CTF (switch) mode, which based on its (current) operation is bypassing entire tables in iptables. Stepping through iptables rule by rule is CPU intensive, so they skip tables and use CTF to shuffle packets around instead. End of the day you're going to be feature limited by a closed-source undocumented kernel module. Rather than do that, I think it would be a better option to use Tomato for WLAN/LAN bridging and get a router capable of giving you all the features you need.

    Even if you were once burned by an earlier setup doesn't mean that setup is exactly the same now. x86 router builds have seen regular updates, give them a shot instead of ruling them out because you once tried one 10 years ago and didn't like how it worked. You're not limited to a single router build, there's dozens of builds available, some based around BSD, others (most) are built around Linux, a few are even based on busybox like Tomato. Core2 systems available for next to nothing are capable of whipping even the mightiest of current ARM routers. ARM does a very good job of consuming the least amount of power for the work done, which is what an embedded system wants... but that doesn't mean that the overall power will be equivalent to even an older desktop CPU.
    Malakai and koitsu like this.
  18. trevorw

    trevorw LI Guru Member

    @Monk E. Boy Thanks for the insightful post. I loved my OpenBSD router however the issue was maintenance - it's so much nicer to have a big community around the router and for upgrades to take a couple of minutes.
    Also the experience is very polished - simply fire and forget.

    I thought about putting a beffy router in front but then Tomato would only work as an AP since otherwise it will become a bottleneck. And thus most of the features (like the DHCP setup, traffic info, etc..) would be disabled.
    One thing that Tomato did write is exposing powerful functionality with a slick UI; it's so much easier to define a host with two MACs and multiple names/aliases instead of hacking some text file from the cmd line.
    Do you have any recommendations for a distro that is close to Tomato? And maybe some router build (outside building one's own)?

    Malakai likes this.
  19. trevorw

    trevorw LI Guru Member

    FTR, after making a custom compile with CTF enabled by default:
    make CTF=y V1=XXX V2=yyy r2e
    I found that, while the build ran fine (it had the ctf.ko in the firmware but disabled by default through nvram cft_disabled=1), enabling or disabling make no difference in speed wan. However using bcm_nat made.
    Funny enough, with CTF, the speed was lower (around 180-185) as oppose to the official shibby build - I suspect because the CTF is outdated and it doesn't provide much.
    Unfortunately I lack the time and knowledge to investigate this further and try finding the differences between Asuswrt-merlin and Tomato in this area...
  20. Monk E. Boy

    Monk E. Boy Network Guru Member

    You could enable the DHCP server in Tomato and have it serving up IPs for the LAN/WLAN. Tomato (or, more appropriately, DNSMasq) will merrily issue DHCP leases with another IP as the default gateway, it doesn't have to be set to itself. This fascination with having everything in one system is kind of a mystery to me. So you have a couple boxes, use what's comfortable to you for the fluffy features and leave the heavy lifting for the hardware capable of doing the heavy lifting.
    gfunkdave, koitsu and Toastman like this.
  21. Toastman

    Toastman Super Moderator Staff Member Member

    I concur with Monk's idea here. I've used multiple gateways for several years and assign clients with particular needs or requirement for best speed and reliability to their own gateways. I have experimented with multi-WAN also, including commercial solutions, but gave them up because when that single router had a problem or went down, it lost ALL internet access instead of only one gateway.

    Info on how to use multiple gateways in Tomato may be found here:

  22. trevorw

    trevorw LI Guru Member

    Old habits die hard :)
  23. gfunkdave

    gfunkdave LI Guru Member

    Since you mention Ubiquity, I'll throw in my 2 cents. I recently got to play with an EdgeRouter X ($60 on Amazon). It is a nice little piece of kit. Features are more or less the same as Tomato, though it doesn't have its own internal DNS server a la DNSMasq. A lot of the functionality needs to be configured via command line (e.g., OpenVPN). Overall it's a nice little package and can route near gigabit speeds.
  24. Hypocritus

    Hypocritus New Member Member

    CTF (Cut-Through Forwarding) is successfully working on Tomato firmware for the Tenda AC15. You have to enable it, though. I was able to break through my slow internet connection of 260Mbps to the full 1Gbps by enabling it. I am also successfully using port-forwarding and other services.
  25. pablo1107

    pablo1107 Network Newbie Member

    Hi, anyone knows if I can compile Shibby Tomato with CTF for a Linksys E3000?
  26. pomidor1

    pomidor1 Networkin' Nut Member

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice