Tomato Firmware w/Reduced Buffer Bloat (Experimental Build)

Discussion in 'Tomato Firmware' started by rcordorica, Jul 23, 2011.

  1. rcordorica

    rcordorica Network Guru Member

    I've been working on building a custom firmware with some minor patches to reduce the buffer bloat in Tomato.

    You can get it here:

    Do a Google search on buffer bloat if you are interested in the issue. There's plenty of articles and blog posts out there.

    It's based on VicTek's RAF branch. Mostly I'm focusing on MIPS R1 routers like the WRT54GL.

    Read my blog post to see the changes I've done.

    Basically I'm reducing the large buffers in Tomato so that latency is better for gaming, web page loads, and VOIP.

    If you are feeling risky, try it out and let me know if it helps. It's been working fine on my GL for a few weeks now. Toastman also tried it on his MIPS R2 based router.

    If you need some kind of custom build, let me know.

    This is probably all I'm going to do with it. I'm not too interested in building in a GUI to control the packet limit buffers, because honestly, Tomato's kernel 2.6.22 is old and is missing out on a lot of cool features to combat buffer bloat in the Linux 2.6.39/3.0 kernel. There are so many better schedulers in the newer kernels (compared to sfq and pfifo).

    In theory, if one of the Tomato dev's wanted too, they could let you control the pfifo packet limit in the GUI. In practice though, just hard coding the limit to be much lower helps out.

    OpenWRT is best for that. So in the future I'll focus my efforts there.

    Still, this greatly improves the buffer situation in Tomato as it is.

    I should point out: Try pfifo and sfq. I think sfq drops less packets, but may have slightly more latency spikes. pfifo is generally the best if you just want low latency, even if it drops some packets.
  2. gutsman7

    gutsman7 Networkin' Nut Member

    i wish there was one for 4mb mipsr2 routers.
  3. QSxx

    QSxx Network Guru Member

    Okay, I'm giving this one a try - first with Victek "stock" 9006 (just modifications you suggested on your page + tweaks clientside)

    Just a quick question: if i'm not using QoS - i should still see some improvement by putting those two lines in init section and modifying NIC settings on client side, right?

    If it shows promise, i'll go flash full version and create some basic QoS rules...

    Toastman, buddy? You're our best bet to test this one out - if you have time ofcourse :)
  4. rcordorica

    rcordorica Network Guru Member

    I can try building it. Assuming there is a profile for it in the TomatoUSB makefiles. I'll get around to it sometime later today and update my MIPSr2 link.
  5. rcordorica

    rcordorica Network Guru Member

    It should help to reduce the txqueuelen without using QoS. In theory the txqueuelen value is how many packets the network card can buffer before dropping packets. In practice, even with a txqueuelen of 2, it never drops packets, because there are plenty of other places for buffers to hide.

    Toastman tried it out on his N16, and wasn't sure if it had a meaningful effect on his latency. But he has hundreds of users on his routers and I don't know what type of QoS settings.

    My firmware most importantly limits the QoS schedulers so that they actually drop packets when appropriate. The existing builds I doubt ever drop packets because the buffers are too large (even on a 100Mbit line).


    I would rather that one of the popular Tomato builders (Toastman, VicTek, SHibby) take my patch file to reduce buffers. But if not, I'll just make this build available. Even better if they put a GUI in for pfifo queues.

    I actually think I can rig up nvram variable (pfifo_limit) so that you can change it at runtime. Should be easy.

    Actually, there exists another 2 variables, "qos_burst0" and "qos_burst1" that control the bursting limits of the scheduler. I may update my blog post to recommend better values for that.

    Lately I've been experimenting on OpenWRT, because they have builds using the 2.6.39 kernel and my WRT54-GL. What's interesting is how MUCH BETTER the newer schedulers are, even with minimal tweaking. I set my bandwidth speeds (upload and download) and the schedulers did the rest automatically. Tomato also has these better schedulers (RED, GRED, HFSC), but they aren't tweakable in the GUI. I'm also not sure if they are compiled in. Space is already at a premium, but it may be possible to add them.

    Ultimately, the newer kernel/schedulers have lots of patches to fix problems for scheduling (SFQ for example can get stuck on fragmented packets). The best thing to do is move on with OpenWRT, or just use pfifo since it is pretty simple and should work, if not the best.
  6. KyleS

    KyleS LI Guru Member

    Interesting concept. Victek did a benchmark before and saw Wan -> Lan throughput really suffered using OpenWRT. Is this no longer the case?

    EDIT: Also, would I be safe to flash the MIPSR2 build on my RT-n16?
  7. gutsman7

    gutsman7 Networkin' Nut Member

    wow thnxz rcordorica im so looking forward to trying out your build.
  8. rcordorica

    rcordorica Network Guru Member

    The biggest difference is the underlying kernel version.

    Tomato uses 2.6.22 in the newer builds. Which IMHO, is old and showing its age. Patches to fix it generally don't get back-ported from newer kernels.

    Actually, even older builds of Tomato, using kernel 2.4.x, would probably be faster for older routers because it uses less resources. But nobody really bothers making those anymore.

    DD-WRT is I think at 2.6.25 for its older devices.

    OpenWRT spans the range, from old 2.4 to 3.0. They take their time to make it work on most routers and not get locked into a particular version.

    If you benchmarked a 2.6.22 Tomato kernel vs a OpenWRT 2.6.2x kernel, the performance should be equal.

    But what I'm focusing on is latency! Most benchmarks are all about throughput, and of course, it's always a trade off of latency vs throughput. Most of our home internet speeds are pretty slow compared to 100Mb or 1GB ethernet. Unfortunately, everything seems tuned for 100Mb or 1Gb speeds instead of our true ISP speed.
  9. rcordorica

    rcordorica Network Guru Member

    I made a "Mini IPv6" build for MIPSR2. It's in the 4MB folder.
  10. gutsman7

    gutsman7 Networkin' Nut Member

    thnx alot rcordorica i will flash this as soon as i can, right now my network is in high demand so ill have to wait a bit, thnx
  11. gutsman7

    gutsman7 Networkin' Nut Member

    hey rcordorica i flashed your modified tomato yesterday night, i did notice a improvement in my gaming sessions my ping times were lower, my clients also game alot, also said there was a little improvement, i played a session of black ops on xbox and was the host for about 1 hr and a half thts never happend before. im liking this alot im gaming as im typing this thnxz.i did also notice tht while my network was bogged up the web pages still loaded at acceptable speed.:)
  12. gutsman7

    gutsman7 Networkin' Nut Member

    rcordorica i was wondering if u lower the values even more what might be the results? id be willing to try it out.
  13. rcordorica

    rcordorica Network Guru Member

    Well I built that with 2 packet limits. What scheduler are you using? You might want to use pfifo if you want the best gaming latency. Although SFQ might work better for web page loading.

    What's your connection speed?

    I can build one with 1 packet limits, and you can set your txqueuelen to 1. It might actually help, but it could also cause lots of packets to drop (which could be a good thing if you care about latency).

    Check out your QoS stats. Log into the router (ssh/telnet) and run tc -s qdisc. See how many packets got dropped in your gaming priority.
  14. rcordorica

    rcordorica Network Guru Member

    In reply to your edit, the standard MIPSR2 builds should work, located here:
  15. gutsman7

    gutsman7 Networkin' Nut Member

    hey rcordorica im using sfq scheduler, my speeds are 6mb down 700kbps up, i most certantly care mostly about latency i dont mind if heaps of packets are dropd for tht exchange.
  16. gutsman7

    gutsman7 Networkin' Nut Member

    im currently running windows cant telnet to router or ssh. im downloading linux mint right now for tht:)
  17. rcordorica

    rcordorica Network Guru Member

    Windows can telnet, but you have to enable it in the optional Windows features. You can also download a ssh client, like Putty.

    I use linux too though.
  18. KyleS

    KyleS LI Guru Member

    Went from this

    to this

    Interesting concept (Browsing certainly seemed to improve), for gaming/VOIP this isn't really the best solution unless if I'm completely missing something.
  19. rcordorica

    rcordorica Network Guru Member

    Your ping and jitter went slightly down. That's good.

    Your packet loss though is bad at 6% for this particular test.

    I just went to, and I ran Wireshark to capture the traffic it sends during the packet loss test. It appears to use port 8080 for the test.

    That's a non-standard VOIP port. I bet if you prioritized that port, the test would come back even better.

    The point is... you SHOULD see packet loss for ports that aren't prioritized, that way your latency for prioritized ports doesn't get too bad.
  20. gutsman7

    gutsman7 Networkin' Nut Member

    it says 10146 droped i think thts the one right im not sure
  21. rcordorica

    rcordorica Network Guru Member

    It should be ordered the same as your QoS list in Tomato. You really don't want too many packets dropping in your gaming class. But your "default" or p2p classes should have dropped packets.

    I have DNS, and gaming prioritized at the top. They get 100% bandwidth. For web browsing I have 80%, and p2p, 65%.
  22. gutsman7

    gutsman7 Networkin' Nut Member

    yeah i got my torrents way low im sure those packets where getting droped big time
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice