1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Critical DSCP bug Affecting WiFi Download Speeds on Comcast Cisco E-Series and Others.

Discussion in 'Tomato Firmware' started by lightsword, Sep 10, 2013.

  1. lightsword

    lightsword Serious Server Member

    There is a critical bug that is likely crippling thousands of routers and pretty much only comcast has this issue. It is an edge case interaction caused by bad DSCP information coming from Comcast and poor condition handling by the WMM driver on certain linksys/cisco eseries routers and possibly others as well. This issue can appear on both stock and modded routers. Comcasts network is misconfigured and has been for years and has likely caused many people to needlessly replace fully functional routers, IPv6 speeds always worked for me because DSCP was configured correctly for that but it is incorrect for IPv4. Since the only packets with this bad information are the ones that are being downloaded upload is not affected nearly as badly.

    This is the fix I made and used on an e2000 and e3000 running shibby 112:

    put this in init scripts section:
    insmod xt_DSCP.ko

    and this in wan up scripts:
    iptables -t mangle -A PREROUTING -i vlan2 -j DSCP —set-dscp 0

    these may need to be modified slightly depending on configuration.

    This corrects the bad DSCP header information as soon as a packet hits the WAN interface on a router.

    DSCP as received from Comcast's network is 0x08 this is the lowest priority possible which WMM interpertes as being unimportant to transmit quickly.

    The scripts listed changes the DSCP on all packets to 0x00 which essentially means unclassified which WMM handles properly. This is how virtually every other ISP uses DSCP and is why the issues is specific to Comcast.

    I have confirmed this issue on connections in both Chicago and Boulder, CO and it is likely to be present on Comcast's entire network.

    Cisco appears to have at some point released new WMM drivers for stock firmware that largely resolve the issue on current stock firmware, otherwise implementing my temp fix as a GUI option would probably be a good idea for the affected routers. The ability to assign DSCP to all traffic on chosen interfaces also might be a good thing to add anyways.
     
    tw39515 and koitsu like this.
  2. Marcel Tunks

    Marcel Tunks Networkin' Nut Member

    People have been reporting the WMM download problems with Comcast for years, but this is the first time I've seen someone come up with a solution. Great work!
     
  3. lancethepants

    lancethepants Network Guru Member

    I ran wireshark to observe the vlan2 interface, and can confirm in my neck of the woods that DSCP is set to 0x08.

    Here is the iptables rule, since the code above isn't nicely copy/pastable.
    Code:
    iptables -t mangle -A PREROUTING -i vlan2 -j DSCP --set-dscp 0
    
    http://ber.gd/post/23025893856/comcast-traffic-prioritization
    Interesting post on the subject.

    What sort of improvements should one expect then?
    The article above show how xfinity services are prioritized over other stuff like netflix.
    Should one then expect to have a better netflix streaming experience using this fix?
     
  4. Trademark

    Trademark Network Guru Member

    Can you please explain how to do this? I would like to test this for myself and make sure I have the right vlan and DSCP set for both my home (E4200,E3000) and business (WRT54G-TM) routers. Thanks!
     
    Last edited: Sep 11, 2013
  5. lightsword

    lightsword Serious Server Member

    Comcast's network doesn't really prioritize traffic for standard internet services, however they are likely using DSCP for internal routing possibly. They may have forgotten to strip DSCP when it reaches the network edge. If your own network doesn't do anything with DSCP this won't make a difference, however it can affect any device/devices that takes DSCP into account when routing which includes nearly every wireless N router to some degree since WMM is a required component, the issue is much more noticeable in these linksys/cisco devices due to how the driver handles WMM but it can affect any of them to some degree.
    The "vlan2" is simply what the wan interface on your main router is, you can find it using the ifconfig command(interface that has the public IP address). Note, you only should use this fix on the router that is connected to the modem, it will not help secondary routers since all problematic packets will have already been fixed as soon as they hit the WAN interface on the primary router if the primary is running this fix.
     
    Trademark likes this.
  6. Trademark

    Trademark Network Guru Member

    Just what I needed. Much appreciated!
     
  7. koitsu

    koitsu Network Guru Member

    Alternately if you aren't sure what interface is for your WAN, just rely on the NVRAM variable wan_iface to determine that for you, i.e. the command sequence becomes:

    Code:
    iptables -t mangle -A PREROUTING -i `nvram get wan_iface` -j DSCP --set-dscp 0
    
    If you get an error along the lines of "No such chain/rule", then you forgot to run either insmod xt_DSCP.ko or modprobe xt_DSCP (which you run is your choice; they do the same thing and if issued repeatedly have no ill effects) prior to the iptables command. The iptables command, however, should only be run one once -- repeated runs will append/add unnecessary rules to your PREROUTING chain in the mangle table.

    You can verify the command worked by looking at iptables -t mangle -L PREROUTING -n -v --line-numbers and seeing something like this:

    Code:
    Chain PREROUTING (policy ACCEPT 685K packets, 347M bytes)
    num  pkts bytes target  prot opt in  out  source  destination
    1  378K  302M DSCP  all  --  vlan2  *  0.0.0.0/0  0.0.0.0/0  DSCP set 0x00
    
    The input interface (vlan2 above) may vary depending on people's routers/configurations as the OP states.

    If you're reading this thread and are already tinkering with/have your own mangle table rules, then it's up to you to figure out/decide where the above rule "fits in" with your other rules.

    Finally, for those wanting to read up on the technical aspects of this QoS-related field in IPv4 and IPv6 (thread only pertains to IPv4), please see below. Warning: highly technical.

    http://en.wikipedia.org/wiki/Differentiated_services_code_point
    http://robert.penz.name/654/howto-to-quick-test-a-dscp-based-qos-system/
    http://www.tucny.com/Home/dscp-tos
    http://www.cisco.com/en/US/tech/tk543/tk757/technologies_tech_note09186a00800949f2.shtml

    A field value of 0x08 indicates a DSCP/PHB class of CS1 (precedence 1). How this gets used on the Comcast side is unknown, but I do have direct contact ability with folks who work in Comcast's IP networking group who I'm sure could comment further if this was discussed somewhere other than a public forum (i.e. Email).
     
    Last edited: Sep 11, 2013
  8. Trademark

    Trademark Network Guru Member

    Is it necessary to reboot the router after entering these commands in the "Init" and "WAN Up" scripts? I ask because after running iptables -t mangle -L PREROUTING -n -v --line-numbers I get this:

    Code:
    Chain PREROUTING (policy ACCEPT 14M packets, 12G bytes)
    num  pkts  bytes  target  prot  opt  in  out  source  destination
     
  9. lightsword

    lightsword Serious Server Member

    vlan2 is just my wan interface I'm pretty sure at least seemed to be for default shibby 112 on e2000 and e3000 routers, I just used ifconfig to find the interface that was getting an external IP address, btw should using
    `nvram get wan_iface` always get you the wan interface in every custom firmware varient? even dd-wrt or openwrt? from what I was seeing online there may have been a way to do the modprobe from iptables directly but I never got it to work, is that possible? This is what it said:
    If these are in the scripts section you will need to reboot, if you want to just test and see if this fixes the problem simply execute them as commands(generally a good idea to do this first so you know if there are any errors).

    FYI I have contacted Comcast about this and they are definitely looking into this although the more that contact them the faster they might fix it on their side. Their sysadmins seemed surprised this was going on so I'm fairly sure it wasn't intentional.
     
  10. Trademark

    Trademark Network Guru Member

    Shibby 112 on my E4200 is the same.

    Nice, I like this command even more!

    I tend to set it and forget it on my routers, but love to tinker. Never had a need to put anything in the Scripts section, so this is good info.

    Not surprised, but hope they actually do fix it. Not holding my breath based on the bandwidth competition of streaming services. Only time will tell...
     
  11. Trademark

    Trademark Network Guru Member

    All is good now:
    Code:
    Chain PREROUTING (policy ACCEPT 510 packets, 115K bytes)
    num   pkts  bytes  target  prot opt  in      out    source       destination        
    1      87   24434   DSCP   all   --  vlan2   *      0.0.0.0/0    0.0.0.0/0     DSCP set 0x00
    
     
  12. lightsword

    lightsword Serious Server Member

    Have your WiFi internet speedtest's improved at all? This is probably still a good thing to do because otherwise local traffic would have a higher priority than internet traffic so if you internal wifi network is at a high-utilization rate the more important internet traffic would be slowed down a lot.
     
  13. Trademark

    Trademark Network Guru Member

    To be honest, my speed tests always exceeded my plans rated speeds both wired (Gigabit) and wireless (5GHz N). I have Comcast 50/10 service and regularly get ~57/~11, but surprisingly, when I switched over to my 2.4GHz channel (Mixed B,G,N), I got the same ~57/~11 results! I have never seen these results on the 2.4 channel as it was always ~30/~11. Although, now that I think about it, I haven't thoroughly tested 2.4 since I swapped out my E3000 (Toastman 1.28.7483) for the E4200 (Shibby 112) recently. Could be that or the patch. I'll have to check this out.
     
  14. lightsword

    lightsword Serious Server Member

    I'm trying to track down what linksys changed in their source code. Your driver may not be one of the ones that responds badly to the comcast packets.
     
  15. koitsu

    koitsu Network Guru Member

    OpenWRT documents all their necessary variables/etc., and have moved many of them out of NVRAM to JFFS-based config files instead. If you want to support this for OpenWRT, you need to talk to the OpenWRT folks or read their docs.

    The same goes for DD-WRT, although they rely heavily on NVRAM like Tomato. The NVRAM variable may be named something different however. Ask them. Just remember respectfully that this is a Tomato-focused forum, not OpenWRT or DD-WRT.

    As for the iptables command you mentioned, re: --modprobe: please read everything I have written below.

    You do not want to mess with --modprobe, because the built-in command used for loading modules already works correctly. That command just overrides it, and you don't want that given the below circumstances.

    There are also netfilter extension modules (which is a type of kernel module but differs in how it's used and how it fits into netfilter/iptables) usable via -m, but you have to know what exact string to give it and it varies per module. Let me expand: there are two actual DSCP-related modules available:

    1. /lib/modules/2.6.22.19/kernel/net/netfilter/xt_DSCP.ko (which is what you're insmod/modprobe'ing manually) -- the description of this module is "x_tables DSCP modification module" (note the word modification). It provides the --set-dscp iptables command as well as the DSCP chain name (which is what you're -j'ing to), and is specific to the mangle table (because it modifies packet headers, and mangle is where that happens).

    2. /lib/modules/2.6.22.19/kernel/net/netfilter/xt_dscp.ko (note lowercase) -- the description of this module is "x_tables DSCP matching module" (note the word matching) and provides the --dscp and --dscp-class commands. It does not provide --set-dscp nor does it provide the DSCP chain name. The --dscp and --dscp-class commands are described as follows:

    Code:
    DSCP match v1.3.8 options
    [!] --dscp value  Match DSCP codepoint with numerical value
      This value can be in decimal (ex: 32)
      or in hex (ex: 0x20)
    [!] --dscp-class name  Match the DiffServ class. This value may
      be any of the BE,EF, AFxx or CSx classes
    
      These two options are mutually exclusive !
    
    You can access/use xt_dscp.ko (lowercase) by using -m dscp in your iptables command, however it will not introduce the DSCP chain nor the --set-dscp command, as I said above. You cannot use -m DSCP or -m xt_DSCP or any other kind of command to "trick" it. I have experimented excessively with these two modules at this point and that's that as I see it.

    The bottom line is just to run modprobe xt_DSCP as mentioned above (which will load xt_DSCP.ko into kernel space and affect netfilter/iptables as a whole), and then do your iptables commands as needed. Repeated runs of modprobe xt_DSCP will not harm the system (only one copy of the module will be ever loaded).

    The author of these modules is the same individual: Harald Welte <laforge@netfilter.org>. You're welcome to contact him if you wish to hash out any details.

    Edit: I should note the versions of xt_dscp and xt_DSCP I've linked to above are actually different (possibly newer?) than what we use in Tomato. I can tell because the Description strings differ slightly between what we have and what is on that site, and the code differs as well.

    What's used in Tomato (at least Toastman's RT-N branch):

    * http://repo.or.cz/w/tomato.git/blob...rc-rt/linux/linux-2.6/net/netfilter/xt_DSCP.c
    * http://repo.or.cz/w/tomato.git/blob...rc-rt/linux/linux-2.6/net/netfilter/xt_dscp.c
     
    Last edited: Sep 11, 2013
  16. lightsword

    lightsword Serious Server Member

  17. jerrm

    jerrm Network Guru Member

    If the change is in the driver itself, you probably won't see it. The Broadcom drivers are binaries, no source provided. If it is a setting or other non-driver code change, you might have some luck.
     
    koitsu likes this.
  18. lightsword

    lightsword Serious Server Member

    Are you sure about that? I thought they were open sourced a few years ago for the most part. Did you check the source code I linked?
     
  19. koitsu

    koitsu Network Guru Member

    The Broadcom wireless driver in Linux consists of two pieces:

    1. A "shim" interface/ABI for which source code is available -- all this does is act as a medium between the Linux kernel's networking layer (including the 802.11 MAC layer) and the binary blob driver, referring to symbol names blindly. I don't remember the name of this file but I can go digging if needed -- someone else here probably remembers the name,

    2. A binary blob driver (kernel module wl.ko) which contains the actual proprietary closed-source code written by Broadcom, and may also contain firmware microcode (since on some wireless NICs you actually have to upload code that runs on the NIC itself before you can use it). They have no plans on making this available to the public, AFAIK, ever. Broadcom is one of those excessively-IP-and-lawyer-driven companies that does stuff like this, and you're legally not allowed to reverse-engineer it (the driver).

    For the firmware to work for third parties, i.e. router vendors such as Asus, the vendors have a closed-door relationship with Broadcom. Broadcom gives them the shim code and the binary blob driver that is "supposed" to be compatible with the version of the Linux kernel they're using (for Tomato that's 2.6.22 patchlevel 19) and Asus crosses their fingers and hopes for the best. RMerlin can probably talk more about this, as he's actually chatted with Asus in the past about their stuff.

    The binary blob driver also varies per router model, if I remember right (jerrm or RMerlin can correct me here if I'm wrong). It's also very difficult to determine if there are actual changes to the driver itself; MD5/SHA1 checksums sometimes vary while version numbers (shown in dmesg, or shown via wl ver) do not change.

    The situation is very similar with the Ethernet switch driver (et.ko).
     
  20. lightsword

    lightsword Serious Server Member

    This is interesting, because as far as I can tell the e3000 wl module is fully open source, however the e2000 does appear to be in binary format. et.ko does appear to be binary only on everything I have seen so far, I'm still looking though.
    Edit: I am wrong about this, both have some binary only components.
     
    Last edited: Oct 29, 2013
  21. koitsu

    koitsu Network Guru Member

    If Broadcom changed their stance with certain models of routers, that wouldn't surprise me either. It doesn't change the fact that the situation varies per model though (in other words, you have your work cut out for you, sadly). :)
     
  22. lightsword

    lightsword Serious Server Member

    Well, I assume whatever was fixed was done largely in the same way for every one of these linksys routers.
     
  23. lightsword

    lightsword Serious Server Member

    Ok, looks like there were binary blobs for the most part, but I'm pretty sure I've isolated the changes by comparing the source from before and after the change, I think the drivers Shibby and everyone else are using are outdated, I have the updated drivers but I don't really know how to compile tomato.
     
  24. Almaz

    Almaz Serious Server Member

    I tried with WRT54G Shibby Tomato Firmware 1.28.0005 108 ND VPN but xt_DSCP.ko file is missing but it worked fine using E3000 Shibby 112 firmware
     
  25. lightsword

    lightsword Serious Server Member

    FYI this is likely a problem that will only be noticeable on wireless N capable routers since it is a WMM issue partially.
     
  26. Trademark

    Trademark Network Guru Member

    Try using insmod ipt_DSCP on that model. I had the same problem on my WRT54G-TM. Also used vlan1 on the WAN, but you should check this to be sure.
     
    Last edited: Sep 15, 2013
  27. lightsword

    lightsword Serious Server Member

    Is this actually helping people with WRT54G's? I was thinking the issue is WMM related and isn't that mainly wireless N?
     
  28. Trademark

    Trademark Network Guru Member

    Not sure, but on mine WMM is an option and I have it enabled. Wireless speed does seem more stable and consistent though, and it certainly seemed to improve things on the 2.4GHz channel on other models.
     
  29. Daryl Martin

    Daryl Martin Serious Server Member

    I'm on Comcast with an E2500 (running Tvlz's build 2), and this fixed the problem for me as well. Thanks!
     
  30. Mangix

    Mangix Networkin' Nut Member

    This fixed a problem I had with a router running dd-wrt where constant upload to Dropbox or whereever would kill my connection to the router.
     
  31. lightsword

    lightsword Serious Server Member

    I've recompiled shibby tomato with the newer fixed driver(rt-n) for the e2000 and it seems to have fixed the issue, however I don't have Comcast anymore so I was only able to test by having the router WAN interface set DSCP to 8 itself. I just recompiled from the tomato-shibby-RT-AC head and from the src-rt folder with the e2000 option. http://sourceforge.net/projects/rou...K-1.28.--defMIPSR2-Max-newdriver.bin/download
    I haven't done much stability testing but so far everything seems fine, the older driver needs to be deprecated since it is causing these issues.
     
  32. bripab007

    bripab007 Network Guru Member

    I've got Comcast's 50/10 down/up service, and I, too, never got greater than ~30Mbps down on any wireless device no matter my wi-fi configuration. I did a packet trace and confirmed DSCP headers other than 0x00, so ran the two scripts on my E3000 running v1.28.7500 MIPSR2Toastman-RT K26 USB VPN, and lo and behold--I'm hitting ~57Mbps down on my iPhone and iPad!
     
  33. Ephemeral

    Ephemeral Reformed Router Member

    I came across this thread when I was looking into buying a new router and someone mentioned this bug in a review.

    Question: If I flash my E2000 router with this fixed driver Tomato version, will I still be able to update to the latest version of Shibby released November 21st without affecting the driver? (I'm a novice when it comes this stuff.)

    I'm also curious if anyone else has installed this and had it work well for them.
     
  34. jerrm

    jerrm Network Guru Member

    Firmware updates are an all or nothing affair. Nothing other than user settings are preserved - and most will say you're better off deleting those and reconfiguring,
     
  35. Ephemeral

    Ephemeral Reformed Router Member

    Thank you for the response! I guess I'll use the latest version of Shibby with the fix detailed in the first post.
     
  36. bripab007

    bripab007 Network Guru Member

    If the fix is all you want, it takes ~30 seconds to enter and apply the scripts to the router vs. the amount of time it takes to reset NVRAM, flash firmware, reset NVRAM, reset all your router settings, etc.
     
  37. Ephemeral

    Ephemeral Reformed Router Member

    True, but I've been meaning to try a different flavor of Tomato anyway since I've been using the original all this time.
     
  38. Scottmsu

    Scottmsu Reformed Router Member

    It seems like using this fix on my e3200 causes random lag spikes. When I remove the fix and revert back to G speeds it goes back to normal. Is there a way around this?

    edit: it is not really noticeable when browsing the web, but when playing a game, it's obvious.
     
  39. koitsu

    koitsu Network Guru Member

    The only other choice is to disable WMM under Advanced / Wireless and cease use of the iptables rule. I'd recommend rebooting the router (via GUI) after saving and doing that; the wireless driver sometimes acts wonky after changing some settings in that menu.
     
  40. bripab007

    bripab007 Network Guru Member

    The commands appear to be overwritten back to defaults after a period of time. They're not persistent.
     
  41. PetervdM

    PetervdM Network Guru Member

    are you sure you're not running out of nvram memory?
     
  42. bripab007

    bripab007 Network Guru Member

    I don't believe so. At the moment I've got ~35k/60k free, and when the dcsp script stops working, nothing else on the router stops working or freezes up as far as I know.
     
  43. jerrm

    jerrm Network Guru Member

    The rule stopped working or is no longer there? What is the result of "iptables -vnL -t mangle" when it stops working? Where are you saving the commands?

    Various things will cause tomato to reset/reload the firewall rules. Wanup is probably the wrong place to add the rule. The only way to make sure your own iptables rule(s) will be re-loaded for any firewall reset is to place it in the admin firewall script area or in a .fire autorun script.
     
    Last edited: Jan 18, 2014
  44. bripab007

    bripab007 Network Guru Member

    I'm not sure how I'd decipher whether the rule stopped working or was no longer there. I can tell you the output of that command when it is working is like this:
    Chain PREROUTING (policy ACCEPT 3878K packets, 3694M bytes)
    num pkts bytes target prot opt in out source destination
    1 2688K 3673M DSCP all -- vlan2 * 0.0.0.0/0 0.0.0.0/0 DSCP set 0x00
    2 2622K 3584M DSCP all -- vlan2 * 0.0.0.0/0 0.0.0.0/0 DSCP set 0x00

    When it stops working, running that same command gives an output more along the lines of this:
    Chain PREROUTING (policy ACCEPT 3878K packets, 3694M bytes)
    num pkts bytes target prot opt in out source destination
    1 0K 0M DSCP all -- vlan2 * 0.0.0.0/0 0.0.0.0/0
    2 0k 0M DSCP all -- vlan2 * 0.0.0.0/0 0.0.0.0/0 (approximated from memory, as I didn't copy it down)

    That is to say, it appears to stop setting the DSCP IP header to 0 on incoming WAN packets. Rerunning the commands appears to re-enable it. At the moment, it's been working properly for the past ~13hrs or so.
     
  45. koitsu

    koitsu Network Guru Member

    It looks to me like there is something else touching/screwing with your iptables rules behind-the-scenes causing you this problem. There is definitely something about your setup/configuration which you haven't disclosed.

    These commands ***are*** persistent if placed appropriately into Scripts/Firewall, otherwise they are lost at reboot. If you have something else going on under the hood (something else in Scripts, etc.) that tinkers, then I am not surprised in the least. I say they are persistent because they have remained working for many people (including myself) without issues. However people with more customised or complex environments may need to adjust things appropriately, and respectfully it is not my responsibility to cover every usage case. I wouldn't be surprised if the issue turned out to be someone using VPNs or having their WAN go up and down or the like.
     
  46. bripab007

    bripab007 Network Guru Member

    The scripts are still working fine at the moment, so we're going on ~36hrs now.

    I do have some firewall scripts that are commented out it appears. I don't even recall putting these in, so I'm not sure if they're default Tomato or Toastman scripts or ones I was playing with in the past and just forgot:

    #Restrict number of TCP connections per user #iptables -t nat -I PREROUTING -p tcp --syn -m iprange --src-range 192.168.1.50-192.168.1.250 -m connlimit --connlimit-above 100 -j DROP
    #Restrict number of non-TCP connections per user #iptables -t nat -I PREROUTING -p ! tcp -m iprange --src-range 192.168.1.50-192.168.1.250 -m connlimit --connlimit-above 50 -j DROP
    #Restrict number of simltaneous SMTP connections (from mailer viruses)#iptables -t nat -I PREROUTING -p tcp --dport 25 -m connlimit --connlimit-above 5 -j DROP
     
  47. koitsu

    koitsu Network Guru Member

    Those are stock defaults and as you noticed are commented out (thus take no effect).
     
  48. bripab007

    bripab007 Network Guru Member

    Just reporting back that it's still working. I wish I knew what caused it to initially break after a day or so. Maybe WAN IP address change with new lease? But if my iptables script is in WAN UP, wouldn't it reapply that after getting a new WAN IP?
     
  49. atlasmugged

    atlasmugged Reformed Router Member

    Is this bug driver dependent, e.g. is the Broadcom wireless driver in the K26RT-N branch of shibby's Tomato build (5.110.27.20012) affected the same as the driver in the K26 branch?
     
  50. koitsu

    koitsu Network Guru Member

    Regardless of where the script is, what's shown in http://www.linksysinfo.org/index.ph...-cisco-e-series-and-others.69006/#post-240296 indicates that your rules are getting flushed (i.e. deleted and reapplied, though incorrectly in some way).

    I can't think of a reason for rules to be flushed like this unless you're messing around in the GUI, sticking scripts into WAN Up (rather than Firewall), or have some other customisations going on. You didn't provide a thorough description of all the configuration settings you change from defaults, or other things about your router, so those trying to help are stuck guessing / grasping at straws.

    I also see that in your initial example you have two DSCP setting lines (two rules), which is amusing in itself. This implies something is amiss as well.

    If this was a universal problem I would expect other people would be complaining.

    Case in point: I have no such problems using tomato-K26USB-1.28.0503.5MIPSR2Toastman-RT-N-Ext.trx on my RT-N16:

    Code:
    root@gw:/tmp/home/root# uptime
    10:37:30 up 39 days, 17:11,  load average: 0.01, 0.00, 0.00
    
    root@gw:/tmp/home/root# iptables -t mangle -L PREROUTING -n -v
    Chain PREROUTING (policy ACCEPT 342M packets, 117G bytes)
    pkts bytes target  prot opt in  out  source  destination
    168M  68G DSCP  all  --  vlan2  *  0.0.0.0/0  0.0.0.0/0  DSCP set 0x00
    
     
  51. bripab007

    bripab007 Network Guru Member

    Define "messing around in the GUI". I mean, the GUI is what I primarily use to configure the router...

    As I said, the ip tables rule is in WAN UP, and the modprobe xt_DSCP is in Init.

    My changes from default on this router have always been fairly rudimentary. Enabling QOS, specific wifi and DHCP settings, obviously, DDNS client...no other scripts running.

    I may have inadvertently executed both modprobe xt_DSCP & insmod xt_DSCP.ko, perhaps explaining two rules currently running.

    I have not at any point expressed belief that this is a universal problem.

    It's still working fine at the moment, so I'll reboot and make sure there's only one active rule and that it continues to work.
     
  52. jerrm

    jerrm Network Guru Member

    The firewall section is the proper place for any iptables rule, not wanup.

    The firewall rules are called anytime tomato resets the firewall. A firewall reset will remove the rule. There are many scenarios where the firewall is reset, but wanup is not called.

    It is also possible there are scenarios wanup is called but the firewall is not reset - the fact you had two copies of the rule suggests there are.

    The wan interface can go up and down multiple times, change ips hourly, etc. The rule should not have to be re-added for any of those scenarios, only if the firewall is reset.
     
    Last edited: Jan 25, 2014
  53. koitsu

    koitsu Network Guru Member

    What @jerrm said is correct -- said rules need to go into Scripts / Firewall, not Scripts / WAN Up. Therein lies the source of at least one problem (bare minimum the doubling of DSCP set rules). I even covered this in my earlier post, quote: "The iptables command, however, should only be run one once -- repeated runs will append/add unnecessary rules to your PREROUTING chain in the mangle table."

    You did follow directions as best they were provided though, @bripab007; the initial post does say to stick one in Init and the other in WAN Up, which is incorrect. Both the modprobe and the iptables command should end up in Firewall under most circumstances.

    There are exceptions to this situation, but those would only apply if someone was running a very customised environment (see other threads on this forum for all the crazy stuff people are doing with their routers, making a big mess :) ); possibly use of VLANs, or WAN connections with PPPoE or PPPoA could need some adjustment.
     
  54. Morac

    Morac Network Guru Member

    I've got the rule running and noticed a near doubling of my WiFi speeds to about 57 Mbps.
    The only downside is when downloading at that speeds on my E3000 running Toastman's firmware, the CPU usage spikes to around 90%, even over Ethernet. That's with downloads maxing at about 60 Mpbs with an IPV4 speed test. With An IPV6 speed test the CPU maxes out at about 40%.

    With Comcast supposedly doubling download speeds by March, I'm don't think the router will be able to handle changing the DSCP without maxing out the CPU.

    Edit:

    Actually I'm not sure the script is making that big a difference CPU-wise. I deleted the iptable rule using the -d parameter and re-ran a speed test and it was still pretty close to 90% CPU (85%). I didn't reboot so whatever the modprobe command did wasn't undid. I don't know if having the kernel driver loaded alone without the iptable rule would cause an increase in CPU usage.

    Anyway when I deleted the iptable rule I could still get 47 Mbps on my iPhone 5S when I tested close to my router. My speeds dropped off dramatically the farther away I got from my router. When I reapplied the iptable rule my speeds increased from 12 Mbps about 30 feet from my router to 40 Mbps (5.2 MHz 802.11n).
     
    Last edited: Jan 26, 2014
  55. koitsu

    koitsu Network Guru Member

    @Morac -- You can determine if the module injection is affecting performance by unloading it. Remove the DSCP-related iptables rules then simply rmmod xt_DSCP and re-do any tests. It's that simple.

    IMO, your issue sounds completely unrelated to the module. But the concerns over CPU usage (speaking on a general level) given what the mangle table does are justified. It's just another example of the fact that these routers were not designed for massively high throughput, since ~98% of the IP layer is pure software in the kernel. Not much can be done about that. These residential routers pack absolutely no where near the amount of CPU power as a desktop system. :)
     
  56. bripab007

    bripab007 Network Guru Member

    Thanks for all the tips and insight, folks. This does provide a noticeable speed benefit in many circumstances on my wireless devices. Pity it sounds like we'll need to look at stronger routers sometime in the near future, as I've otherwise quite liked the E3000.
     
  57. tw39515

    tw39515 Network Newbie Member

    Worked for me for comcast in Glaveston Tx on a e1200
     
  58. Morac

    Morac Network Guru Member

    My Internet went down today. I rebooted my modem and router, but modem was down (it gave router a dummy IP address). When it came back up, I had 3 PREROUTING DSCP rules, which obviously was wrong. I manually removed 2 of them. I was setting the rule in FIREWALL script, so apparently that ran 3 times without removing the iptable rules.
     
  59. koitsu

    koitsu Network Guru Member

    Doesn't surprise me. There are certain situations/conditions where certain parts of the built-in "services" (that's what it's called, and it's hard to explain without actually showing people source code) will re-run the Firewall scripting section.

    The workaround is simple: instead of using modprobe xt_DSCP + iptables -t mangle -A PREROUTING ..., use the below instead (which will only run the rule insertion if there isn't an existing rule already found):

    Code:
    if [ -z "$(iptables -n -t mangle -L PREROUTING | grep 'DSCP set 0x00')" ]; then
    modprobe xt_DSCP
    iptables -t mangle -A PREROUTING -i `nvram get wan_iface` -j DSCP --set-dscp 0
    fi
    
     
    Last edited: Apr 10, 2014
    HunterZ and Monk E. Boy like this.
  60. Morac

    Morac Network Guru Member

    Thanks.

    I'm assuming the fact that the router ran the modprobe statement multiple times won't hurt anything?
     
  61. koitsu

    koitsu Network Guru Member

    modprobe will only install/add the module if it isn't already loaded.
     
  62. LanceMoreland

    LanceMoreland Network Guru Member

    Applying this fix did improve speed test results on my E4200 v1.
     
  63. Victek

    Victek Network Guru Member

    Applied in tomato code, also fixes ipv6 to refuse anycast addresses for /128. Thanks to @tvlz for pointing it.

    http://goo.gl/FHUkrq
     
  64. Morac

    Morac Network Guru Member

    Just a FYI, applying the mod rule could potentially max out the CPU on older routers.
     
  65. Victek

    Victek Network Guru Member

    And if we don't do then Comcast users have problems... I tested and the sirq it's exactly the same..
     
  66. kthaddock

    kthaddock Network Guru Member

    Suggestion deleted.
     
    Last edited: May 10, 2014
  67. Victek

    Victek Network Guru Member

    tested and no sirq increase, I'm very interested to know Morac's evaluation.. a check box it's always a doubt question for beginners .. the switch it's in the code.
     
  68. Morac

    Morac Network Guru Member

    I don't know if it's that specifically or just Tomato in general, but I have the rule added and when running speed tests on the 100 Mbps speed tier my E3000's CPU usage shoots up over 90% (for ipv4 only for some reason, ipv6 uses much less CPU). Comcast' slowest download speeds are now 50 Mbps.
     
  69. jerrm

    jerrm Network Guru Member

    I'd vote for a checkbox, if just to keep noise out of the rules if not needed. Default it to being enabled, with some text explaining it.
     
  70. tvlz

    tvlz Addicted to LI Member

    A checkbox is fine with me, even if it's just used to test the effect of these changes on different routers
     
  71. Victek

    Victek Network Guru Member

    :) .. Since both of you are more than beginners and great know how, why you don't contribute with your code for the switch function?... help us! ;), I saw 11 clicks in the link ;) ... http://goo.gl/FHUkrq .. tomato is a teamwork!
     
  72. lancethepants

    lancethepants Network Guru Member

  73. Morac

    Morac Network Guru Member

  74. lancethepants

    lancethepants Network Guru Member

    Not sure how you came up with that conclusion.
    I compiled and tested it first. If you like to test, here it the link to the firmware I made.
    http://files.lancethepants.com/TomatoRAF Test/

    I made it for my RT-N16, but it is a very stripped version so it didn't take long to compile.

    It works correctly when booting, enabled or disabled. It also works correctly when being toggled on/off after boot.
     
  75. Morac

    Morac Network Guru Member

    By looking at what you changed. You added code to issue the mod change and add the iotable rule when the firewall processing runs when the box is checked. It does nothing if the user had the box checked and then unchecked it.
     
  76. tvlz

    tvlz Addicted to LI Member

    Thanks:)
     
  77. lancethepants

    lancethepants Network Guru Member

    Other than by hitting the 'Save' button it rebuilds the iptables rules. If in doubt, test, I have provided a sample.
     
  78. Victek

    Victek Network Guru Member

    correct, see my comment in your merge request and suggested code to use corner case easier... iptables are flushed. Could you test?, I put the condition before to save work, no need to follow when value not equal to 1. Thanks for your help ;)
     
  79. Hogan773

    Hogan773 Networkin' Nut Member

    I am behind on my firmware updates (3 years) and came across this thread as I was about to update to a recent victek or shibby release. . I am a comcast customer and have noticed relatively slow wifi speeds on my e3000.

    Question is have any of the newest releases applied a fix for this comcast issue or a check box that allows me to turn on the fix, or do I still have to figure out how to appl the script sshown above? Thanks
     
  80. Morac

    Morac Network Guru Member

    As far as I know you still need the script.
     
  81. lancethepants

    lancethepants Network Guru Member

    With the latest shibby and Raf they should both have the fix and automatically turned on.
    You can check whether it's enabled in Advanced -> Firewall.
     
  82. Morac

    Morac Network Guru Member

    Apparently Toastman's version, which is what I use, doesn't have the fix in it.
     
  83. Monk E. Boy

    Monk E. Boy Network Guru Member

    Unless I'm reading it wrong, koitsu's script (his 2nd to last post in this thread) should work even on systems with the fix implemented - it just won't trigger the script's payload. Just paste it into firewall tab under Administration->Scripts.
     
  84. Morac

    Morac Network Guru Member

    Yes it works. That's what I'm using.
     
  85. Toastman

    Toastman Super Moderator Staff Member Member

    I'll add the fix too, then we should all have the same fix, with Lance's GUI addition too.
     
    Morac likes this.
  86. Mangix

    Mangix Networkin' Nut Member

    I noticed something interesting on a comcast connection today. This is a screenshot from wireshark: https://i.imgur.com/gLphNja.png

    The DSCP bit seems to be set to 0x08 as well...

    As far as I know, the fix only applies to IPv4 packets since I did not see the problem there. Any way IPv6 could also get the fix?
     
  87. Morac

    Morac Network Guru Member

    When I tested on http://speedtest.comcast.net site which tests both IPv4 and IPv6, I never saw any slow downs on the IPv6 tests, so IPv6 might not need the fix.
     
  88. koitsu

    koitsu Network Guru Member

    Where was the capture done on? The router's WAN interface, or capturing traffic on the local workstation? It needs to be done on the router's WAN interface (tcpdump -p -i vlan2 ...).

    The screenshot is also intentionally omitting source and destination -- whether this is an inbound or outbound packet matters. In the future please don't post "snippet" screenshots like this, they're unhelpful in that form. For example, with xt_DSCP in use with --set-dscp 0x00, outbound packets will have a DSCP class of CS0 (Default), but inbound packets from Comcast (at least for me) have a value of 0x08 (CS1). I've attached a capture showing that fact (see IPv4 header, Differentiated Services Field, in the initial SYN (outbound) and the SYN+ACK response (inbound)). This capture was done on the WAN interface of the router. If I was to have done a capture on the workstation doing the HTTP request itself, the response packets from Comcast should (given the way the fix works) use CS0.

    1. iptables only processes IPv4 packets (ip6tables is for IPv6),
    2. The xt_dscp.ko netfilter module only is for IPv4 (there is no IPv6 version built),
    3. The DSCP rule (specifically --set-dscp 0) causes the value to be set to 0 (Class Selector 0).

    A DSCP value of 0x08 (%001000 in binary (I'll explain why I'm only showing 6 of the 8 bits in a moment)) would indicate CS1 (Class Selector 1), thus IP precedence 1 ("Priority").

    Of the raw byte, bits 0 and 1 are for ECN and aren't controlled here. So usually when adjusting the DS field, it's the upper 6 bits you're controlling. When you give a value of something like --set-dscp 12 (0x0C in hex, %00001100 in binary), the values don't actually map 1:1 to each raw bit of the full byte -- they actually only affect the upper 6 bits of the DSCP value. In other words, the bits are shifted left by 2. So a value of 12 decimal would actually become %001100xx in the DSCP byte, which would set Priority, Low Delay, Normal Throughput, and Normal Reliability. The DSCP class for this is called AF12.

    The Wireshark capture denotes that correctly, showing a value of 0x08 means %001000xx thus CS1.

    My general/strong opinion is that the wireless WMM implementation on some Linksys/Cisco routers is ultimately what's responsible for this problem. The workaround in question is to keep the WMM portion of the wireless driver from behaving like an idiot.

    Reference materials:

    * http://bogpeople.com/networking/dscp.shtml
    * http://en.wikipedia.org/wiki/Differentiated_services_code_point
    * http://www.linuxtopia.org/Linux_Firewall_iptables/x4172.html
    * http://tinyurl.com/dscpbug
     

    Attached Files:

    Last edited: Aug 14, 2014
  89. Mangix

    Mangix Networkin' Nut Member

    sorry about that. didn't want to leak IP addresses(not sure if needed with IPv6).

    the source was listed as Cisco-Linksys which is the router and the destination was evga.com

    the capture was done from a desktop PC connected through ethernet to the router. I could use tcpdump on the router but i'd need a binary first...

    It seems like comcast is writing the 0x08 value for IPv6 packets as well. This does not happen with my HE tunnel.

    the packet capture that you posted was an IPv4 one. I'm specifically talking about IPv6.
     
  90. koitsu

    koitsu Network Guru Member

    The DSCP value is 0x08 on IPv4 (and apparently IPv6) from Comcast. All the DSCP rule on TomatoUSB does (for IPv4, because there's no IPv6 module equivalent built right now) is "mangle" inbound packets (from Comcast to the router's WAN IP, before they get forwarded/NAT'd and shoved through the wireless driver and sent to the appropriate LAN IP), setting DSCP to 0x00. This works around whatever idiocy there is in the wireless drivers relating to WMM for specific models of routers.

    I think the core issue is within the wireless driver and only affects specific models of chips/routers. For example, I get the same throughput and performance out of my RT-N16 as well as RT-N66U, from wireless clients on my LAN (a couple laptops, a Nintendo 3DS XL, and a Motorola MotoG phone), both with and without the "DSCP fix". But the OP stated clearly it fixes something for him on Linksys E2000 and E3000 routers, which almost certainly have a completely different set of code (compared to RT-N16 and RT-N66U) being used within the wireless driver. The wireless driver is big and fat and monolithic and has tons of code written for many different chips and revisions of chips. It's closed-source so nobody knows what exactly is going on.

    It also could be be that the affected wireless drivers only are inspecting IPv4 packets and instead just "pass through" IPv6 (which in effect would mean the WMM isn't really used when IPv6 is involved).

    I have to point out, of course, that I'm sure some people have experienced "problems" that they then say the above fix "magically solves" -- but for all we know they implemented it in Scripts/Init and rebooted the router, which causes the entire wireless driver and hardware to be reset. Meaning: it's very possible just a simple reboot (no DSCP fix) would have alleviated their issues. Really it's one of those things where speed tests have to be done before and after with someone applying and removing the iptables rule in real-time.
     
  91. Morac

    Morac Network Guru Member

    I've done this with an iPhone and my E3000, adding and removing the iptable rule. The fix definitely makes a difference.

    Of note, the issue has supposedly been fixed in the current wireless drivers for Linksys routers. The drivers being used in TomatoUSB are older.
     
  92. Mangix

    Mangix Networkin' Nut Member

    I have no idea either. However, this fix also works for me on an atheros based router that runs dd-wrt. The only change to the rule needed is to change vlan2 to eth0.
     
  93. koitsu

    koitsu Network Guru Member

    Okay, but IPv6 doesn't appear affected? Do you have a way to turn off the entire IPv4 stack on your iPhone, or exclusively use IPv6 somehow (maybe even some kind of firewall rule that just blocks all IPv4 packets to a specific client?), and run a speed test/whatever it is you ran that confirmed the DSCP fix improved things for you? I mention this because there's a lot online that "falls back" to IPv4 in certain circumstances, so you'd really have to make sure that all your tests are truly using IPv6 (through packet captures, etc.).

    If pure IPv6 tests are consistent in behaviour/speed/etc., then to me that means the wireless driver is not inspecting IPv6 DSCP fields, hence IPv6 is immune to the problem.
     
  94. Morac

    Morac Network Guru Member

    Actually for the IPv6 test I used a Windows 7 laptop and the speedtest.comcast.net site since that site does two speed tests: to an ipv4 address and a ipv6 address.

    I can try again at some point in the future.
     
  95. Morac

    Morac Network Guru Member

    I had a chance to retest 802.11n speeds with the DSCP fix on and off. I had it on, ran the speed test on my laptop (plugged into AC) over WiFi-N and then deleted the iptables rule (no other changes) and ran it again. Here's the results.

    With fix:

    [​IMG]

    Without fix:

    [​IMG]


    As you can see the IPv4 speeds dropped by about 75%, while the IPv6 speed actually went up slightly (within margin of error), so the DSCP fix doesn't appear to be needed for IPv6. At least not on a Linksys E3000.

    That said, as far as I'm aware the DSCP fix simply makes it so that the 802.11n traffic isn't categorized as low priority. It's possible that there's so little IPv6 traffic that low priority IPv6 isn't throttled.
     
  96. koitsu

    koitsu Network Guru Member

    Wow, quite an impact. That wireless driver bug sure is awful then. :(

    I'd be equally as interested in seeing the DSCP fix turned off (iptables -t mangle rule removed, but I guess now you can just do it through the GUI) but WMM disabled under Advanced / Wireless.

    I think regarding IPv6, it's that the wireless driver doesn't bother to analyse IPv6 packets and tie IPv6 DSCP into WMM, which is why it's not affected.
     
  97. Morac

    Morac Network Guru Member

    If I get a chance I can try that. As far as I'm aware the bug only affects the 5.4 GHz band. I can actually get faster speeds with the fix turned off if I'm within about 7 feet of my router, but speeds drop dramatically the farther I get away. Before I found out about the fix, I used to just think 5.4 Ghz had horrible range, but that wasn't the case. I ran the tests above about 25 to 30 feet away from the router.
     
  98. bripab007

    bripab007 Network Guru Member

    I, too, can confirm this Comcast and WMM related issue affects me quite dramatically on my E3000 router. Without the fix I'm locked in to ~29Mbps downstream for any of my iDevices. Implementing the fix bumps it up to ~55Mbps.
     
  99. Mangix

    Mangix Networkin' Nut Member

    how did you apply the fix to IPv6 packets? the needed module shouldn't be built for tomato? or maybe ip6tables needs to be used.
     
  100. Morac

    Morac Network Guru Member

    I didn't.

    Hence my comment that apparently the fix isn't needed for IPv6.
     
    koitsu likes this.

Share This Page