1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

QoS development thread

Discussion in 'Tomato Firmware' started by Porter, Mar 26, 2010.

  1. Porter

    Porter LI Guru Member

    Hi everyone,

    some of you might have read this rather large post by me: http://www.linksysinfo.org/forums/showpost.php?p=361164&postcount=209

    Since the thread I posted this in was rather about using QoS than the development of QoS features I felt the need for a new thread.

    I thought there should be a thread where all the information is gathered and the newly developed source can be tested and made available to a bigger audience.

    I'm a real newbie when it comes to writing code and testing it, but I've found that some people around here use a management system for it, which can be found here: http://repo.or.cz/w/tomato.git

    Here's the thread that started it which is also about using Git: http://www.linksysinfo.org/forums/showthread.php?t=59959

    Or for a direct start: http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html

    The essentials on QoS with Linux can be found here:


    Furthermore I already found out that rewriting the UI probably isn't half the part that needs to be done. One central part seems to be /sbin/rc which isn't a simple shell script but written in C.

    If there's a guide to "hacking" your own Tomato I would appreciate a hint to it, because it takes a lot of time to figure it out all by yourself and I don't even understand C. It's easier for me to figure out what tc and iptables need to do rather than learn C from scratch.

    What's the agenda so far?

    Here are some suggestions:

    - get real inbound QoS working
    - check if icmp classification works
    - get inbound pie charts, probably even a real-time traffic graph
    - what else?

    That's all I can think of right now.
  2. karogyoker

    karogyoker Addicted to LI Member

    - rate/limit for inbound too
    - Show transfer speed (down/up) and trandsferred bytes (down/up) for each connection in the details menu.
    - at bandwith statistics show transferred total bytes for MAC addresses which are on the LAN
    - at real time chart show transfer rates for each host in the LAN, not just the total
  3. rhester72

    rhester72 Network Guru Member

    If it helps at all, I was able to dig up this old (late 2005) script I was working on to implement IMQ on OpenWRT. It was not the latest version (there were a number of subsequent revisions that were unfortunately lost), but at least it gives a possible framework for research:

    # Traffic shaping script
    # Need to implement DirectPlay 7
    # Current mapping
    # ---------------
    # Very high  1:10  ICMP (ping), SYN/RST, ACK, UDP, NTP
    # High       1:20  Cisco VPN, HTTP client, NNTP, Skype, *DirectPlay 7, SSH (short burst), SSH (small packet),
    #                  SMTPS/POP3S, VNC/RDP, IM, FTP client
    # Medium     1:30  SSH (SCP/long burst/large packet), HTTP server  <DEFAULT>
    # Low        1:40  FTP server
    # Very low   1:50  p2p, tor
    . /etc/functions.sh
    # Set the following values to somewhat less than your actual download and uplink speed in kilobits
    # Set the following value to the desired outbound MTU
    # Set the following value to the desired outbound queue length
    # WAN device
    WANDEV=$(nvram get wan_ifname)
    # ip binary location
    # tc binary location
    # iptables binary location
    # modules list
    #MODULES="cls_fw ipt_helper ipt_layer7 ipt_length ipt_limit ipt_IMQ sch_htb sch_sfq"
    MODULES="cls_fw cls_u32 ipt_helper ipt_layer7 ipt_length ipt_limit sch_htb sch_ingress sch_sfq"
    # Provide status upon request
    if [ "$1" = "qdisc" ]; then
      $TC -s qdisc ls dev $WANDEV
    #if [ "$1" = "dlq" ]; then
    #  $TC -s qdisc ls dev imq0
    #  exit
    if [ "$1" = "class" ]; then
      $TC -s class ls dev $WANDEV
    #if [ "$1" = "dlc" ]; then
    #  $TC -s class ls dev imq0
    #  exit
    if [ "$1" = "iptables" ]; then
      iptables -t mangle -L UL -nv | grep -v RETURN
    #if [ "$1" = "dlt" ]; then
    #  iptables -t mangle -L DL -nv | grep -v RETURN | grep -v IMQ
    #  exit
    if [ "$1" != "stop" ] && [ "$1" != "start" ] || [ "$1" = "" ]; then
      echo Bad args
    # Bring imq0 interface down, hide errors
    #$IP link set imq0 down                                2> /dev/null > /dev/null
    # Delete existing downlink and uplink qdiscs, hide errors
    #$TC qdisc del dev imq0 root                           2> /dev/null > /dev/null
    $TC qdisc del dev $WANDEV ingress                     2> /dev/null > /dev/null
    $TC qdisc del dev $WANDEV root                        2> /dev/null > /dev/null
    # Clean iptables rules and delete chains, hide errors
    #$IPTABLES -t mangle -D PREROUTING  -i $WANDEV  -j DL  2> /dev/null > /dev/null
    #$IPTABLES -t mangle -D DL                      -j IMQ 2> /dev/null > /dev/null
    #$IPTABLES -t mangle -F DL                             2> /dev/null > /dev/null
    #$IPTABLES -t mangle -X DL                             2> /dev/null > /dev/null
    $IPTABLES -t mangle -D POSTROUTING -o $WANDEV  -j UL  2> /dev/null > /dev/null
    $IPTABLES -t mangle -F UL                             2> /dev/null > /dev/null
    $IPTABLES -t mangle -X UL                             2> /dev/null > /dev/null
    # Unload the kernel modules, hide errors
    for module in $MODULES
      rmmod $module                                       2> /dev/null > /dev/null
    #  rmmod imq                                           2> /dev/null > /dev/null
    # Stop upon request
    if [ "$1" = "stop" ]; then
    # Load the needed kernel modules, hide errors
    for module in $MODULES
      insmod $module                                      2> /dev/null > /dev/null
    #  insmod imq numdevs=1                                2> /dev/null > /dev/null
    tc_setup() {
      # All get Stochastic Fairness
      $TC qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
      $TC qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
      $TC qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10
      $TC qdisc add dev $DEV parent 1:40 handle 40: sfq perturb 10
      $TC qdisc add dev $DEV parent 1:50 handle 50: sfq perturb 10
      $TC qdisc add dev $DEV parent 1:99 handle 99: sfq perturb 10
      # Filter traffic into classes by fwmark
      $TC filter add dev $DEV parent 1:0 prio 0 protocol ip handle 10 fw flowid 1:10
      $TC filter add dev $DEV parent 1:0 prio 0 protocol ip handle 20 fw flowid 1:20
      $TC filter add dev $DEV parent 1:0 prio 0 protocol ip handle 30 fw flowid 1:30
      $TC filter add dev $DEV parent 1:0 prio 0 protocol ip handle 40 fw flowid 1:40
      $TC filter add dev $DEV parent 1:0 prio 0 protocol ip handle 50 fw flowid 1:50
      $TC filter add dev $DEV parent 1:0 prio 0 protocol ip handle 99 fw flowid 1:99
    build_chains () {
      # tor                               [1:50]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 443,143,9001,9030                                               -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 443,143,9001,9030                                               -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 9000,9001,9002,9090,9091,19001,9030,9031                        -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 9000,9001,9002,9090,9091,19001,9030,9031                        -j RETURN
      # Cisco VPN                         [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p udp -m multiport  --${DPORT} 500,4500,10000                                                  -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p udp -m multiport  --${DPORT} 500,4500,10000                                                  -j RETURN
      # SYN and RST                       [1:10]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp  -m tcp       --tcp-flags ! SYN,RST,ACK ACK                                              -j MARK --set-mark 10
      $IPTABLES -t mangle -A ${CHAIN} -p tcp  -m tcp       --tcp-flags ! SYN,RST,ACK ACK                                              -j RETURN
      # SSH server - small packet (<512b) [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 22,222,10022,10222        -m length --length 0:512              -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 22,222,10022,10222        -m length --length 0:512              -j RETURN
      # SSH server - short burst          [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 22,222,10022,10222        -m limit --limit 2/s --limit-burst 10 -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 22,222,10022,10222        -m limit --limit 2/s --limit-burst 10 -j RETURN
      # SSH server - default              [1:30]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 22,222,10022,10222                                              -j MARK --set-mark 30
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 22,222,10022,10222                                              -j RETURN
      # SSH client - small packet (<512b) [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 22                        -m length --length 0:512              -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 22                        -m length --length 0:512              -j RETURN
      # SSH client - short burst          [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 22                        -m limit --limit 2/s --limit-burst 10 -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 22                        -m limit --limit 2/s --limit-burst 10 -j RETURN
      # SSH client - default              [1:30]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 22                                                              -j MARK --set-mark 30
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 22                                                              -j RETURN
      # HTTP server                       [1:30]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 80,81                                                           -j MARK --set-mark 30
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 80,81                                                           -j RETURN
      # HTTP client                       [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 80,443                                                          -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 80,443                                                          -j RETURN
      # NTP                               [1:10]
      $IPTABLES -t mangle -A ${CHAIN} -p udp               --${DPORT} 123                                                             -j MARK --set-mark 10
      $IPTABLES -t mangle -A ${CHAIN} -p udp               --${DPORT} 123                                                             -j RETURN
      # NNTP                              [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 119                                                             -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 119                                                             -j RETURN
      # VNC                               [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 5800,5900                                                       -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${SPORT} 5800,5900                                                       -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 5800,5900                                                       -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 5800,5900                                                       -j RETURN
      # RDP                               [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 3389                                                            -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 3389                                                            -j RETURN
      # SMTPS/POP3S                       [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 465,995                                                         -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 465,995                                                         -j RETURN
      # IM                                [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 5050,5190                                                       -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN} -p tcp -m multiport  --${DPORT} 5050,5190                                                       -j RETURN
      # FTP client                        [1:20]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 21                                                              -j MARK --set-mark 30
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 21                                                              -j RETURN
      # FTP server                        [1:40]
      $IPTABLES -t mangle -A ${CHAIN}          -m helper   --helper ftp                                                               -j MARK --set-mark 40
      $IPTABLES -t mangle -A ${CHAIN}          -m helper   --helper ftp                                                               -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 21                                                              -j MARK --set-mark 40
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 21                                                              -j RETURN
      # BitTorrent                        [1:50]
      $IPTABLES -t mangle -A ${CHAIN}         -m layer7    --l7proto bittorrent                                                       -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN}         -m layer7    --l7proto bittorrent                                                       -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 8161                                                            -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 8161                                                            -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 6881:6884                                                       -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 6881:6884                                                       -j RETURN
      # eDonkey                           [1:50]
      $IPTABLES -t mangle -A ${CHAIN}         -m layer7    --l7proto edonkey                                                          -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN}         -m layer7    --l7proto edonkey                                                          -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 4662                                                            -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${DPORT} 4662                                                            -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 4662                                                            -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p tcp               --${SPORT} 4662                                                            -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p udp               --${DPORT} 4672                                                            -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p udp               --${DPORT} 4672                                                            -j RETURN
      $IPTABLES -t mangle -A ${CHAIN} -p udp               --${SPORT} 4672                                                            -j MARK --set-mark 50
      $IPTABLES -t mangle -A ${CHAIN} -p udp               --${SPORT} 4672                                                            -j RETURN
      # Skype                             [1:20]
      $IPTABLES -t mangle -A ${CHAIN}         -m layer7    --l7proto skypetoskype                                                     -j MARK --set-mark 20
      $IPTABLES -t mangle -A ${CHAIN}         -m layer7    --l7proto skypetoskype                                                     -j RETURN
      # ACK                               [1:10]
      $IPTABLES -t mangle -A ${CHAIN} -p tcp  -m tcp       --tcp-flags SYN,RST,ACK ACK   -m length --length :64                       -j MARK --set-mark 10
      $IPTABLES -t mangle -A ${CHAIN} -p tcp  -m tcp       --tcp-flags SYN,RST,ACK ACK   -m length --length :64                       -j RETURN
      # UDP                               [1:10]
      $IPTABLES -t mangle -A ${CHAIN} -p udp                                                                                          -j MARK --set-mark 10
      $IPTABLES -t mangle -A ${CHAIN} -p udp                                                                                          -j RETURN
      # ICMP (ping)                       [1:10]
      $IPTABLES -t mangle -A ${CHAIN} -p icmp                                                                                         -j MARK --set-mark 10
      $IPTABLES -t mangle -A ${CHAIN} -p icmp                                                                                         -j RETURN
      # Unclassified <default>
      $IPTABLES -t mangle -A ${CHAIN}                                                                                                 -j LOG --log-level debug --log-prefix "Unc ${CHAIN}: "
    ### UPLINK
    # Set outbound MTU and queue length
    $IP link set dev $WANDEV mtu  ${MTU}
    $IP link set dev $WANDEV qlen ${QLEN}
    # Create qdisc/classes, point unmarked traffic to 1:30
    $TC qdisc add dev $WANDEV root       handle  1:0  htb default 30
    $TC class add dev $WANDEV parent 1:0 classid 1:1  htb rate ${UPLINK}kbit
    $TC class add dev $WANDEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit           ceil ${UPLINK}kbit           prio 1
    $TC class add dev $WANDEV parent 1:1 classid 1:20 htb rate ${UPLINK}kbit           ceil ${UPLINK}kbit           prio 2
    $TC class add dev $WANDEV parent 1:1 classid 1:30 htb rate $((90*$UPLINK/100))kbit ceil ${UPLINK}kbit           prio 3
    $TC class add dev $WANDEV parent 1:1 classid 1:40 htb rate $((75*$UPLINK/100))kbit ceil ${UPLINK}kbit           prio 4
    $TC class add dev $WANDEV parent 1:1 classid 1:50 htb rate $((25*$UPLINK/100))kbit ceil $((25*$UPLINK/100))kbit prio 5
    $TC class add dev $WANDEV parent 1:1 classid 1:99 htb rate ${UPLINK}kbit           ceil ${UPLINK}kbit           prio 99
    # Create iptables chains
    $IPTABLES -t mangle -N UL
    ### DOWNLINK
    # Bring imq0 interface up
    #$IP link set imq0 up
    # Create qdisc/classes, point unmarked traffic to 1:30
    #$TC qdisc add dev imq0 root       handle  1:0  htb default 30
    #$TC class add dev imq0 parent 1:0 classid 1:1  htb rate ${DOWNLINK}kbit
    #$TC class add dev imq0 parent 1:1 classid 1:10 htb rate $((DOWNLINK/5))kbit ceil ${DOWNLINK}kbit prio 1
    #$TC class add dev imq0 parent 1:1 classid 1:20 htb rate $((DOWNLINK/5))kbit ceil ${DOWNLINK}kbit prio 2
    #$TC class add dev imq0 parent 1:1 classid 1:30 htb rate $((DOWNLINK/5))kbit ceil ${DOWNLINK}kbit prio 3
    #$TC class add dev imq0 parent 1:1 classid 1:40 htb rate $((DOWNLINK/5))kbit ceil ${DOWNLINK}kbit prio 4
    #$TC class add dev imq0 parent 1:1 classid 1:50 htb rate $((DOWNLINK/5))kbit ceil ${DOWNLINK}kbit prio 5
    #$TC class add dev imq0 parent 1:1 classid 1:99 htb rate ${DOWNLINK}kbit     ceil ${DOWNLINK}kbit prio 99
    # Create iptables chains
    #$IPTABLES -t mangle -N DL
    #$IPTABLES -t mangle -A DL -j IMQ --todev 0
    $TC qdisc  add dev $WANDEV handle ffff: ingress
    $TC filter add dev $WANDEV parent ffff: protocol ip prio 50 u32 match ip src police rate ${DOWNLINK}kbit burst $((100000*$DOWNLINK/19632))b drop flowid :1
    ### FINALIZE
    # Enable iptables rules
    #$IPTABLES -t mangle -I PREROUTING  -i $WANDEV -j DL
    Hope it helps.

  4. Azuse

    Azuse Addicted to LI Member

    Pardon my noobishness but that looks like it also contains classifications. Hopefully I'll have time to read up on this, but changing tomato can't be an easy or quick thing to do. Is it not possible to insert an overall limit into the inbound and have them throttle classes rules via a firewall script?
  5. Porter

    Porter LI Guru Member

    Azuse: Which part are you referring to when you are mentioning classification?
    You can have a rather simple inbound overall limit just by deleting the root qdisc and then setting the limit to 1000Kbit:

    tc qdisc del ppp0 ingress 2>/dev/null
    tc filter add dev ppp0 parent ffff: protocol ip prio 50 u32 match ip src police rate 1000kbit burst 10k drop flowid :1

    Actual classification takes a lot more commands, but most of the things required for this are already available and don't have to be written from scratch. That's why I meant it wouldn't be a really difficult task. Reading and understanding QoS is, though. ;)
  6. Porter

    Porter LI Guru Member

    I'm having problems understanding a specific iptables-command that Tomato uses in QoS:

    iptables -A QOSO -p udp --dport 53 -m bcount --range 0x800 -j CONNMARK --set-return 0x5/0xFF

    What is the "--set-return" part doing exactly. It's not in the manpage of iptables: http://unixhelp.ed.ac.uk/CGI/man-cgi?iptables+8

    I found it in Tomato's source, but couldn't figure out what it does. Does --set-return differ from the three possibilities mentioned above?

    A more general question would be how the mask feature is useful. If anybody could try to explain this I would be very thankful.
  7. rhester72

    rhester72 Network Guru Member

    iptables -m bcount -j CONNMARK -h

    CONNMARK target v1.3.8 options:
      --set-mark value[/mask]       Set conntrack mark value
      --set-return [--mask mask]    Set conntrack mark & nfmark, RETURN
      --save-mark [--mask mask]     Save the packet nfmark in the connection
      --restore-mark [--mask mask]  Restore saved nfmark value
    It's just a shortcut saving you from having to write (and have kernel parse) two iptables rules, one to --set-mark and another jump to RETURN.

    Based on tomato/release/src/router/rc/qos.c, it looks like he's trying to distinguish his CONNMARK set by mask 0xFF (he ignores all other masks in the jumptable comparison at line 50).

  8. Toastman

    Toastman Super Moderator Staff Member Member

  9. Toastman

    Toastman Super Moderator Staff Member Member

  10. Azuse

    Azuse Addicted to LI Member

    While I'd love this, I'm sure I'm in a minority as far as development is concerned. Would a better starting point not be integrating port forwarding names into the qos system? For example, both skype and msn voip/video etc use random ports and frequently fall into the general qos group (as do some games) but always have the same names under port forwarding. Being able to enter one of those names under qos, rather than ports or l7 filters which miss things, would be both useful to everyone as well as a lot of the ground work for the above.
  11. Porter

    Porter LI Guru Member

    Unfortunately things haven't progressed a great deal because I needed to study alot. Right now I propably have a few weeks left where there is a bit more spare time which would allow me to get involved again.

    One major problem remains though: I have no programing skills in the relevant languages, which made me consider buying at least a book about C/C++, because the internet doesn't offer something like this for free.

    Making UPnP and QoS work together certainly is a good idea for the future, but right now you would still only be able to use this in outbound direction. On the other hand I don't think inbound shaping is a precondition to connect theses two systems. So developing this now for outgoing traffic should be possible.

    I really like the idea for the inbound graphs, too. But unless there is someone who can bring programming skills and QoS-Knowledge together it's very difficult to build.
  12. Toastman

    Toastman Super Moderator Staff Member Member

    Azuse, OK on your idea, but that is not really what this thread is about, however it could be the subject of another thread. I personally believe that if you want this kind of QOS to work, you have to understand how it works and define your own rules. The one exception to this I would make is that if UPnP could be connected in some way to the QOS class, I can't see any objection to doing it. The big problem is simply that at the moment there isn't a good way to achieve this - L7 filters try but fail. And there's simply no black magic that works out how and why to block or prioritize with no information.

    Going back to the subject of this thread, at present the QOS in Tomato can be said to be incomplete, it needs to be completed. Think about it - as far as I am aware, there has never been a third party firmware that has been properly completed. Before the main functions have had the bugs totally ironed out, the authors have always lost track and rushed off to add some other fancy "features". Tomato has not gone this way yet, but the fact remains the QOS isn't finished - but it is almost finished.

    The fact is, we have no overall limit on the incoming data. The limits we set on individual classes can add together to exceed our available bandwidth,which means that it can become congested. Also, if more than one class is in use, the total bandwidth is shared equally between those classes. Therefore each of those classes is often not even able to rise to the limit we set for it! In other words - Tomato's QOS doesn't work in the way we thought it did. It's perhaps unkind to say it's a mess, but for a long time we've misunderstood how some parts of it work. It's lucky that most of the time it works well regardless of that, but it doesn't work as well as it could.

    Porter, (the discoverer of this omission) - please correct me if my description is wrong ?

    So, since (a) I hate things that don't work properly and (b) I don't see why we should not be able to use the bandwidth we have paid for to the full, I'm rather interested in trying to get sufficient people together to finish it.

    There are many people with coding skills reading these forums, we do need to ask for help from them. But first we have to clearly define the problem and find the solution, before we can ask anyone to code it. A third graph is easy to put on the page - just copy the existing code, what is needed is the ingress IMQ code and a method to input it into the graph. It might well not be such a big task as we think, assuming someone has the necessary knowledge!

    Porter, there's loads of time, the studies come first...

    Perhaps those with some coding skills might take a look at Gargoyle source code and see how the incoming QOS part works, and see if it can be useful here (that is - if you can find the code, cos I can't) ?
  13. Azuse

    Azuse Addicted to LI Member

    Reinstalling tomato, and redoing every setting (should really automate thing :)), has once again reminded me how annoying - read ineffective/almost useless - the inbound limits are. Setting p2p at 80, or even 70% will still result in it filling up the line.

    Seeing as it's unlikely imq shaping will be reaching a tomato router near me anything soon, could we not construct a script (one that could be inserted via the gui, rather than putty* :)) in which people could simply edit the traffic values for their lines? It would go some way to alleviating both our loss of download speeds at quiet times (1/3 in my case :() and presumably making more people aware that inbound shaping is possible.

    *Is the one on the first page for this, or gui?
  14. Porter

    Porter LI Guru Member

    If you want to have a script which lets you enter only your line speed in every direction and then calculates the class speeds accordingly there is no other choice then to use the command line and therefore some ssh-client like putty. The reason for that is that all the script input fields in the Tomato GUI won't parse there input. They are just executing the commands line by line - correct me if I'm wrong.

    If you don't mind scrolling through lots of lines of commands and changing the speed limits to your liking then you can simple use my script, which should be pastable in WAN UP of the GUI. Please keep in mind, that you won't see any errors when the execution should fail. Working on the command line has another advantage: you can debug the script right there, whereas you would have to do a reconnect each time you changed something in the script when you are just pasting it in the GUI.

    You can debug the script right there because I'm providing two things here: a start and a stop script.

    Now something about the shaping script: It's been used and tested on a WRT54GL for almost have a year and it utilizes IMQ-Devices, which means that there is real inbound shaping - something that hasn't been integrated into the Tomato-GUI (that's mainly why I opened this thread). The only real shortcoming lies with FTP-Uploads, which unfortunately end up in the bulk class which means that with parallel p2p-uploads ftp is rather slow (5-6kbit). The script is configured for a line that should do 4500/640; with a safety net this means 4200/480.

    Since everybody has a different dsl-connection you certainly need to change some rates. First of all you need to adjust the root-class in UP and DOWN direction.

    ######### Incoming Rules
    ######### chain router
    /usr/sbin/tc class add dev imq0 parent 1:1 classid 1:11 htb rate 4200Kbit prio 3 quantum 1532
    Change 4200 to your actual line speed ("actual" meaning you have tested your connection speed over some time and acquired a rather precise knwoledge about what your connection can handle; it is advisable to deduct at least 15% from the actual line speed; i.e. my actual line speed is 5000 -15% = 4250)


    ######### Outgoing Rules
    ######### chain router
    /usr/sbin/tc class add dev imq1 parent 1:1 classid 1:11 htb rate 480Kbit prio 3 quantum 1532
    Change 480 to your actual line speed. To be on the safe side I actually deducted 25% of my line speed.

    For the time testing your new rules there shouldn't be other traffic on the line! Changing everything first and then put the script to use isn't a good idea, because finding errors will be so much more difficult. Therefore you should probably first learn with something familiar and that would be HTTP-traffic.

    For instance:

    ######### pipe Web-Traffic
    /usr/sbin/tc class add dev imq0 parent 1:11 classid 1:18 htb rate 1024Kbit ceil 3600Kbit burst 256Kbit prio 3 quantum 1532
    rate ist the guaranteed speed of this class, which is 1024kbit.
    ceil is the top speed this class can have, which is 3600.
    burst is the amount of bytes that can be burst at ceil speed, in excess of the configured rate. (I will repeat myself here: don't touch the burst rates, unless your line has significantly lesser parameters than mine, in which case you should only decrease it. Burst rates of around 1-2% of the class rate are usually a good idea. For "bursty" traffic like HTTP this is more like a guideline and not a rule.)

    If your line is a 768kbit line you might want to change the command to:
    /usr/sbin/tc class add dev imq0 parent 1:11 classid 1:18 htb rate 250Kbit ceil 500Kbit burst 64Kbit prio 3 quantum 1532

    On the other hand for a 16000kbit line you could change it to:
    /usr/sbin/tc class add dev imq0 parent 1:11 classid 1:18 htb rate 6000Kbit ceil 12000Kbit burst 512Kbit prio 3 quantum 1532

    Always have a big download running so that you can follow your changes!

    The script contains classes for: ICMP, some online games, DNS, SSH, VoIP (which probably hasn't been tested thorougly enough), P2P (only bittorrent; port 55200 is used for your favourite bt-client), HTTP/S (flashvideo and streams _should_ end up in this class, too), Mailing (SMTP, POP3, IMAP and probably some SSL-Ports...) and FTP.

    Generally you just need two commands to add a rule: one for the source-port (sport) and one for the destination-port (dport).

    Please refrain from changing the burst rates (i. e. "burst 256Kbit") because very often they just mess things up. Mine are probably still too high. If your ping times tend to be chaotic that might be the reason.

    Unfortunately the script is too long for a message on this board, therefore you have to use the links.

    The above one you can paste into the WAN UP section.

    If you would like to debug on the command line use the additional script above.

    If you would like to use the command line here's a short guide:

    1. Install putty (other clients might be fine, too, but that's what I used).
    2. Connect to your Tomato via ssh (as root).
    3. Now a few commads you will need to enter:
    4. cd /root
    5. cat > mshaper
    6. go to the mshaper link and copy everything to your clipboard
    7. get the putty window active again and just press the right mouse button somewhere over the black background - the text should appear in there
    8. In the putty window hit ctrl+d. This should bring you back to the command prompt showing this: #
    (if it doesn't, try to hit enter once and then ctrl+d) (if something goes wrong abort with ctrl+c)
    9. chmod 744 mshaper (this will make our script executable)
    10. cat > mshaper-stop
    11. repeat steps 7 and 8 for mshaper-stop-link
    12. chmod 744 mshaper-stop

    I hope this covers the basics. Have fun adjusting!
  15. Toastman

    Toastman Super Moderator Staff Member Member

  16. Toastman

    Toastman Super Moderator Staff Member Member

    Extract from "Traffic Shaping", Diane Trout

    which was taken from Dan Singletary's article here:

    This author's approach set up only two classes for incoming data. TCP and Non - TCP. In other words, incoming TCP packets related to any of the outgoing classes (he had 7) could be dropped randomly. A better solution would presumably use the same classes as outgoing, and drop them in the same order of priority.
  17. Azuse

    Azuse Addicted to LI Member

    UDP is really only used for latency sensitive traffic anyway (games/voip) which should be at the top of the qos tree anyway, thus not need shaping.

    uTP, the only UDP traffic you might want to shape, throttles it's self when packets are delayed or dropped thus shaping it doesn't result in a lot of retransmission and shaping makes sense.
  18. Toastman

    Toastman Super Moderator Staff Member Member

    Yes, doing some heavy reading, seems that many VOIP implementations also have their own methods of slowing down links and also reverting to lower quality/less bandwidth modes to allow them to work on links that are becoming congested. Take care of TCP and the UDP aspect isn't quite as gloomy as it first appears. Maybe.

  19. Toastman

    Toastman Super Moderator Staff Member Member

  20. Toastman

    Toastman Super Moderator Staff Member Member

    More thoughts on QOS development following on with Porter's suggestions. I'm not a programmer so this is my feeble attempt to visualize what may need to be done, following some discussion between us. I have cut and pasted bits into a whole and edited what I can to make it into 10 qdiscs (classes) instead of 5. There will be mistakes here which need to be corrected, but have I got the general idea right?

    I'm hoping, that this will spark some interest in some benevolent and talented guy out there who has the time and enthusiasm to become interested in this QOS problem and fix it :biggrin:

    Set up incoming IMQ device (being careful not to conflict with any IMQ devices used by other programs (e.g. IP/MAC limiter)

    This example asumes a 4mbps downlink.

    We make one IMQ device and attach 10 qdiscs or clases to it.

    # Load modules for IMQ-support
    modprobe imq
    modprobe ipt_IMQ

    # Set up incoming IMQ-device
    ip link set imq0 up

    # Load iptables rules for redirection:
    iptables -t mangle -I PREROUTING -i ppp0 -j IMQ --todev 0

    ####### Readjust the rest

    # IMQ-device
    TQA="tc qdisc add dev $I"
    TCA="tc class add dev $I"
    TFA="tc filter add dev $I"
    Q="sfq perturb 10"

    case "$1" in
    tc qdisc del dev $I root 2>/dev/null
    $TQA root handle 1: htb default 40

    ## now DOWNLINK rates!!! just an example... burst rates should be configured better!

    ## First 5 classes below the parent - taken from Porter's trial QOS system which had only 5 classes

    $TCA parent 1: classid 1:1 htb rate 4000kbit ceil 4000kbit

    # egress 0: 90-96%
    $TCA parent 1:1 classid 1:10 htb rate 512kbit ceil 4000kbit burst 16Kbit prio 1 quantum 1492
    $TQA parent 1:10 handle 10: $Q
    $TFA parent 1: prio 10 protocol ip handle 1 fw flowid 1:10
    # egress 1: 80-94%
    $TCA parent 1:1 classid 1:20 htb rate 128kbit ceil 3000kbit burst 16Kbit prio 2 quantum 1492
    $TQA parent 1:20 handle 20: $Q
    $TFA parent 1: prio 20 protocol ip handle 2 fw flowid 1:20
    # egress 2: 80-87%
    $TCA parent 1:1 classid 1:30 htb rate 32kbit ceil 64kbit burst 16Kbit prio 3 quantum 1492
    $TQA parent 1:30 handle 30: $Q
    $TFA parent 1: prio 30 protocol ip handle 3 fw flowid 1:30
    # egress 3: 80-93%
    $TCA parent 1:1 classid 1:40 htb rate 512kbit ceil 2000kbit burst 16Kbit prio 4 quantum 1492
    $TQA parent 1:40 handle 40: $Q
    $TFA parent 1: prio 40 protocol ip handle 4 fw flowid 1:40
    # egress 4: 85-92%
    $TCA parent 1:1 classid 1:50 htb rate 128kbit ceil 1024kbit burst 16Kbit prio 5 quantum 1492
    $TQA parent 1:50 handle 50: $Q
    $TFA parent 1: prio 50 protocol ip handle 5 fw flowid 1:50

    ## Now I've added the next 5 by just cut and paste of the 1st 5 so we will need to change parameters to suit. (As yet I don't understand completely what to do here, such as burst rates etc). I changed the egress and prio nos. These parameters should be set by the GUI as needed. Mickup of QOS-Basic page is below

    # egress 5: 90-66%
    $TCA parent 1:1 classid 1:10 htb rate 512kbit ceil 4000kbit burst 16Kbit prio 6 quantum 1492
    $TQA parent 1:10 handle 10: $Q
    $TFA parent 1: prio 10 protocol ip handle 1 fw flowid 1:10
    # egress 6: 80-74%
    $TCA parent 1:1 classid 1:20 htb rate 128kbit ceil 3000kbit burst 16Kbit prio 7 quantum 1492
    $TQA parent 1:20 handle 20: $Q
    $TFA parent 1: prio 20 protocol ip handle 2 fw flowid 1:20
    # egress 7: 80-87%
    $TCA parent 1:1 classid 1:30 htb rate 32kbit ceil 64kbit burst 16Kbit prio 8 quantum 1492
    $TQA parent 1:30 handle 30: $Q
    $TFA parent 1: prio 30 protocol ip handle 3 fw flowid 1:30
    # egress 8: 60-93%
    $TCA parent 1:1 classid 1:40 htb rate 512kbit ceil 2000kbit burst 16Kbit prio 9 quantum 1492
    $TQA parent 1:40 handle 40: $Q
    $TFA parent 1: prio 40 protocol ip handle 4 fw flowid 1:40
    # egress 9: 65-72%
    $TCA parent 1:1 classid 1:50 htb rate 128kbit ceil 1024kbit burst 16Kbit prio 10 quantum 1492
    $TQA parent 1:50 handle 50: $Q
    $TFA parent 1: prio 50 protocol ip handle 5 fw flowid 1:50

    I think so too. Also, any incoming rules should apply to TCP only. Non-TCP incoming data should always be accepted and delivered immediately as there is no point in dropping it. We cannot slow down incoming streams by doing so, and it's already reached the router anyway.

    I've made a mockup below - the incoming classes should take the same class names as the outgoing classes so that traffic in both directions can be related to one another by class, and also plotted on incoming pie chart (and any other future addons).


    1 - How the QOS-BASIC page would look


    2 - How the QOS-GRAPHS page might look


    To be further edited.
  21. bigjohn

    bigjohn Addicted to LI Member

    This is rocking good stuff.

    Isn't most P2P traffic UDP though?

    any further development?
  22. Toastman

    Toastman Super Moderator Staff Member Member

    I think it depends on your situation. Here in a large apartment block with up to 20 P2P users typically downloading there is a hell of a lot of outgoing UDP traffic but this actually results in quite a small amount of downloads. We've experimented with this for months, especially since uTP from uTorrent appeared, and we have all agreed that all of us get the best downloads if we turn off DHT and uTP. When this is done, traffic drops by at least 60% and our download speeds (using TCP alone) double or treble. Of course this also coincides with better throughput for other applications and web surfing too. uTP really screws our networks.

    Since we have no control over what people run on their computers inside their rooms, the only way we have to prevent this excess of UDP traffic from screwing up the internet for 100+ other users is simply to place it in a class and then throttle it completely - i.e. dump all UDP unless covered by an earlier rule. Running this for maybe a year now and the situation has not changed.

    There are others who find their situation is different.

    NB - Problem with further development of QOS is shortage of programmers !
  23. bigjohn

    bigjohn Addicted to LI Member

    Yeah, I'm not a programmer, don't play one on TV either.
    Some day I'll get time to learn a little bit more about linux...and scripting... but i don't think there are enough years left in my life to find the time to encourage myself to learn C. I played with PL-1, Pascal, and PL-C over 20 years ago... put me off the concept entirely.

    Anyway, like the look of what you're doing!
  24. tiomo

    tiomo Networkin' Nut Member

    Hi everyone,

    this is my first post on this forum. I own a Linksys E3000 and have installed "Tomato v1.28.7493 MIPSR2-Toastman-RT K26 USB VPN" on it a week ago.

    First let me thank you for all the work you've done on Tomato. Coming from DD-WRT switching to Tomato was really a great experience!

    I've been fiddling around with QOS over the past week and I've now come to a point where everything pretty much works as expected. The only thing (as you guys have discovered as well) is that if a download saturates the downlink you run into problems.

    In my particular case - watching BBC iPlayer while downloading a file over Bittorrent. Once you get some high-bandwidth seeders your download goes up and the streaming movie starts buffering.

    I also have to say that I'm possibly more prone to this problem since I have a 2000/240 kbit line. We're getting a faster line soon but the problem will remain for some situations.

    Since you guys have already pretty much posted the solution in this thread I thought I could give it a shot and change the code in Tomato which is responsible for generating the QOS script which in turn sets up TC and IPTABLES.

    I did some heavy digging through Tomato's sources and found the 'rc' command which is ultimately responsible for generating the QOS script. The bottom line is that all the parameters (wan device name, inbound link bandwidth, ... ) are retrieved from the NVRAM.

    I've modified the release/src/router/rc/qos.c file (from Toastman's 1.28.7493 release) which now contains the code to do the following:
    1. Set up IMQ device
    2. Set up a root class/qdisc which limits incoming traffic to the incoming bandwidth as set in the QOS webinterface
    3. Set up a class and qdisc for every QOS class as defined in the QOS webinterface. (The current percentage that you can set is used as RATE, CEIL is the incoming bandwidth)
    4. Set up a filter to move the incoming packets into the correct class/qdisc after they've been marked
    5. Add rule to PREROUTING chain of the 'mangle' table to move incoming TCP packets on WAN device to IMQ device

    Now I was able to compile a version of 'rc' which has testing enabled and so I was able to generate the file which is saved to /etc/qos by hand.

    I haven't deployed a custom firmware image to my router yet (I've read somewhere that the Linksys E-series has 'special needs' and I don't want to brick the thing) but I copied the 'rc' executable to my router and generated a new 'qos' script which I was able to execute.

    After some quick tests I'm able to confirm that it seems to work as expected, now I'm hoping for your help to integrate this into a proper testing build.

    - Incoming bandwidth is now an OVERALL LIMIT
    - The incoming percentage is the MINIMUM guaranteed bandwidth (RATE as per HTB definition) a QOS class will get (maximum is at this moment the overall limit)
    - The incoming percentages of all QOS classes MUST BE less than or equal 100%

    - I've only tested this for a little while, but it seems to work (it needs to be tested properly)
    - I'm using imq0, I found that the BW limiter code uses imq1 and imq2 so I hope there is no conflict there
    - We still need to be able to set the incoming CEIL percentage for every QOS class in the web interface (and save it in the NVRAM!) so that I can set it in the code.
    - The web interface must show a warning when the percentages of the guaranteed incoming bandwidth for all QOS classes are greater than 100%

    I've attached the modified qos.c file and the qos script which was generated on my router by executing 'rc rctest qos' (won't work if testing is not enabled). I've checked out the git repository to get the source code but am not confident how to go about to create a new testing branch and how to create a build that won't brick my E3000.

    That's it for now. Looking forward to hearing from you guys!

    Merry (belated) christmas,

    Attached Files:

  25. phuque99

    phuque99 LI Guru Member

    The difference in "tab" length between your file and the one in git caused a lot of unchanged lines to show up on diff. Would you mind posting or sharing the actual line diff that you modified in your version of qos.c?
  26. Toastman

    Toastman Super Moderator Staff Member Member

    Tiomo, this is very interesting. I'm sending you a PM, keep a look out for it?

    Happy New Year!

  27. tiomo

    tiomo Networkin' Nut Member

    Hey there,
    sorry for the whitespace changes in the file. My IDE messed that one up. I've attached a unified diff (diff -u -E -b) to the original post which only shows the real changes which I've made.
  28. tiomo

    tiomo Networkin' Nut Member

    You've got mail!

    By the way, I was able to successfully build an image for my E3000 ('make help' is your friend!) which I have flashed yesterday evening. I was successfully able to watch a movie on BBC iPlayer while Bittorrent was happily downloading the Ubuntu installation ISO at the same time. Web browsing still felt quite snappy even though my line was downloading at 1800 kBit/s!

    I've noticed that HTTP download speeds don't seem to be reported properly in Chrome. I guess it is because the speeds may be averaged over the whole time of the download? The bandwidth graphs and the per-IP traffic graphs as well as the transfer rate page in Tomato did show that it does work properly as Porter reported a while ago.

    Happy New Year!

  29. Toastman

    Toastman Super Moderator Staff Member Member

    Yes, it seems to be working quite well. I have it online now in a building which has around 140 clients, usually around 25 or more are active with many of them using P2P. I'm going to play around and see what it does. But quite successful, I think. If this had an incoming pie chart it would be a lot easier to evaluate, maybe soon :)

    Congrats on working this out!
  30. tiomo

    tiomo Networkin' Nut Member


    The pie chart for the incoming bandwidth would indeed make it a lot easier to evaluate the effects. We also need to be able to set a limit (CEIL) for each QOS class as we can for the outgoing bandwidth. I've had a look at qos-settings.asp and at the code in qos.c which deals with the outgoing bandwidth - it shouldn't be too difficult to get that working as well.
  31. tiomo

    tiomo Networkin' Nut Member

    Okay here are an updated qos.c file (belongs in release/src/router/rc) and an updated qos-settings.asp file (belongs in release/src/router/www) which will enable you to set an individual incoming ceiling for every QOS class.

    This information is stored in NVRAM the same way as it is for outgoing rate and ceiling. Your NVRAM content will still have the old format after flashing a new image so the only thing you need to do as well is to go to the basic QOS settings and save your settings once.

    I've added a message which appears if you're trying to set rate higher than ceil for a class (both for outgoing and incoming) since I assume that it doesn't make sense to do that. You can still save that to NVRAM though. No further validation yet.

    Attached Files:

  32. tiomo

    tiomo Networkin' Nut Member

    Another thing which I've noticed while looking through the code in qos-settings.asp:

    The settings (both incoming and outgoing) for the last QOS class (named "Crawl" by default) are ignored and not written to NVRAM at all. Is that intended behaviour?
  33. Toastman

    Toastman Super Moderator Staff Member Member

    No, it's not intended. Maybe it needs fixing, presumably this is a legacy from long ago, but I can't see why it should be.

    Hmmm - however, if I reboot, these setting do appear on my RT-N16. So maybe something is busted?

    I'm going to try that new file now :)
  34. tiomo

    tiomo Networkin' Nut Member

    My bad. I misread a statement. I've also set 'Crawl' to zero which disables it, therefore it doesn't show up in my /etc/qos file.
  35. Toastman

    Toastman Super Moderator Staff Member Member

    ;) no worries

    Not got much time to look, but things to check:

    In order to totally throttle a particular class - such as the crawl - ideally we need to be able to set to zero bandwidth in and out. Not tried it yet.

    The old setting that enabled a class to ignore all limits might be useful, does it still do that?

    You're right that the rate higher than ceiling check would be useful, cos it wouldn't make much sense. Probably in future best to disable save until a correct entry is made.
  36. tiomo

    tiomo Networkin' Nut Member

    Okay happy new year everyone! Spent yesterday on the couch, not many brain cells left for intense router hacking :)

    Here's how the settings work at the moment:

    For outbound traffic:
    - setting '0' (zero) for the rate totally disables TC code generation for that class (which effectively places all of the traffic of that class in the default class)
    - setting '0' (zero) for the ceiling will (probably, not tested I only looked at the code) set an (invalid) ceiling of 0 kbps

    For inbound traffic:
    - setting 'No Limit' (which equals zero) for the rate totally disables TC code generation for that class (which effectively places all of the traffic of that class in the default class, this is NOT what we want?)
    - setting 'No Limit' (which equals zero) for the ceiling will (probably, not tested I only looked at the code) set an (invalid) ceiling of 0 kbps

    So the "Ignore limit" setting isn't doing what it should do. If we want to be able to completely exclude traffic of one or multiple classes from being sent to the IMQ device we will possibly need to define this by iptables rules.
  37. tiomo

    tiomo Networkin' Nut Member

    I have successfully added a graph with the incoming bandwidth distribution to the page with the two existing graphs. It seems to work as expected, I wasn't able to test it for long since our line has been down since Saturday afternoon :/. Hopefully they'll fix it today.

    Put the files into the appropriate directories (refer to the directory structure in the zip file) and compile!


    Attached Files:

    teaman likes this.
  38. teaman

    teaman LI Guru Member

    Hi there!

    While playing around with this code, trying to merge this whole new QoS thing into Teaman-ND to see how it would be.... I noticed something interesting that might cause possible problems at some point on different versions of config_base out there:
    However, on release/src[-rt]/include/bcmparams.h we have:
    #define DEV_NUMIFS16/* Max. # of devices/interfaces supported */
    Appently, that constant is used on these other files:
    Although I'm aware only a handful of IMQ devices are created, we might possibly consider reducing such limit at some point, don't you agree?


    BTW: nice work, tiomo!
  39. tiomo

    tiomo Networkin' Nut Member

    Hi teaman!

    Thanks, it's actually really good fun to hack your own router's firmware! ;-)

    As for your finding - if I understood it correctly you're saying that the total number of all interfaces in the system is limited to 16?
    And that we also allow up to 16 IMQ devices?

    I guess that should be definitely limited then. It doesn't make sense to allow so many IMQ devices to be created. I only know of three IMQ interfaces/devices which are used at the moment - imq1 & imq2 (used by BW limiter code) and imq0 (used for the incoming QOS).

    I've also found a bug in the current incoming QOS implementation: the iptables rule to direct all incoming traffic to the imq0 device which is inserted into the INPUT chain of the mangle table doesn't survive a WAN reconnect. Somehow some code is executed then which removes all rules and the QOS isn't stopped/started during the WAN reconnect. Do you have any idea where I should start looking to get QOS stopped/started on a WAN reconnect?

  40. tiomo

    tiomo Networkin' Nut Member

    Okay, I think I fixed the issue with the missing iptables rule on a WAN reconnect. I've moved the generation of the iptables rule to qos_ipt (that's where it should've been in the first place) and it seems to work now.

    I've also created a const char which defines the name of the imq device in qos.c. That way we just have to change it in that one place if we (ever) have to change it.

    I've attached a zip file with the latest version of the files.


    Attached Files:

  41. Toastman

    Toastman Super Moderator Staff Member Member

    Long time ago when we were using IMQ to try to display user bandwidth graphs, I played with increasing number of IMQ. There's an overall setting (imq_maxdev or something) but I've forgotten where it was hidden now. I enabled around 100 to experiment. Well, it worked but of course it was not a very good way to do this so I never took it further. Anyway, several months later I did the same thing in a more recent Tomato and the damned thing refused to give me any more than 16 devices. Not that it was practical anyway of course, I'm just mentioning it because I never found out why suddenly I couldn't create more than 16.

    Then Teaman's various monotoring progs. came along, and the whole thing was forgotten. In my firmware, I thought I had already reduced to opening only 3 devices, IMQ0 I kept reserved for QOS ingress, and IMQ1/2 for Bandwidth Limiter. Each IMQ device does take up a few hundred bytes so no particular reason to open them if not necessary. This is in RT branch, maybe the ND branch is still 16?

    [FYI - I was figuring on maybe dumping the bandwidth limiter IF anything better came along. It is, after all, a QOS system in its own right, but works by limiting bandwidth instead of "doing it properly" :eek:. It is commonly used when a P2P hog uses too much bandwidth, but the new QOS ingress system MAY henceforth render it somewhat irrelevant. I would replace it if the functionality (limiting/quotas/whatever) were achieved in a different way which was better. Otherwise it stays for now. ]
  42. Toastman

    Toastman Super Moderator Staff Member Member


    There are 2 things to consider here.

    1) Ideally when you want to completely throttle a class, kill it stone dead, we would like to drop everything, give it zero bandwidth, whatever. Previously we had to give 1% rate and limit and allow some bandwidth. If this could be addressed elegantly without any further user involvement apart from the QOS settings, it would be great :) A setting of "No Bandwidth" for example ...

    2) Allow a class to "break" the limit but still show as belonging to that class. i.e. Although we have to set Max Bandwidth less 15-30% for QOS to actually be able to work, Tomato previously did allow a class with this setting to ignore all limits and use whatever bandwidth was available, even though this would remove the "headroom" that we needed. I don't think this is necessary now and can wreck QOS, although previously with no ingress it was useful.

    The original wording was No Limit and the setting 0.
  43. teaman

    teaman LI Guru Member

    What I meant is... we can allow creating as many IMQ devices as we want. In fact, that setting only tells the value of X when modprobing the module, if not specified:

    modprobe imq numdevs=X
    The possibly bad news about allowing lots of those 'extra' interfaces would be... whenever networking is restarted... as it loops thru a max of DEV_NUMIFS on a few pieces of the code:
    /* find the interface name to which the address is assigned */
    for (j = 1; j <= DEV_NUMIFS; j ++) {
    ifr.ifr_ifindex = j;
    if (ioctl(s, SIOCGIFNAME, &ifr))
    And... that could be bad ;)
  44. Toastman

    Toastman Super Moderator Staff Member Member

    Been thinking about how to address UDP - I think it should be "visible" in the QOS system graphs, meaning it should pass through the IMQ device. So therefore we would simply need to define a rule in "Classification" to put all incoming UDP into a high priority class, that would comply with the statement in the last post about it being pointless to drop UDP. This may be so, but if we have a situation where we actually want to PREVENT UDP for applications that aren't trusted, such as P2P, then this gives us a way to discourage it's use by dropping it. Maybe.

    It would also allow us to place it in a lower priority class if we wished, for some reason I can't think of at the moment :)

    Just trying it now - passing all traffic through IMQ instead of just tcp

  45. Toastman

    Toastman Super Moderator Staff Member Member

    Teaman, this was what you asked about ?

    Perhaps not, it was Gargoyle's active QOS bandwidth calculation, but anyway, this is also of interest.


    Porter played with this and posted a nice screenshot of the difference it made:


  46. Toastman

    Toastman Super Moderator Staff Member Member

    http://www.opalsoft.net/qos/DS-23.htm is nice tutor ...


    Tiomo, Porter, what do you think of this - what about sending ALL traffic through the IMQ device, so that it can be classified as normal and be shown along with everything else in the monitoring graphs and stats. We already have the ability in QOS to apply QOS and priority to any protocol, so they should be included in QOS for those who may need them.

    While it is true that UDP is a connectionless protocol and we can't slow it down by dropping packets, there are other applications and protocols using UDP as transport which DO have such congestion avoidance mechanisms built in. So there's an argument there for passing it through IMQ and processing it as we have always done.

    But - then add a "Prioritize UDP" box at the top which would remove UDP from the normal classes and place it into the highest class - or bypass IMQ altogether - may give a simple way for quickly improving VOIP and stuff without having to worry about the details ???

    in qos-settings:

    //<% nvram("qos_classnames,qos_enable,qos_ack,qos_syn,qos_fin,qos_rst,qos_icmp,qos_udp,qos_default,qos_pfifo,qos_obw,qos_ibw,qos_orates,qos_irates,qos_reset,ne_vegas,ne_valpha,ne_vbeta,ne_vgamma"); %>

    fom.qos_icmp.value = E('_f_qos_icmp').checked ? 1 : 0;
    +fom.qos_icmp.value = E('_f_qos_udp').checked ? 1 : 0;
    fom.qos_reset.value = E('_f_qos_reset').checked ? 1 : 0;

    <input type='hidden' name='qos_icmp'>
    + <input type='hidden' name='qos_udp'>
    <input type='hidden' name='qos_orates'>

    { title: 'Prioritize ICMP', name: 'f_qos_icmp', type: 'checkbox', value: nvram.qos_icmp == '1' },
    +{ title: 'Prioritize UDP', name: 'f_qos_udp', type: 'checkbox', value: nvram.qos_udp == '1' },
    { title: 'Reset class when changing settings', name: 'f_qos_reset', type: 'checkbox', value: nvram.qos_reset == '1' },

    In tomato.c

    { "qos_udp",V_01},

    in qos.c

    ipt_write("-A PREROUTING -i %s -j CONNMARK --restore-mark --mask 0xff\n", qface);
    +ipt_write("-A PREROUTING -i %s -j IMQ --todev %s\n", qface, qosImqDeviceNumberString);// pass everything thru ingress
    //ipt_write("-A PREROUTING -i %s -p tcp -j IMQ --todev %s\n", qface, qosImqDeviceNumberString);// pass only tcp

    #ifdef TCONFIG_IPV6
    if (*wan6face)
    ip6t_write("-A PREROUTING -i %s -j CONNMARK --restore-mark --mask 0xff\n", wan6face);
    +ip6t_write("-A PREROUTING -i %s -j IMQ --todev %s\n", wan6face, qosImqDeviceNumberString);// pass everything thru ingress
    //ip6t_write("-A PREROUTING -i %s -p tcp -j IMQ --todev %s\n", wan6face, qosImqDeviceNumberString);// pass only tcp


    if (nvram_get_int("qos_icmp")) {
    fputs("\n\t$TFA parent 1: prio 13 protocol ip u32 match ip protocol 1 0xff flowid 1:10\n", f);

    +if (nvram_get_int("qos_udp")) {
    +fputs("\n\t$TFA parent 1: prio 13 protocol ip u32 match ip protocol 1 0xff flowid 1:10\n", f);

    Does this seem correct, and:

    What should the params in the last line (in red) be for this to work?


    fputs("\n\t$TFA parent 1: prio 16 protocol ip u32 match ip protocol 17 0xff flowid 1:10\n", f);

    This removes outgoing UDP from my "Crawl" class but not from incoming Crawl class or Connections.


    I've used a different method, either pass UDP through IMQ or don't !

    ipt_write("-A PREROUTING -i %s -j CONNMARK --restore-mark --mask 0xff\n", qface);

    if (nvram_get_int("qos_udp")) {
    ipt_write("-A PREROUTING -i %s -p tcp -j IMQ --todev %s\n", qface, qosImqDeviceNumberString);// pass only tcp
    else {
    ipt_write("-A PREROUTING -i %s -j IMQ --todev %s\n", qface, qosImqDeviceNumberString);// pass everything thru ingress

    I'm posting a version for people to try soon, 1.28.7494, using this method. I labelled the button "No Ingress QOS for UDP".

    I also removed the "No Limit' selection from "rate" and "limit".
  47. tiomo

    tiomo Networkin' Nut Member

    Hi there,

    sorry I have been off the radar for a while. I was busy with relatives on the weekend and had a business trip abroad at the beginning of the week. I will be able to spend some more time on the weekend.

    What you've posted looks OK. I have my reservations about he 'prioritze UDP' option though. As you said it can be an easy way for people to make sure incoming UDP traffic doesn't get dropped but it can also be a source of problems if there is UDP traffic on the network which should actually not be allowed.

    Since UDP doesn't implement flow control as TCP does, dropping the packets is no guarantee that incoming UDP traffic will slow down. I think most UDP connections will implement some kind of feedback if data is not arriving as expected (just guessing here).

    My take on this is that I think we should put incoming UDP in the appropriate class (therefore also drop it if a higher class takes priority) and not ignore it altogether. The 'optimize UDP' option I would use with caution, it may cause mayhem if e.g. UDP is placed in highest class and incoming UDP traffic takes up bandwidth to the point where there is not sufficient bandwidth for the actual services using the service class.

    I have read up about the TC-atm patch, which I found most interesting. A real eye-opener indeed! I will definitely have to try it. If it really works as expected it would allow us to use more of our precious bandwidth. I don't expect it to be the 'silver bullet' but I have always wondered why the net bandwidth of my connection is lower than the actual reported connection speed. We could possibly have an option to use the atm calculation if the overhead is provided
  48. Toastman

    Toastman Super Moderator Staff Member Member

    Thanks for the reply. It is still working very well here with no real hiccups.

    Porter and I are both puzzled by the last 4 classes apparently having the same prio of 7. So we've all been looking to see if that's really a limitation. The available documentation is not very helpful, do you have any slant on this? Presumably Jon had a good reason for doing this but it would be better if it can be changed. Was it because he was playing with CBQ? [http://lartc.org/howto/lartc.qdisc.classful.html#AEN939 - refs to defmap and Prio]

    I also had a few PM's raising the following:

    The pie charts show the class percentage of the currently used bandwidth clients in the queue. This isn't actually very useful.

    The question is, can it be changed to show the overall pie as the Max Bandwidth set in QOS, so that a quick glance at the pie will show the current bandwidth useage and the most active class.

    The unused bandwidth would need to be shown as an appropriate color (white / black / ?)

    [Possibly with a checkbox to select either type of display?]

    Actually, this has been requested several times before in the different forums, and I think it's a pretty good idea.
  49. Porter

    Porter LI Guru Member

    I just found out why the prio parameter won't accept values greater than 7. The maximum number of prios gets defined in the source and the default value is 8. Don't get confused, as always the counting starts from 0, so 0 to 7 equals 8 prios, or levels of priority.

    The file responsible for this is /release/src/linux/linux/include/linux/pkt_sched.h

    This is true for HTB and CBQ:

    I'm not implying that this value needs to get changed now. But it might be useful to hink about the practical value of more prio-levels. Does it make sense? Right now it might be useful to change the value to 10 and tweak the part of Tomato which is generating /etc/qos to not start at 1 but 0, because right now there are only 7 priorities. Otherwise the GUI would need some changes.

    What might be useful too is to think about whether we need Tomato's QoS-System to be a bit more fine tuned. HTB (_hierarchal_ token bucket filter) offers the possibility to build classes with traffic levels. Right now Tomato has one hevel. I haven't looked into the practical value of this either, but it might proof useful in some areas.

    This could be really usefull:
  50. Toastman

    Toastman Super Moderator Staff Member Member

    At the moment I'm interested in making sure that priorities work in those last classes - for example, my class 8 is P2P and I would like it to get less priority than class 7 which is used for file transfers, at the moment it's got the same priority, I assume. We always assumed that each increase in class no. got less priority than the last.

    So presumably there's no actual reason why we can't increase no. of prios TC_HTB_NUMPRIO - that's great, at first it seemed to be implied that there was a "hard" limit of 8 in the LARTC guide.

    Seems like we should change it I reckon.

    Re the extra complexity of levels, I'm not sure it would help us much. And it may introduce extra overheads onto our little routers. Discussion?
  51. Toastman

    Toastman Super Moderator Staff Member Member

    Tiomo, we've been playing a little with the outgoing prios - see posted git code. Increased the prio number to 10 and changed a little of the code to get what seemed to be the correct prio for the classes.
  52. alfred

    alfred Networkin' Nut Member

    Just something minor.

    I noticed that inbound/outbound pie charts persist in there, while the QoS is disabled from the other browser window elsewhere.

    It will not disappear until the browser window is refreshed while the 3 seconds refreshing spin still keeps on running in the bottom.

    I'm sorry, this should not be posted here,
    please help to move it to the thread "Toastman Releases 1.28.7493/4", thanks.

    Attached Files:

  53. Toastman

    Toastman Super Moderator Staff Member Member

    That's normal behaviour. The browser doesn't have any way of knowing that you stopped QOS using another window.
  54. Toastman

    Toastman Super Moderator Staff Member Member

    Here's a snapshot of the new QOS ingress in operation on a 16Mbps ADSL line. Max incoming bandwidth is set to 15000, and as you can see, the QOS system is now limiting at 15,000 to prevent congestion.

  55. alfred

    alfred Networkin' Nut Member

    Please think about this:

    The browser needs not to know whether the QoS is disabled or enabled.
    The pie charts must be matched to the left side percentage table accordingly.
    Also, three pie charts must be matched to one another.

    And, all the ten 0% classes will not give a total 100% result.
  56. teaman

    teaman LI Guru Member

    Hi alfred!

    I don't mean to be blunt/indelicate (or anything), but... I feel tempted to bring something that's sorta... been there since the very first versions of the Tomato web UI:

    It was never designed considering the user would be keeping multiple browser windows open and/or refreshing stats/data on some of the simultaneously! :)

    Let's just try to keep in mind that back in the days this whole thing got started, people were trying to run Tomato in devices with very few resources, limited to just a few MHz of CPU power and just 16M of RAM to handle everything from WAN/LAN/wireless, firewall, routing, Qos, the webUI itself, etc... (in fact, I'm still using a pair of WRT54GLs to do most of my development/testing to this day - devices quite under-powered when compared to some of the most recent/modern devices coming out these days ;) ).

    With all that in mind... I just wanted to ask: why is it so important for the page/webUI running on another window to almost instantaneously detect when some other feature had just been disabled/enabled? Why not just assume we're using and accessing this wonderful firmware that has been so elegantly designed and extended over the years, but can still run just fine on the very same hardware it was designed to run in the first place? ;)

    Anyways - I'm not saying what you're asking from the webUi is not reasonable - it's just that... such thing might not be exactly on the top of the TODO lists of the developers/volunteers out there...

    (still - I want to thank you for bringing this up - in fact, we rely on yours and everyone else's feedback, daily - thanks!)

  57. alfred

    alfred Networkin' Nut Member

    Hi, Teaman,
    Thanks for the explanation.

    First, I must declare for Tomato, and for you all the developers.
    All my routers are Tomato installed, and I always recommend my freinds to use it and help them for the installation.

    Yes, I totally agree with this is a wonderful firmware, I like it very much and really enjoy. It was immediatedly Tomato flashed when the new routers were brought home and in the first power on, I even didn't know what the GUI of its original firmware looked like.

    Second, I am not a trouble maker, and don't want to be under the impression that what I had posted will sink lower the morale and fighting power of Tomato, and all of your great efforts.

    Generally, you love it more, you will be exacting more, that's it. I just posted straight what I had seen, and hope it would be helpful.

    Third, I agree with you, that is not something very important. It can be proved by what I had started with: "Just something minor". I also found and was sorry for that I had posted in this thread improperly.

    Then, about your question, yes, I alway use multiple windows, one for the UI operation, the others for the observation. (again: not trouble maker!) I just want to know what and how the features can do for me. And the most important is, with these observation windows, I check if the setting changes I have made was correct, and see if it was matched expectedly before and after. This helps me very much for the settings. It is inconvenient to change the browsing pages back and forth within a single window, especially when it is checking some timing charts and history comparison.

    Now I know it is not designed for multi windows, we should not expect the observation windows will respond simultaneously (although I found most of them are working fine), but how about the next auto-refreshing? Now I would like to know, is my method not applicable?

    Finally, I will learn to ignore something that is minor, if you don't like it. And so will not disturb the major marching forward.
    If you felt unpleasant/annoyed by what I had posted, I apologize, again. But please understand that I am always in good intention, no offense.

    Thank you again, I love you all the developers. :)
  58. teaman

    teaman LI Guru Member

    Cool - so I guess we're all pretty much in the same page - trying to do our best to help improving/enhancing Tomato ;)
    Thanks for your feedback (and also for bringing this up, in the first place).
  59. netWilk

    netWilk Networkin' Nut Member

    It would be great to keep this feature. I have a crappy connection that theoretically gives me 5 Mbits, but throttles down to 900 Kbits after about 20 MB of transfer. With the No Limit, I am able to keep my short web browsing transfers uncapped, so that web browsing is snappy, but control downloads/P2P as not to trigger the throttle (which buffers instead of dropping packets, making it all bad).

  60. Porter

    Porter LI Guru Member


    could you be a bit more specific on the details of your ISP limits, please?

    Does every single connection get 20MB@5MBit and gets throttled to 900KBit then or is the whole link throttled to 900KBit after it has seen 20MB this day/week/or whatever?

    Is this a satellite connection?
  61. netWilk

    netWilk Networkin' Nut Member

    It's a fixed wireless, Motorola Canopy based system. The throttling is based on transfer on the entire link.

    Best way to describe is the system provides x base transfer slots (at 900 Kbits) + y extra transfer slots (which permit transfer up to 5 Mbits). There is enough y slots for about 25 MB of continuous transfer at 5 Mbit. After they are used up, they regenerate. I think it takes about 30 sec of no transfer to get them all back. They will still regenerate if the transfer rate is above 0 but below 900 Kbit, just more slowly.
    Basically rechargeable burst.
  62. tvlz

    tvlz Addicted to LI Member

    Comcast also has Powerboost , it would be great if you could find some way to support that.
  63. Porter

    Porter LI Guru Member


    _if_ Powerboost's behaviour is foreseeable to the point where we can expect it to give _every_ new connection this boost (even if it's only http this shouldn't be a problem) then that's already in the QoS-system. Look at both the filters that mark http (port 80). There is one filter for 0-512KB which puts this traffic in the WWW class which is fast and the other filter with the 512 - unlimited will classify each connection with the matching connection data inside the download class, which gets a lower priority and can get a lower bandwidth. So you would just have to tweak the settings. At least this is what I think right now.


    if the Canopy system acts differently than the Powerboost system then this probably is rather difficult, because the Tomato QoS system would need to know the rules that decide how much data can be send at a specific point of time. Somebody would have to modify the source and emulate this. This might never happen.

    There is some manual for the Canopy system, which I haven't found, yet. If I find it, I will probably read it...

    Feel free to contribute! :)
  64. Toastman

    Toastman Super Moderator Staff Member Member

    Question - there is a mechanism in Gargoyle firmware to adjust bandwidth automatically. Don't know if it would work for powerburst / canopy.
  65. netWilk

    netWilk Networkin' Nut Member

    Try this one:


    Sadly, my iptables experience is very lacking
  66. Toastman

    Toastman Super Moderator Staff Member Member

    guys, TC_HTB_NUMPRIO has to be changed in these 3 files - I missed the last one. Thanks, Augusto!

  67. Porter

    Porter LI Guru Member

    I've tried the new QoS-system for a few days now and these are my observations/ideas for improvement.

    The guaranteed rates (the left values) for inbound and outbound need a check so that the sum of all the classes never gets higher than 100%.

    Low latency for higher classes especially VoIP/Game is something everybody wants to achieve. For this to happen, the rate (left value) has to be higher than the actual anticipated bandwidth this class will ever see. So for instance when usin a VoIP-phone which uses approximately 128kbit per line, the rate should probably be as high as 160kbit or even a bit more. The underlying cause for this is that as long as a class is higher than any other class and hasn't reached its guaranteed rate (aka isn't overlimit) it will get to send its traffic.

    An overall idea which refers to the discussion above is general usability. Using QoS is difficult, especially for unexperienced users. If somebody wants to use VoIP, but doesn't know the bandwidth it uses, he will probably run into problems. So maybe there should be put in a few lines of text to help and give some hints how much bandwidth some typical traffic needs. I've thought about displaying warnings when the rates of some classes are too low to function, but then again it's possible to change the names of the classes and they could be meant for something completely different. Maybe it's time for a normal user view and an expert user view setting. But I'm just a bit brainstorming and that's just something that could be thought about in the long run. For now I think there should be at least some hints and maybe a warning for people who have entered very low values for inbound and outbound, so low that they wouldn't be able to use VoIP for instance.

    The Skype-L7 filters should be deactivated by default or at least become the last filters to be applied, because they match bittorrent, http and even git! So most of the traffic should be already classified when the rest of the traffic hits the skype filters. What I've read so far is that Skype doesn't want to get filtered. For VoIP/Video over Skype to work well the best way seems to configure Skype to use a specific port and then make a filter accordingly.

    Another thing about the filters is that there are too many. There don't have to be two filters for http and an http-proxy. This could all be one filter and make it less cluttered.

    I'm using a K24 build for my WRT54GL. I've noticed that ICMP prioritization doesn't work when enabled. Here's how I found out: I limited my p2p-class to 100KBit which is well below my line capabitilites and loaded some torrents. When the speed reached 100KBit I started 100 pings on google. The latency was between 500-900ms. Then I put a new filter for icmp in and made it the second filter that put everything in the service class. After that my pings where around 30-40ms. This means that at least on my router ICMP ends up in the default class and moreover the small packet prioritization probably doesn't work either. Can anyone confirm this?

    I've noticed another thing and maybe I should open a new thread for this: I've been using this version of Tomato http://www.linksysinfo.org/index.php?threads/speedmod-with-tc-atm-qos-patch-for-adsl.31541/ and now switched to the new Toastman builds. For some reason they are noticably slower. Logging in via ssh doesn't take 2-3s anymore, but 5-7s. Loading webpages has gotten slower too, even with QoS deactivated and no other traffic on the line. I've measured the times and it takes 1.5 to 2 times more the time than before to load a webpage. Because everything seems to be affected I suspect some change in the kernel. Can anyone confirm this?

    That's all for now! Happy testing!
  68. Toastman

    Toastman Super Moderator Staff Member Member

    Agree with most of your comments.

    The rules in my builds, by the way, are intended to serve as examples, each one is documented so that people will see what it is for. Previously, when the rules were combined, there were simply too many PM's asking for help to cope with, which is why I separated them out. In fact on my RT-N16 at 1/16Mbps downlink, it doesn't seem to make any significant difference in speed how many rules are in use. Since some lucky people are now using routers at 100Mbps speeds, that will no longer be the case for them. They should use the fewest rules they can get away with, I guess, and can also combine rules where necessary. At those speeds QOS probably isn't so important as for us mere mortals :D

    Ref. the allocation for the rate. Interested in that observation because I have always believed that a VOIP call would initially be allocated a bandwidth of e.g. 5% and when it reaches that limit, and needs more, then the bandwidth would be increased to the specified limit for that class, when it would stop. And that always appeared to happen here. So I just quickly connected remotely to 3 businesses that run Tomato QOS for Linksys/Belkin VOIP phones / adapters, and since I know they work really well, I looked at their router's (all RT-N16's on sub 512kbps/10Mbps lines) QOS settings. Actually each one was set low - 5% on two of them and 15% on the third.

    I've moved the L7 filters down. We'll revise these rules together and see if we can improve them a little but still leave them understandable as examples.
  69. jives

    jives Networkin' Nut Member

    I hope that I am posting this in the right thread (I don't want to clog up the developer discussion, but I think this has something to do with the latest development for ingress QoS).

    Yesterday I updated my E4200 to Tomato v1.28.0495 (MIPSR2-Toastman-RT-N K26 USB Ext). It is working great in principle, but I noticed an odd behaviour:
    I have a rule to match HTTP(S) downloads: TCP/UDP, Dst Ports 80,443,8080 and 512+ KB transferred. I have no incoming BW limitation for this class, but an outgoing BW limitation (~6MB/s). As soon as I start a download (tested with wget on one machine, Firefox on another), the rule is applied as expected, but the incoming connection is limited to 6MB/s. The connection shows up in the "Incoming" as well as in the "Outgoing" pie chart.
    If I turn off QoS, the download saturates my link as expected. The outgoing BW limitation seems to work just fine (tested with and iperf upload).

    I have to say that I am quite new to QoS and probably just misunderstood how (ingress) QoS works, but I did not expect the outgoing BW limitation to limit a HTTP download. Am I doing something wrong here?

    P.S.: Keep up the great work - it is truly amazing what you guys (and everyone else involved in the development of Tomato) have already achieved!
  70. Porter

    Porter LI Guru Member


    that's indeed a bit unexpected. I'd suggest that you post some screenshots of QoS/Basic Settings and QoS/Classification. Thanks for testing this so thouroughly and getting back to us.
  71. jives

    jives Networkin' Nut Member

    I had a nice reply lined up and formatted, when I thought of another test I could run - and I think found the problem. The E4200 is just not powerful enough to handle the new QoS at that throughput: Even with the limit set to 100% and very close to my measured speed and the CPU frequency set to 533MHz it doesn't reach more than 7.5MB/s download. With the standard clock rate it caps at around 6.5MB/s, which was coincidentally exactly what I set as a upload limit for that class; hence the confusion.

    When I turn the QoS off, I am able to saturate my downlink.

    I didn't notice that behavior in earlier builds, and I'll try and downgrade to a pre-ingress-QoS build to confirm (I assume ingress QoS requires more CPU time?).
  72. Porter

    Porter LI Guru Member

    Wait, you actually overclocked your router for this? Please stop overclocking, there has to be another error somewhere. This router should easily handle much higher bandwidths. I have a WRT54GL, which only runs at 200MHz and it worked just fine with around 5MBit. Btw, you mean MBit, right?

    What is your line speed?
  73. jives

    jives Networkin' Nut Member

    No, I really mean MByte ;) My linespeed is 100/100 Mbit (measured a tad over 100Mbit down and 94Mbit up).
  74. Porter

    Porter LI Guru Member

    Hmm, even 7,5MB/s is probably not what you will be happy with.

    Tomato's QoS-system uses iptables-filters to match traffic which is slower than pure tc filters and as far as I know L7-filters will slow down the most. If you can live without L7-filters you should disable them.

    Generally we will have to test how the amount of filters and the amount of L7-filters will slow down the router. Maybe you can run some tests if your time allows it.

    Apart from that we are still testing the QoS-system as it is. Implementing the faster tc-filters that are less accurate but sigificantly faster might be an option. But right now that's not on the agenda. But you can always write your own shaping script with tc-filters...

    There is a thread that investigates speed differences between Tomato versions:
  75. Porter

    Porter LI Guru Member

    I've taken the time to build a new L7-filter for youtube, since it has been broken for several years now.

    I'm hereby calling this a beta-test for this new filter and encourage everybody to give it a try.

    I'm not an expert on regular expressions. But I have tried several patterns and optimized them with the help of the benchmarking scripts that come with the layer7 package (details below).

    If you are just here to get the new pattern-file and are not a user of Tomato then put a file with the name "youtube-2012.pat" in /etc/l7protocols/extra with this content:

    # HTTP - Youtube over HyperText Transfer Protocol (RFC 2616)
    # Pattern attributes: good fast fast subset
    # Protocol groups: streaming_video document_retrieval
    # Wiki: none.
    # Copyright (C) 2012 by "Porter" on http://linksysinfo.org/index.php?threads/qos-development-thread.31886/#post-204861
    # Usually runs on port 80
    # This pattern has been roughly tested and seems to work well. It's a Beta!
    # To get or provide more information about this protocol and/or pattern:
    # http://linksysinfo.org/index.php?threads/qos-development-thread.31886/#post-204861
    # If you use this, you should be aware that:
    # - It's a beta! Pattern overmatches slightly, meaning it not only matches youtube traffic.
    #  The crossdomain-part matches small HTML documents.
    # - Benchmarks show that this pattern is fast.
    # - Obviously, since this is a subset of HTTP, you need to match it
    #  earlier in your iptables rules than HTTP.
    GET (\/videoplayback\?|\/crossdomain\.xml)

    For all the Tomato users (Not the devs! The devs put the above file under .../router/layer7/protocols/extra):

    If we want to test them, we'll have to do the following:

    The directory with the l7-protocols resides in a part of our router that is read-only, so the first thing we'll need to do is to get a writable copy of this directory:

    Enable the JFFS-partition and format it.

    Connect to your router using putty.

    Copy the folder with the L7 patterns to this partition:
    cp -R /etc/l7-protocols/ /jffs/l7-protocols
    Delete the old symlink and make a new one:
    rm /etc/l7-protocols/
    ln -s /jffs/l7-protocols/ /etc/l7-protocols
    Enter the new directory:
    cd /jffs/l7-protocols
    Now we need to create the new pattern file:

    cat > youtube-2012.pat
    (don't do anything else now!)

    Paste this text into your putty-window and hit ctrl+d:
    GET (\/videoplayback\?|\/crossdomain\.xml)

    After this go to QoS/Classification. If you already had the page open make sure to reload it.

    Now you can select the new pattern (youtube-2012) and make a new filter with it!

    Group this filter together with the other Media-class ones. Keep in mind that this filter has to be above any filter concerning port 80!

    Don't remove the old "Flash Video, (Youtube)"-filter. It still catches most of the flash-stuff that's not youtube.

    That's it, we are done! Please report any problems.


    Now some techincal stuff for people who are interested in writing their own patterns. As the developers of the layer7-package mention in their documentation there are some performance quirks with the librarys. To compare and test performance you can use the tools provided under /testing. Just read the README. I would recommend to use the doallspeeds-script. Let me warn you: if you want to just test your pattern, move the other patterns out of the directory or rename them. A full test with all the patterns took about an hour.

    Now an example of the performance quirks (benchmarks under Linux Mint 12 on an Intel Core i5-2520M):

    This pattern

    (GET \/videoplayback\?|GET \/crossdomain\.xml)
    is terribly slow with a userland time of 8.38s and kernel time of 426.77s (!)

    Just a small change to:
    GET (\/videoplayback\?|\/crossdomain\.xml)
    makes it blazingly fast in comparison:
    userland time of 8.99s and kernel time of 6.10s!

    I've looked at other examples and noticed that often a pattern that looks complex, isn't really slower than a pattern that looks very simple. The best way to find a good pattern is to use the doallspeeds-script and just run tests. I've made the benchmarking tests of all the other patterns public so that you can compare them and find better patterns more easily: http://pastebin.com/CRiB87Hi .

    Have fun testing!
  76. CardinS2U

    CardinS2U LI Guru Member

    this youtube rules doesn't work anymore. Does anyone have an updated versoin of this?
  77. Porter

    Porter LI Guru Member

    I just checked: this still works for me.

    Is this filter not working for anybody else?
  78. shadowken

    shadowken Networkin' Nut Member

    Still works for me too .

    Sent from my GT-I9300 using Tapatalk
  79. Marcel Tunks

    Marcel Tunks Networkin' Nut Member

    Works for me.
    @CardinS2U How is it being classified? Do you have any new rules above the YouTube L7 rule? Are you running BW limiter, VPN, or any other additional services that you weren't running before?
  80. Derbo

    Derbo Reformed Router Member


    It seems to be broken for me. I had this working on my E4200V1. I just upgraded to a N66R router using this firmware:
    Tomato Firmware v1.28.0503 MIPSR2Toastman-RT-N K26 USB VLAN-Ext

    I've reset the NVRAM and still have no luck.

    I used the default QOS rules that comes with the firmware with the exception of adding a port 22677 to FTP rule for my server.



    It only seems to be with youtube content. I am stumped.
  81. Porter

    Porter LI Guru Member

    Are you using Youtube over HTTPS = encrypted? L7-filters need to be able to look into the packets, which they cannot do, if the traffic is encrypted.
  82. Derbo

    Derbo Reformed Router Member

    Ahh yes it is. Darn. Thanks! I am unable to force everyone to use HTTP on youtube so I might have to live with this. Is there a workaround possibly?
  83. Porter

    Porter LI Guru Member

    I always thought that Youtube wouldn't use encryption as default so there shouldn't be a problem or am I wrong about that?
  84. shadowken

    shadowken Networkin' Nut Member

    You're right !
    May be this is a facebook videos traffic or other websites using encrypted HTTP (HTTPS) ... Youtube just uses HTTPS when sign in to the website but when watching videos it uses HTTP .

    In other words , The youtube pattern that Porter made is still working perfectly for any website which don't use HTTPS .

    Sent from my GT-I9300 using Tapatalk
    Last edited: Apr 3, 2014
  85. Spyros

    Spyros LI Guru Member

    In my case Youtube is being classified as FileXfer because the L7 filter doesn't work in IPv6 :(
  86. shadowken

    shadowken Networkin' Nut Member

    Why don't you use IPv4 instead of IPv6 if you have problem with it ?
    Disable IPv6 and check what happens with L7 filter whether it's classified correctly or not ?

    Note : I still don't use IPv6 yet so I don't know what kind of behavior it could be with QOS .

    Sent from my GT-I9300 using Tapatalk
    Last edited: Apr 3, 2014
  87. Spyros

    Spyros LI Guru Member

    as a workaround i made a rule with tcp dst port 443 512+ kb to match youtube video since we rarely use ssl file transfers.
  88. Eri

    Eri Serious Server Member

    I just realized, it doesn't work for me anymore.
    Somehow Youtube over HTTPS = encrypted, it uses port 443 instead of 80.

    Would there be patch of this to fix?
    Last edited: Dec 14, 2014
  89. Porter

    Porter LI Guru Member

    The filter does still work it just doesn't work for youtube any longer.

    The reason behind this is that youtube encrypts its traffic, but for the filter to work, the filter needs to be able to look inside the payload of a packet. Encryption prevents that. Unfortunately this also means that I cannot fix the filter.
  90. shadowken

    shadowken Networkin' Nut Member

    Do you use IPv6 in your home network ?
    The layer7 isn't supported for IPv6 yet ...
  91. Eri

    Eri Serious Server Member

    I don't use IPv6, just use IPv4.

    Thanks Porter for your concern.
    Then I treat youtube as normal http instead.
  92. Porter

    Porter LI Guru Member

    I'd like to talk about this post of mine a bit more: http://www.linksysinfo.org/index.ph...rial-and-discussion.28349/page-12#post-247479

    In the last few days I did some digging and as @koitsu in http://www.linksysinfo.org/index.ph...rial-and-discussion.28349/page-12#post-250169 already noticed, this is QUIC - a protocol developed by Google to speed up communication with webservers.

    After watching some videos about QUIC this seems to be quite an interesting development. To try out QUIC you'll need to use Chrome and enable it in chrome://flags . After this all the Google pages (including YouTube) will be delivered over QUIC.

    QUIC also encrypts by default, so there is no use in designing new L7-filters to detect video streams. But this brings me to my next topic: changing the default QoS config because of the rise of encrypted web traffic.

    Here's what I'd like to propose:
    1. Separate the web traffic into unencrypted and encrypted traffic.
    2. Add filters for QUIC.
    3. Separate HTTPS and QUIC filters merely for statistical reasons.
    4. Put the encrypted filters in front of the L7 filters.
    5. Prioritise HTTPS and QUIC traffic a bit by using the Messenger class for it so that undetectable YouTube videos get a slight boost. Make sure that the Messenger class can get up to 90% inbound traffic. I'm not sure how negative the side effects of this will be. At least at home I was finally able to watch YouTube with less stutter again.

    Here are the changes for QoS/Basic:
    Optional: rename the Messenger class to HTTPS+Msgr
    Inbound: 10-90%
    Outbound: 5-100%

    Here is how it looks like for QoS/Classification:
    [delete all the old filters for the ports 80, 8080, 443]

    The list:
    [unchanged stuff in front]
    TCP, Dst: 443, WWW, 0-512 KB, HTTPS
    UDP, Dst: 443, WWW, 0-512 KB, QUIC
    TCP, Dst: 443, HTTPS+Msgr, 512+KB, HTTPS
    UDP, Dst: 443, HTTPS+Msgr, 512+KB, QUIC
    [here are the L7-filters]
    TCP, Dst: 80,8080, WWW, 0-512 KB, HTTP, HTTP Proxy
    TCP, Dst: 80,8080, FileXfer, 512+ KB, HTTP, HTTP Proxy

    What do you think, @Toastman?
  93. Toastman

    Toastman Super Moderator Staff Member Member

    I'm trying it out, will reply later!
  94. Toastman

    Toastman Super Moderator Staff Member Member

    Well, I can not see any QUIC traffic (youtube or any google stuff) - and I have 3 pc's in the house with it enabled. And no QUIC from anyone on any of my apartment blocks. Yet Chrome does show my youtube movie as an active QUIC session, however, I see no UDP 443 packets from this machine.

  95. mvsgeek

    mvsgeek Addicted to LI Member

    Just for testing purposes, I modified rules 26 & 27 (Skype) as follows :

    26 : UDP, Dst: 443, VOIP/Game, 0-512 KB, QUIC
    27 : UDP, Dst: 443, Media, 512KB+, QUIC

    I then fired up a YouTube video with QUIC disabled, and it was classified as TCP, 443, FileXfer (Rule 37).
    With QUIC enabled, the same video was split between the new QUIC rules 26 and 27. Non-YouTube videos are still classified as TCP, 443, FileXfer.

    This was on a test RT-N12 running Tomato Firmware v1.28.0503 MIPSR2Toastman-RT-N K26 Std.
  96. Toastman

    Toastman Super Moderator Staff Member Member

    I wonder if Google's experiments are country-specific?

    Chrome is currently showing

    ID Source Type Description
    23047 QUIC_SESSION s.youtube.com
    23194 QUIC_SESSION s.youtube.com
    23319 QUIC_SESSION www.youtube.com
    23389 QUIC_SESSION i.ytimg.com
    23459 QUIC_SESSION s.youtube.com
    23549 QUIC_SESSION s.youtube.com
    23751 QUIC_SESSION ajax.googleapis.com
    24098 QUIC_SESSION s.youtube.com
    24101 QUIC_SESSION r4---sn-5np5po4v-c33e7.googlevideo.com
    24180 QUIC_SESSION r4---sn-5np5po4v-c33e7.googlevideo.com
    24228 QUIC_SESSION s.youtube.com
    24240 QUIC_SESSION r4---sn-5np5po4v-c33e7.googlevideo.com
    24304 QUIC_SESSION r4---sn-5np5po4v-c33e7.googlevideo.com
    24376 QUIC_SESSION s.youtube.com
    24521 QUIC_SESSION ajax.googleapis.com
    24822 QUIC_SESSION s.youtube.com

    But all connections are TCP
  97. Toastman

    Toastman Super Moderator Staff Member Member

    No, I am wrong. I had a test rule for port 443 UDP that I had left there from the last time. Now it is working as expected.

    So far, no sign of any noticeable improvement. But that's probably because youtube is normally pretty solid here anyway. But it does no harm, I will keep the rules active. However, none of our users will enable QUIC, as of course they aren't even aware that it exists, so it is only my machines that will show up. At some point Google may enable QUIC for all Chrome users.

    Also, I have moved HTTPS/Msgr up above Mail, which is less important than the video streams.

    This is my experimental setup:
    nvram set qos_orules="0<<-1<d<53<0<<0:10<<0<DNS>0<<-1<d<37<0<<0:10<<0<Time>0<<17<d<123<0<<0:10<<0<NTP>0<<-1<d<3455<0<<0:10<<0<RSVP>0<<-1<d<9<0<<0:50<<3<SCTP, Discard>0<<-1<x<135,2101,2103,2105<0<<<<3<RPC (Microsoft)>0<<17<d<3544<0<<<<-1<Teredo Tunnel>0<<6<x<22,2222<0<<<<2<SSH>0<<6<d<23,992<0<<<<2<Telnet>0<<6<s<80,5938,8080,2222<0<<<<2<Remote Access>0<<-1<x<3389<0<<<<2<Remote Assistance>0<<-1<x<1220,6970:7170,8554<0<<<<4<Quicktime/RealAudio>0<<-1<x<554,5004,5005<0<<<<4<RTP, RTSP>0<<-1<x<1755<0<<<<4<MMS (Microsoft)>0<<-1<d<3478,3479,5060:5063<0<<<<1<SIP, Sipgate Stun Services>0<<-1<s<53,88,3074<0<<<<1<Xbox Live>0<<6<d<1718:1720<0<<<<1<H323>0<<-1<d<4380,27000:27050,11031,11235:11335,11999,2300:2400,6073,28800:29100,47624<0<<<<1<Various Games>0<<-1<d<1493,1502,1503,1542,1863,1963,3389,5061,5190:5193,7001<0<<<<5<MSGR1 - Windows Live>0<<-1<d<1071:1074,1455,1638,1644,5000:5010,5050,5100,5101,5150,8000:8002<0<<<<5<MSGR2 - Yahoo>0<<-1<d<194,1720,1730:1732,5220:5223,5298,6660:6669,22555<0<<<<5<MSGR3 - Additional>0<<-1<d<19294:19310<0<<<<5<Google+ & Voice>0<<6<d<6005,6006<0<<<<5<Camfrog>0<<-1<x<6571,6891:6901<0<<<<5<WLM File/Webcam>0<<-1<x<29613<0<<<<5<Skype incoming>0<<6<x<4244,5242<0<<<<1<Viber TCP>0<<17<x<5243,9785<0<<<<1<Viber UDP>0<<17<x<3478:3497,16384:16387,16393:16402<0<<<<5<Apple Facetime/Game Center>0<<6<d<443<0<<0:512<<3<HTTPS>0<<6<d<443<0<<512:<<5<HTTPS>0<<17<d<443<0<<0:512<<3<QUIC>0<<17<d<443<0<<512:<<5<QUIC>0<<-1<a<<0<skypetoskype<<<1<Skype to Skype>0<<-1<a<<0<skypeout<<<-1<Skype Phone (deprecated)>0<<-1<a<<0<youtube-2012<<<4<YouTube 2012 (Youtube)>0<<-1<a<<0<httpvideo<<<4<HTTP Video (Youtube)>0<<-1<a<<0<flash<<<4<Flash Video (Youtube)>0<<-1<a<<0<rtp<<<4<RTP>0<<-1<a<<0<rtmp<<<4<RTMP>0<<-1<a<<0<shoutcast<<<4<Shoutcast>0<<-2<a<<0<rtmpt<<<4<RTMPT (RTMP over HTTP)>0<<-1<a<<0<irc<<<5<IRC>0<<6<d<80,8080<0<<0:512<<3<HTTP, HTTP Proxy>0<<6<d<80,8080<0<<512:<<7<HTTP, HTTP Proxy File Transfers>0<<6<d<20,21,989,990<0<<<<7<FTP>0<<6<d<25,587,465,2525<0<<<<6<SMTP, Submission Mail>0<<6<d<110,995<0<<<<6<POP3 Mail>0<<6<d<119,563<0<<<<7<NNTP News & Downloads>0<<6<d<143,220,585,993<0<<<<6<IMAP Mail>0<<17<d<1:65535<0<<<<8<P2P (uTP, UDP)"
    nvram set qos_irates="5-100,2-20,5-100,10-90,20-90,5-90,5-70,5-100,5-30,1-1"
    nvram set qos_orates="5-100,5-30,5-100,5-70,5-70,5-70,5-70,5-100,5-30,1-1"
    nvram set qos_classnames="Service VOIP/Game Remote WWW Media HTTPS/Msgr Mail FileXfer P2P/Bulk Crawl"
  98. Porter

    Porter LI Guru Member

    Ah, giving the HTTPS/Msgr class a higher priority might be a good idea!

    If it doesn't hurt today, it will hopefully be beneficial in the future. Let's see how this develops. :)
  99. xdrag

    xdrag Addicted to LI Member


    I've been using gargoyle on a secondary site and really come to love the way the QOS system is setup for gargoyle.

    Is there a possibility of a hybrid QOS system incorporating the strengths of the gargoyle QOS? (i.e. ACC - intelligent congestion control, max WAN utilization - allowing for maximum transfer until congestion becomes an issue)

    I feel with the these added features will significantly the current system

    I believe toastman is aware of these features

  100. Porter

    Porter LI Guru Member

    I'm aware that Gargoyle has such a system and I would certainly like to know how well it actually works. But that's only part of the equation. Somebody would have to code this for Tomato and I don't see anybody voluntaring to do that...

Share This Page