QoS mis-identifying traffic

Discussion in 'Tomato Firmware' started by bmupton, Feb 27, 2013.

  1. bmupton

    bmupton Networkin' Nut Member

    Hello all,

    Running Shibby Big-VPN 106 on an E3000 (Which I believe uses Toastman's default QoS rules) and I noticed a couple of things:

    1.) NNTPS traffic is being tagged as L7 RTP traffic
    2.) Netflix is being tagged as File Xfer

    For #1, it appears it does this on SOME downloads but not others. For example, I run a small usenet index on my LAN just for fun, and the traffic it generates is classified properly (I haven't noticed it not being classified as nntps anyhow) but when my PC running sabnzbd downloads something, it gets classified as L7 RTP at times and properly as NNTPS on port 563 at other times. I am assuming it would be safe to move the rules for NNTP up the list so they are triggered first.

    For #2 I assume this is because Netflix traffic is using port 443 and it doesn't trigger any other rules before that point.

    To be quite honest, this QoS ruleset is probably overkill for my purposes, but I figured I'd alert the community to my finding.

  2. Toastman

    Toastman Super Moderator Staff Member Member

    Moving the rule is the best idea. Thanks for the info.
  3. bmupton

    bmupton Networkin' Nut Member

    Unlike large installations like you maintain, I actually want to prioritize Netflix traffic at home...

    Any suggestions you have that might help? Adding all the IP ranges Netflix is known to use seems unfeasible to me, wondering if there's a better way.
  4. Porter

    Porter LI Guru Member

    If you can't tell netflix to use a specific port, your only option is to design a new l7 pattern. Unless somebody already made one. A very short search on google didn't yield any results, but you might have more motivation... ;)
  5. kthaddock

    kthaddock Network Guru Member

    ip firewall layer7-protocol
    add comment=NETFLIX name=NETFLIX regexp="^.*(host|HOST).+(netflix).*\$"
    add comment=YouTube name=Youtube regexp=\"get /videoplayback[\\x09-\\x0d -~]* http/[01]\\.[019]"
    add name="Netflix Alternate" regexp=\"^.*(get|GET) /s.+(wma|wmv|token|random|p=).*\$"
    add comment=Hulu name=Hulu regexp="add name=rtmp regexp=\"^\\\\x03.+\\\\x14.+\\\\\x02.+\\\\x07.(connect)\\\?.+(app)\\\?\""
  6. bmupton

    bmupton Networkin' Nut Member


    I'm fairly obtuse feeling tonight. Where do I add what you've posted? I'm normally pretty smart, but for some reason I'm not getting this tonight.

  7. koitsu

    koitsu Network Guru Member

    These regexes look completely, entirely messed up. In fact, they are. You've got escaping of quotes when escaping was being used on some of the lines for line continuation. For example, you turned this:

    add name="Netflix Alternate" regexp=\
        "^.*(get|GET) /s.+(wma|wmv|token|random|p=).*\$"
    Into this:

    add name="Netflix Alternate" regexp=\"^.*(get|GET) /s.+(wma|wmv|token|random|p=).*\$"
    And that's completely entirely wrong. What you SHOULD have turned it into was:

    add name="Netflix Alternate" regexp="^.*(get|GET) /s.+(wma|wmv|token|random|p=).*\$"
    Or possibly this (keep reading):

    add name="Netflix Alternate" regexp="^.*(get|GET) /s.+(wma|wmv|token|random|p=).*$"
    Next I don't know why $ (end-of-line) is being escaped, unless the issue is that someone was pasting this into a Linux command line somewhere and $ was being expanded. It's silly.

    Next, whoever wrote these regexes has absolutely no bloody concept of how to write them. For example:

    ^.*(get|GET) /s.+(wma|wmv|token|random|p=).*\$
    Is the *exact same thing* as:

    (get|GET) /s.+(wma|wmv|token|random|p=)
    There is no point to using beginning-of-line (^) and end-of-line ($) anchors when what, respectively, succeeds them or precedes them is dot-star (.*). I see people doing this all the time in the working-class world and it shows that they do not understand regex. It's a waste of CPU time, and this matters immensely with things like L7 filtering.

    And finally, the above example regex would apply to more than just """Netflix""" per se. All it's matching against is a web browser issuing this in its HTTP request:

    GET /s{one or more of any character}{strings "wma", "wmv", "token", "random", or "p="){anything else}
    So visiting a URL like:


    Would therefore match "Netflix Alternate", because the web browser would issue this HTTP request:

    GET /sellers/products?p=LargeTrousers HTTP/1.0
    People who do not understand how regex works, or how to write things accurately, should not be writing L7 filters (not you, I'm talking about the guy in that thread you referenced).
  8. Monk E. Boy

    Monk E. Boy Network Guru Member

    Traffic on your LAN won't be classified by QoS because QoS only applies to traffic flowing through the router and out the WAN port.

    You likely just need to add the additional ports your NNTPS servers are using to the existing NNTP/NNTPS rule. You may have more luck if you change the "Dst Port" to "Src or Dst" bit too.

    Netflix is being classified as File Xfer because when any HTTP/HTTPS transfer exceeds 1MB it gets bumped down to File Xfer. If this bugs you then remove the 0-1024KB from the initial rule so all HTTP/HTTPS traffic gets classified at a higher priority. And while you're editing it, make the rule only apply to TCP traffic instead of TCP/UDP.
  9. bmupton

    bmupton Networkin' Nut Member

    Hi Monk,

    The Usenet index I run grabs headers and a few kb of the binary from my Usenet provider, so it does us QoS for that, and when it's doing its job, those packets are properly identified as NNTPS (At least, I've never noticed that traffic being classified incorrectly). When my downloader is downloading from Usenet, however, it's getting classified as RTP in some cases, and properly as NNTPS in others (I haven't identified the difference yet though). The L7 regular expression for RTP must be triggered by some types of downloads from Usenet, but not others.

    At any rate, it was more a heads up for Toastman, as that may actually cause a problem in his installations if file transfers are getting wrongly classified as media. I just moved my NNTP rules above the L7 RTP rule and it's classified properly all the time now.

    I'm not too concerned about the Netflix thing, since I control all the devices on this network I can do QoS the old way and shut them off...

  10. Monk E. Boy

    Monk E. Boy Network Guru Member

    Oh I see, sorry, I thought you were talking about local traffic to the index server. And I forgot that the default rules have some L7 filters above some port-based filters.

    I put all my port-based rules on top and the L7 rules at the end, specifically to avoid cases like these, but also to decrease load on the router. L7 is a method of last resort that I really don't like depending on, so I normally end up with only a couple L7 filters in use.

    Glad its working. One constant in networking is that you have to tailor the installation to the environment, there's no such thing as one size fits all...
  11. JustinChase

    JustinChase Networkin' Nut Member

    I had this same issue, and this also worked great for me. My SABnzbd/usenetserver downloads are all going to FileXfer as they should. I have given this class up to 100%, but it sees to be giving other classes the bandwidth as it should, so I'm pretty happy so far. Thanks for the tip!
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice