1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Using QOS - Tutorial and discussion

Discussion in 'Tomato Firmware' started by Toastman, Dec 24, 2008.

  1. Toastman

    Toastman Super Moderator Staff Member Member

    Using QOS to control Data Flow in Residential Buildings

    Tomato QOS Tutorial


    I've moved a number of posts here from odd places on the forum so this can become a QOS thread with a particular slant on large installations. While the article is concerned with a large number of users in big premises, obviously, if the QOS will work here, it will also work (and probably even better) for you - as a normal home/standalone user. So this thread is useful for everybody! Now - when I say it will work - I don't mean it will be optimum in every case, but it will give you a base to start. You will see how I have used the different classes for the various protocols and modify them if they aren't right for your setup. It's up to you to decide what to do and how to change it to suit your needs. But to do that you do need to understand how QOS works and the reasoning behind the rules.

    I should point out the difference between a standalone user or one with a couple of family members and a residential building. These lucky "standalone" people have control over each PC and know what applications they are running. Whereas in a residential building, we have no idea what people are running on their PC's and we have no access to, and no control over them at all. So the only thing we can do to prevent one or two users or applications from hogging our valuable bandwidth, is to set up the router's QOS system in a way that prevents it happening. If you are a standalone user, it is often much easier to change what is happening on the PC than it is to try to use a QOS rule to control it afterwards, but we simply don't have that luxury. If you have a family you may have the same problem at home.

    Now, the comment about even a "basic" home setup. People often think they have a "basic" setup. That they browse the web and nothing more, so a single rule to prioritize port 80 HTTP is all they think is necessary. But they fail to understand that almost every single web page has links to other pages, flash videos, advertisers, scam sites, online video, links to messenger, facebook, photo sharing sites. Not to mention commercial sites such as Google that spy on our computers (Often more than 20 of them on a single web page). Some of these are secure connections so that involves other ports. Some play music - now we have other streaming protocols. They also use Windows update service and usually Messenger - which itself uses several protocols and many ports.

    That is why QOS rules tend to get quite complex after several months of hard use, even for a home system with a couple of users. A single user is OK, he knows what he is doing. Add another user, and immediately one of them gets annoyed when the other gets his windows update or downloads his email !

    So, we use QOS for a variety of reasons. What I have to do in residential buildings is to KEEP THE SYSTEM RUNNING when a hundred are so people are all trying to use it at the same time. To do that it is often necessary to limit some types of traffic, this is a trade-off. How much you need to do this depends on your own system. That's why you need to understand it and tweak the settings yourself.

    While reading this series of article it is important to remember that they were originally separate posts - some of which have now been bundled together, so please forgive any repetition or duplication of information.

    This link is a useful place to find answer to a lot of common problems http://www.linksysinfo.org/forums/showthread.php?t=63486


    The author has been involved in setting up WiFi in several large residential blocks, where it was important that the result not only worked but was simple to maintain by reception staff. What was achieved has surprised many people here, including myself.

    Ever sat in an internet shop, a hotel room or lobby, a local hotspot, and wondered why you can't access your email? Unknown to you, the guy in the next room or at the next table is hogging the internet bandwidth to download the Lord Of The Rings Trilogy Special Extended Edition in HDTV format. You're screwed - because the hotspot router does not have an effective QOS system. In fact, I haven't come across a shop, hotel, or an apartment block that has any QOS system in use at all. Most residents are not very happy with the service they [usually] pay for.

    So what is "QOS" ??

    A "QOS" (Quality Of Service) system is a firmware strategy used in a router connected to the internet gateway to allow it to give priority to those applications which are important. Without it, anarchy rules, and the downloader will usually wreck the internet access for everybody else.

    The normal systems installed in hotspots and residential buildings use a simple router with no QOS, running splash screen and access portal software, and a bunch of AP's nailed to the walls. The user often has to buy a card with an access code, and somebody makes heaps of money out of administering the access controls. Unfortunately, the actual web access is so slow and congested as to be unuseable, the router regularly fails, and everyone in the block is angry and feels cheated.

    It doesn't have to be like this!

    Almost all normal SOHO (small office/home) routers have no real way to prioritise applications and make sure that P2P downloaders do not take over. However, some routers which happen to run Linux as an operating system can use third-party firmware (software) to turn a cheap lump of plastic into something akin to a professional router. All for around 50 - 100 dollars! And hotspot owners, cafes, hotels can also use them to provide a superior WIFI system to that which they currently have. That firmware, is called TOMATO - and it was written by Jonathan Zarate and subsequent developers have been adding to it ever since.

    It is quite easy for residential block owners to install and run a system themselves, with the benefit that the web access works well, and they don't have to pay anyone for a third rate access control service. And best of all - it doesn't have to be prohibitively expensive :biggrin: A side benefit of installing what is in effect a wireless network covering your building, is that you can also use it for other purposes. For instance, I also have a 32 camera security system online.

    Now, you don't have to use expensive equipment. The Linksys WRT54GL is adequate for most purposes. We aren't aiming to supply ultra-high-speed internet to all users, and ADSL lines from 2 to 5 Mbps are available easily and cheaply in most countries. This will provide adequate service for most users. 8 and 16Mbps lines will be better, but not so much as you might think! Most users will never see any difference. A router with a bit more memory is useful and more stable, try to get the ASUS WL500gP v2 (32MB RAM, 8MB Flash). Even better - if you can get the WRT54G-TM, which is a router that also has 32MB RAM and 8MB Flash, and also runs nicely overclocked to 250MHz. It's faster than the ASUS WL500gP v2, the wireless is better. and would be a better router for this application.

    Faster and better routers will become available as time goes by, but we do need to be able to run Tomato on it. Tomato, a third party firmware which uses Linux, is the secret of getting this stuff to work properly in an apartment block using cheap hardware. On a WRT54GL clocked at 250Mhz, 1,000 mixed connections, but mostly P2P, usually results in a CPU load of about 20%. At this level, it's still fast.

    JANUARY 2010

    The ASUS RT-N16 router is now available in most counties, it is clocked at 480MHz and has 128MB of RAM. Teddy Bear is the first to port Tomato over to it - and even the first "beta" is stable. Keep an eye on this thread http://www.linksysinfo.org/forums/showthread.php?t=63587 From now on, it would be best to use this for the main router and WRT54GL for AP's. There seems little point in type "N" AP's unless they operate on the 5GHz band, due to interference problems. A "G" 54Mbps connection is going to be the standard for some years yet, and for many reasons will be the best solution.

    You'll need more access points, just use more WRT54GL's and set them up as AP's wired with CAT5e cable to your main router, via switches if necessary. For God's sake don't try to do it with WDS. There is a very severe speed and reliability penalty even with a single WDS connected AP, with a couple or more you will be lucky to download anything this century.

    If you wish to use the network in your building for other purposes too, such as office, security cameras, then it might be a good idea to use gigabit switches, otherwise at the moment they aren't necessary and are more expensive.

    You may find cheaper AP's but the twin external antennas on the WRT54GL's and the ability to set higher transmit power have been an advantage for me. The additional information given by using Tomato firmware on the WRT54GL even when used as an AP is an invaluable tool for faultfinding.

    This is the easy part of the setup. The rest is up to you to get right and maintain.

    Tomato firmware has the most effective and configurable QOS of any SOHO router around. If you have a real need for QOS to control multiple users, you will find DD-WRT etc. almost useless.

    The secret of a successful residential system is the ability of Tomato's QOS to allow you to actually share your ISP's service between all of your clients, hence the title of the first article. And the methods used here can and will work for anyone, what will work for a large residential system should work just about anywhere else, just modify to suit your needs.

    Now, a warning - you'll find some people tell you that you cannot do this job with a small SOHO router, with or without Tomato firmware, because there are too many users. Please don't think about the number of users because it doesn't matter. The overall throughput is limited by the connection to your ISP and it makes no difference if you have one user or 100, as long as the firmware can handle the overall number of connections and throughput. Tomato makes this possible. In fact, many of my colleagues have been replacing their business Cisco routers with routers running Tomato, because they are just too difficult for them to administer.

    Let's begin by making some things a little clearer for newcomers to Tomato.

    "Incoming" versus "Outgoing" QOS

    Unfortunately many posts on the subject of QOS confuse people, especially newcomers, into misunderstanding what the router's QOS is, what it is NOT, what it is used for, and what it can really achieve if understood and used properly. QOS runs on the router connected to the internet gateway or ISP. It works on the WAN (Wide Area Network) port or Internet port.

    NOW - let's get this straight. There isn't a "QOS for Upload" or a "QOS for Download" in Tomato :wall: Tomato's QOS system operates on outgoing data, but it also has class limits on incoming data which can be used to drop packets, and cause link stabilization, at those class limits. We use both of these as part of QOS.

    This ongoing battle seems to arise from the fact that the QOS system operates on outgoing traffic. Therefore, many people do not understand how it can manipulate the situation to control INCOMING traffic. So they confuse everyone by swamping the forums with comments like "QOS doesn't work" and "the Incoming QOS is rubbish" - etc. This actually makes me extremely angry, because it is just not true. If it were true, then none of the people in the apartments I administer could use the internet. So throughout these articles you will find warnings to disregard posts by such persons. I'm an engineer, I believe in things that work, and only if they work.

    QOS would actually be of no interest whatsoever to us unless it helped us with our incoming data flow. It really doesn't help to look at it as either "incoming" or "outgoing" QOS. Those people who keep insisting that because QOS only works on outgoing traffic (uploads) then it's useless, are missing the whole point. I must stress this, because there are hundreds of people making stupid statements like this in the forums and unfortunately, too many people believe what they are saying. These people are spreading misinformation, based on ignorance. You CAN control incoming data to a great extent, but there's no "magic button". You have to learn how to do so.


    and also, adopt this philosophy:

    Router QOS is best viewed as an OVERALL strategy for improving your flow of data.

    So HOW does the router's QOS work, how does it make any difference to incoming traffic - if it only acts on the outgoing data?

    Well, it's actually very simple.

    Take this analogy. Suppose there are a thousand people out there who will send you letters or parcels in the mail if you give them your address and request it (by ordering some goods, for example). Until you make your request, they don't know you and will not send you anything. But send them your address and a request for 10 letters and 10 parcels and they will send you 10 letters and 10 parcels. Ask for that number to be reduced or increased, or ask (pay!) for only letters and no parcels, and they will do so. If you get too much mail, you stop sending the requests or acknowledgements until it has slowed down to a manageable level. Unsolicited mail can be dealt with by ignoring it or by delaying receipt (payment) and the sender will send less and give up after a while.

    In other words, you stop more goods arriving at your house by simply not ordering more goods!

    If you have letters arriving from several different sources, you stop or delay sending new orders to the ones you don't feel are important.

    That's it!

    The amount of mail you receive is usually directly proportional to the requests you send. If you send one request and get 10 deliveries, that is a 1:10 ratio. You've controlled the large amount of deliveries you receive with only the one order which you sent. Sending 1,000 requests at a 1:10 ratio would likely result in 10,000 letters received - more than your postman can deliver. So based on your experience, you can figure out the ratio of packets you are likely to receive from a particular request, and then LIMIT the number of your requests so that your postman can carry the incoming mail. But if you don't limit what you ask for, then the situation quickly gets out of control.

    It's not a perfect analogy, sure, but router QOS works in a similar way. You have to limit the requests and receipts that you send - and the incoming data reduces according to the ratio you determine by experience.

    The problem is you can have no absolute control what arrives at your PC - because your router does not know - and can never know - how many packets are in transit to you at any given time, in what order, and from what server. The only thing your router can directly control is what you SEND, see what comes back, and then respond to it. And the QOS system attempts to influence your incoming data stream indirectly by changing the data that you SEND in much the same way that you can control incoming mail simply by reducing your demand for it.

    That is the whole purpose of the router-based QOS systems, and that is why it they have been developed, not merely to control uploads! However, you can't just check a magic box marked "limit all my P2P when I am busy with something more important" - you have to give clear instructions to the router in how to accomplish your aim. To do this it is necessary to understand how to control your incoming data by manipulating your outgoing requests, class priorities, and receipts for received packets. Added to this we also have the ability to shape traffic by using bandwidth limits on outgoing total traffic, and also on the incoming individual traffic classes. Then we have to also consider UDP packets (rather less easy to control) and how to effectively control applications that use primarily UDP (VOIP, Multimedia etc). Depending on your requirements that may take hours or months to get working satisfactorily.

    Something has to be said, so I'll just come right out and say it:

    The default QOS rules in Tomato are almost completely useless and should be immediately changed.

    The worst problem is the feeble attempt at classifying P2P. P2P cannot be classified by the means shown - assuming that it will be using ports 1024-65535. Neither does using IPP2P or L7 filters (as used in most SOHO routers as the usual way to magically "LIMIT ALL P2P". That is just advertising BS for the nice glossy box. The only way that does work, is to set a default class, (I use class and then delete ANY rule pertaining to P2P. Then address everything that you REALLY want to use on your system by placing them in higher classes. Now, anything that is NOT addressed by one of your rules, will bypass them and end up in your default class D. This will include P2P, and anything else that you don't want.

    Next you have to define the limit rules for your DEFAULT class as mentioned above, so decide what you actually want to do with P2P. Usually we want to permit some but prevent it from hogging the bandwidth. As an example, set outgoing rate and limit to 1% and 5%. Set the incoming class limit at 50%. Now you should see it throttled. After this, you can adjust it to suit your own needs.

    The rest of these articles will expand on this, and show you how to effectively control your traffic.
    wuhanren and Natey2 like this.
  2. Toastman

    Toastman Super Moderator Staff Member Member

    Setting QOS rate and limit

    Setting Your Limits

    A look at QOS rate/limit settings with special reference to P2P Traffic - and why QOS often fails to work properly..

    Before we begin, let’s define a few things and clear up some confusion.

    QOS stands for "Quality Of Service". It's aim is to provide an improved level of service for critical applications. The term “QOS” in engineering circles refers to the marking of packets for critical applications so that they can be easily identified - and given priority by every internet router between your PC and the remote server. Hence providing a guaranteed “quality” of service necessary for some applications like VOIP to work without jitter, dropouts, and delays.

    The SOHO router's simple "QOS" system doesn't do this, and a somewhat different method is implemented in the router to make sure that our most critical applications are accorded at least some priority by the link. Actually, it is little more than traffic shaping used in conjunction with a priority system. But nonetheless, we can still make it work for us provided that we understand how to do so.

    The router QOS system attempts to ensure that all important traffic is sent to the ISP first, and then tries to control or "shape" other traffic so that the higher priority incoming data is not delayed.

    Packets from your PC will be “inspected” and compared with the router’s QOS settings in order to find out if they need priority, and then assigned a place in the outgoing queue waiting to be sent to your ISP. Other mechanisms may also be used to manage the traffic so that the returning data from the remote server is delivered before that which is less important.

    But someone has to define a set of QOS rules for a particular environment. Hey Dude - that's YOU! :poke:

    If you are a standalone user with one PC then you probably don't need QOS at all. If you are a P2P user and wish to download at absolute maximum speed, you will usually find QOS counter-productive.

    The worst problem faced by all of us in multi-user environments is P2P traffic, which tries to take all available bandwidth. Hence, most discussions of QOS operation refer to P2P when giving examples of traffic control. We normally give P2P a low priority because most people want to browse online websites - and the P2P traffic slows their web browsing down.

    The faster your ADSL line, the better your system will work, the more P2P you can allow on your network, and the better your VOIP and games will work. This is because of two things - firstly, obviously the overall speed improves. Secondly and more important, it is more difficult for P2P applications to actually generate enough traffic to fill the pipe. Overall, everything becomes less critical. The assumption here is that you will have between 1Mbps and 5Mbps at least, with 512k uplink. Later in the thread, I will post any new QOS rule improvements to reflect higher speeds than this.

    If you have a small network of 2 or 3 PC's then you may benefit from QOS, but it doesn't have to be too complicated. But if you have a larger network, similar to mine, which are large apartment blocks with about 250-400 rooms and maybe around 600-1200 residents, then QOS is essential. Without it, nobody will be able to do anything. Just a single P2P user will often ruin it for everyone else. However, the rules for correct QOS operation are the same for large or small networks - but you must decide for yourself how complex you want your rules to be, what applications running on your PC's you need to address.

    In a large block like mine, you have to try to cover everything, so your rules need a lot of thought. What we do is of the utmost importance if we want things to work properly, because if we screw up, everyone is dead in the water. Unfortunately, that means a very steep learning curve. It's also important to keep an open mind, and to understand that if a set of rules don't work, there is a reason - that reason is usually that you have overlooked and failed to address a particular set of circumstances.

    The QOS in our router can only operate on outgoing data, but by “cause and effect” – this has a significant influence on the incoming data stream. After all, the incoming data to our router is what our QOS is *really* trying to control. QOS works by assigning a priority to certain classes of data at the expense of others, and also by controlling traffic by limits and other means - so as to enable prioritised traffic to actually get that priority.

    Since UDP operates in a connectionless state, the main methods used by our router to control traffic involve manipulation of TCP packets. UDP, used for VOIP, IPTV applications, can't be controlled as such, but it can be helped by the reduction of TCP and other traffic congestion on the same link. In fact, some kinds of UDP traffic can be a huge drain on resources - and we will place a limit on it to prevent it from swamping our router.

    We would usually like to allow WWW browsing to work quickly, and get our email, but aren’t too bothered about the speed of P2P – for example. In the event of huge amounts of traffic occurring which is too much for our bandwidth limitations, we also have to control the maximum amount of data which we attempt to send or receive over those links. This is called “capping”, “bandwidth limiting” or “traffic management”. This is also managed by the QOS system in our router and is a ***part*** of QOS.

    So, once again a reminder - we must not refer to "incoming" or "outgoing" QOS. All of these mechanisms are PART of the "QOS" system on the router.

    Now to the Nitty Gritty

    Now, let’s have a look to see why many people fail to get QOS to work properly or at all, especially in the presence of large amounts of P2P. The default rules in Tomato are almost useless - if better than nothing. So let's improve on them.

    Firstly, let’s start by making the statement that “slow” web sessions are usually due to “bottlenecks” – your data is stuck in a queue somewhere. Let’s first assume that the route from your ISP to the remote web server is fast and stable. That leaves us with our router - which is something that we have some control over.

    We are left with two points commonly responsible for bottlenecks.

    1) Data sent by your PC’s, having been processed by QOS, is queued in the router waiting to be sent over the relatively slow “outgoing” uplink to your ISP. Let’s assume a 500kbps uplink.

    2) Data coming from the remote web server, in response to your PC’s requests, is queued at the ISP waiting to be sent to your router. Let’s assume a 2Mbps downlink.

    Bottleneck No. 1

    Our PC's can usually send data to the router much faster than the router can pass it on to the ISP. This is the cause of the first "bottleneck". However, we can just leave the normal TCP/IP mechanisms in the PC to back off and sort out the problem of data being sent to the router too quickly, and it will take care of itself. But there is now another function associated with the sending of data by your router - to the ISP, which is the key to QOS operation. Let me try to explain:

    The incoming/outgoing data is queued in sections of the memory in the routers - these are known as “buffers”. It is important not to let these “buffers” become full. If they are full, they are unable to receive more data, which is therefore lost. The lost data therefore has to be resent, resulting in a delay.

    The transmit buffer in your own router contains data waiting to be sent to your ISP. This is an extremely important function. There must be room to “insert” packets, so that it can be sent first - in order for QOS priorities to work properly. If there's no room to insert the data in the buffer, then QOS cannot work.

    If your PC('s) can be slowed down so that they send data to the router at a slower rate then your router is sending it to the ISP, you will see that there will always be some free space in the buffer. This is the reason I recommend you to set the “Max Outbound" bandwidth in QOS-BASIC to approximately 85%, or even less, of the maximum “real” (measured) uplink speed. This ensures that data in the transmit buffer is sent out faster than it is being filled back up, and hence leave room in the buffer - ready to received sorted data from the QOS system.

    I must stress that it is an absolute necessity that you set the outgoing limit at about 85% of the minimum bandwidth that you EVER observe on the line, and often even less. THIS IS NOT NEGOTIABLE! You must measure the speed at different times throughout the day and night with an online speed test utility, with QOS turned off, and no other traffic - to determine the lowest speed obtained for that line. You then set 85% of this figure as your maximum permitted outgoing bandwidth useage. Just because this seems low to you, don't be tempted to set a higher figure. If you do, then the QOS system will not work. In fact, you will get BETTER performance by setting a LOWER limit, this will be covered later.

    [EDIT - I have had a steady stream of mail from people who don't seem to understand English. "Minimum" does NOT mean "average".]

    When a maximum outgoing bandwidth limit is reached - packets from the PC's are dropped by the router, causing the PC's on your network to slow down by backing off, and to resend the data after a wait period. Note that this is actually "traffic shaping" between your PC('s) and the router. This takes care of itself and is only mentioned in passing. You don't have to do anything.

    [*** ADDITION - Since this article was first posted I have had HUNDREDS of people PM me to say they have followed my advice and QOS still does not work. When I follow this up I almost always find they have NOT followed even the basic advice above. They have neither set the Maximum Bandwidth limit to 85% (or less) of the measured speed nor have they set their limits, either class or incoming, as I advised. The moral is - if you don't do what I recommend, don't complain when it doesn't work !! ]

    Next, let’s consider QOS in operation.

    Imagine some unimportant data that you wish to send to your ISP, presently stored in the router's transmit buffer. As it is being sent, you might start up a new WWW session which you would prefer took priority. What we need to do is to insert this new data at the head of the queue so that it will be sent first. When you set a “priority” for a particular class, you are instructing the router that packets in certain class groups need to be sent before other classes, and the router will then try to arrange the packets in the correct order to be sent, with the highest priority data at the front of the queue, and the lowest at the back. This is quite independent of any limits, or traffic shaping, that the QOS system may ALSO do.

    Now, we are going to assume that we have defined a WWW class of HIGH with no limits. Let’s imagine the router has just been switched on, and we then open a WWW session. A packet (or packets) is sent to the remote server requesting a connection - this is quite a small amount of data. The server responds by sending us an acknowledgment, and the session begins by our requesting the server to send us pages and/or images/files. The server sends quite large amounts of data to the us, but we respond with quite a small stream of ACK packets acknowledging receipt. There is an approximate ratio between the received data and our sent traffic consisting mostly of receipts for that data, and requests for resends.

    Bottleneck No. 2 - The BIG ONE

    This relationship between the data we send and the date we receive varies with the applications and protocols in use, but is usually of the order of 1:10 or 1:20, but it can rise to around 1:50 with downloads and P2P connections. So an unlimited outgoing data rate of 500kbps *could* result in an incoming data stream of anything from 5 to 25Mbps - which would of course be far too much for our downlink of 2Mbps – and our data would therefore be queued at the ISP waiting to be sent to our router. Most of it will never be received – it will be “dropped” by the ISP’s router. All other traffic will also be stuck in the same queue, and our response time is awful. This is bottleneck no. 2 in the above list.

    It is vitally important that the incoming "pipe" never gets full. If you allow it to do this, you have lost the battle. At this point let me warn you about the many forum posters who insist that if your pipe isn't FULL there was no need for QOS in the first place. Please ignore them because they clearly don't understand what they are talking about. You will see why later.

    So how do you prevent this bottleneck? Well, you have to restrict the amount of data that you SEND to the remote server so that it will NOT send too much data back for your router to process. You have absolutely no control over anything else - you cannot do anything except play around with what you SEND to the remote server. And what you SEND determines what, and how much, trafffic will RETURN. Understanding how to use the former to control the latter is the key to successful QOS operation. And how to do that, you can only learn from experience.

    Let's go back for a moment to the analogy in the introduction:

    Suppose there are a thousand people out there who will send you letters or parcels in the mail if you give them your address and request it. Until you request it, they don't know you and will not send you anything. Send them your address and a request for 10 letters and 10 parcels and they will send you 10 letters and 10 parcels. Ask for that number to be reduced or increased, or ask for only letters and no parcels, and they will do so. If you get too much mail, you stop sending the requests or acknowledgements until it has slowed down to a manageable level. Unsolicited mail can be dealt with by ignoring it or delaying receipt and the sender will send less and give up after a while.

    The amount of mail you receive is usually directly proportional to the requests you send. If you send one request and get 10 letters, that is a 1:10 ratio. You've controlled the large amount of letters you receive with only the one letter which you sent. Sending 1,000 requests at a 1:10 ratio would result in 10,000 letters received - more than your postman can deliver. So based on your experience, you can figure out the ratio of letters you are likely to receive from a particular request, and then LIMIT the number of your requests so that your postman can carry the incoming mail. But if you don't limit what you ask for, then the situation quickly gets out of control.

    So, we have to understand how the amount of incoming data is influenced by what we send. Experience tells us that for some applications aproximately a 10% ratio of sent to received data is normal, while for others it can be less than 5% (esp. P2P). In the case of UDP packets, we have much less control. UDP operates in a connectionless state and no receipt is sent for incoming packets. Hence we lose the use of this mechanism for controlling the data flow. The ratio of transmitted to received data is not known with any certainty either. In a VOIP connection, for example, a large data stream will be initiated to your PC when the other person speaks, while your own computer sends almost no traffic at all unless you are also speaking. Therefore you can improve VOIP / IPTV by slowing your TCP connections to make room. You can control UDP by limiting the outgoing/incoming bandwidth of the appropriate class.

    To examine the effect of this "ratio" between sent and received TCP data in more detail we’ll use P2P – the real PITA for most routers. We will define a class of LOWEST for P2P with a rate of 10% (50kbps) and a limit of 50% (250k). Now we look at the result. The link starts sending at 50kbps and quickly increases to 250kbps outgoing data, and following the 5% ratio between send and receive, we get perhaps 5Mbps INCOMING data from the P2P seeders in response. That is far too fast for our miserable little downlink of 2Mbps, and is queued at the ISP’s router waiting for our own router to accept it. The downlink has become saturated. Any other traffic is also stuck in this queue. When most of these packets fail to be delivered, after a preset period of time they are discarded by the ISP’s router and are lost.

    As it does not receive any acknowledgement of receipt from our PC for the missing packets, the originating server “backs off” and resends the lost data after a short delay. It keeps doing this, increasing the delay exponentially each time, until the data rate is slowed down enough that packets are no longer dropped. It may take a long time to do this, but in theory, at least, eventually the link will stabilize. By looking at the “realtime” graph in Tomato, it is easy to see when your downlink is being saturated. The graph will “flat top” at maximum bandwidth, with very few and small peaks and troughs noticeable in the graph.

    Right - let’s see what we can do about this !

    There are some different mechanisms available for us to use which will have the effect of slowing down an incoming data stream. At first I will concentrate on the most important one, which would produce the best speed and response for other classes despite having several online P2P clients.

    1) Reducing outgoing traffic for a class.

    We drop the P2P class rate down to 1% (5k) and the limit to 10% (50k) - and watch what happens. The incoming data from the remote server(s) now also drops to 500kbps - 1Mbps (cause and effect). This is OK and fits within our available 2Mbps bandwidth downlink, while a simultaneous WWW session is still quite fast and responsive. However, this is a simplistic view, because the “5-10% ratio” is not *always* applicable, and high-bandwidth seeders may actually send you more data than expected, nevertheless it will still probably be within the 2 Mbps link speed. However, if you try to do better than this and increase the outgoing limit to 20%, it MIGHT still be OK – or it more probably might NOT, depending on the material being sent to you, the number of seeders, the number of connections open at any given time, and many other factors which all have an effect on the link. And at 20% the simultaneous WWW session starts to slow down, and is generally unresponsive as the link starts to saturate. So you really do need to err on the low side to be absolutely certain that the downlink does NOT become saturated, or the QOS will break. I will discuss the pros and cons of increasing this setting to enable us to download more P2P later. For the moment, stay with me.

    TO RECAP - It is quite likely that setting your outgoing P2P traffic limit to more than 15-20% will saturate your downlink with P2P, causing QOS to be much less effective. You have to decide on a compromise setting that allows higher P2P activity while still allowing a reasonably quick response to priority traffic like HTTP. [Shortly, we will see how to combine two methods to achieve this].

    Still, let’s set it to 20% (100k UP) and be optimistic - phew – everything’s still OK. But we’ve hit a snag already – especially with P2P applications :angry:

    Consider what happens, for example, when your P2P application needs to UPLOAD a lot of files in order to gain “credits”. Your PC uploads lot of data, perhaps quickly filling your “upload” allocation of 100k. BUT this class is shared with the receipts you are sending out in response to incoming files. These packets no longer have exclusive access to the router's buffers, and since they have no special priority in the queue, may be delayed. Now your downloads can no longer reach the normal speed - they may even drop down to almost nothing. At this point you might think there is something wrong with QOS. But QOS is actually working correctly, and it is your application of the rules that is in question.

    Your uploads have dominated the connection because you didn't anticipate what might happen. You allowed uploads to dominate your connection, when what you really wanted to do was to allow downloads.

    2) Limiting the incoming data rate of a class

    A partial solution can be achieved by using the “incoming” traffic limit in Tomato P2P class to set a limit on incoming P2P data. So how does this work? The connection tracking section of the router firmware keeps a record of all outgoing P2P packets and then attempts to keep a tally on any incoming packets associated with it. It can therefore add them all up and then calculate the speed of the incoming P2P, which can then be limited. So we could theoretically set an incoming limit of something under 2 Mbps. If this is exceeded, the router will drop packets, forcing the sender to back off and resend the data – once again allowing the link to stabilize. It is actually a form of crude congestion control. To better understand how the normal built-in backoff strategies of the TCP/IP protocols operate, you must use Google and read up primers on TCP/IP operation.

    This is, of course, the reason why a maximum incoming limit is sometimes recommended to be initially set in QOS/BASIC for rather less than the maximum “real” speed normally achievable from your ISP. It is an attempt to slow down the link before it becomes saturated. That is why it is often recommended to set to something LOWER than the maximum, usually 85% or so. However, if you run a busy network, you've probably noticed that in practice th is actually unable to keep the incoming data pegged low. This is partly because while the link is busily "stabilizing itself", new connections are constantly being opened by WWW, Mail, Messenger, and especially other P2P seeders, while other connections may close unpredictably, and that upsets the whole thing. The goalposts are constantly moving! You will see from this that P2P in particular is very difficult to accurately control. Over a period, the average should approximate the limit figure. Best latency is achieved with a combination of 1) and 2).

    If you want to see your QOS working quickly and with good latency, set the incoming limit low at around 66% of your ISP's maximum speed.

    Here is a graph showing the latency of a 1.5Mbps ADSL line under differing loads, and the result of limiting inbound traffic. If you use VOIP you must limit your incoming bandwidth to allow low latency. You can't have both low latency and a full pipe!!


    (graph thanks to Jared Valentine).

    It is important not to rely 100% on the incoming limit especially while you set up QOS. Set it only when all else has been adjusted and you can see if your outgoing settings are causing congestion. If you try to set up your QOS with incoming limits set, it will actually make it rather difficult for you to see what is happening as a result of your settings, because the limit will kick in and mask what is going on. Initially, it is useful to set the incoming overall limit to 999999 so that it is in effect switched off, this will make things easier for you while examining your graphs and adjusting your QOS parameters. But once your QOS rules are in place it ALWAYS pays to impose an incoming limit for many applications as well as an overall limit.

    Incidentally, there is a big difference in the class limits between 100% and NONE. 100% = 100% of your overall limit, NONE means ignore the overall limit.

    To recap - For best throughput and reasonable response times and speeds, set incoming limits quite high, near the inbound maximum limit if you wish. You can set NONE=no limit at all for an important priority class such as WWW browsing. For best latency, set incoming limits lower, see the graph below. I found 50% maximum limit to be extremely responsive, 66% good, 80% still reasonable but ping times beginning to suffer under load, and things dropped off noticeably after that. As a compromise, I use 80% for my maximum incoming limit, and most residents appear to be happy with the result.

    You sacrifice bandwidth for response/latency.

    In order for WWW to be snappy when using a restriction on other traffic, I usually set my WWW class to "NONE" so that it will ignore any limits.

    What about uploading seeds?

    We're always stuck with a problem with P2P on ADSL connections. We can't upload enough, especially on a shared link, to get enough "credit" or "points" to make our downloads go quickly. Apart from "leecher mods", there isn't really a lot you can do, so if you have complaining customers, then you better be good at explaining this. P2P really isn't very successful on shared "public" WiFi networks. In order to get what I call good downloads, I need to continuously upload at least 200k (not seeds - just outgoing P2P requests/receipts). This isn't usually possible in shared networks. Users can help themselves a little by limiting their seeds to the lowest possible setting (e.g. 1k) or switching them off altogether (if allowed in the client) so as to use what little uplink bandwidth is available for actually downloading files. Not many people appreciate that seeding is not necessary or desirable, if your aim is just to download files. [FYI, killing off all uploads in uTorrent (e.g. set upload=1kbps) while in a session will show an INSTANT increase in download speed.

    The net effect of this is: P2P users on shared networks who seed files not only screw up their own P2P operation but everyone elses too.

    ADSL isn't so good for people trying to use private trackers who insist on good upload ratios and limit your download speeds drastically if you don't seed files. Public torrents work OK but don't try to seed anything.


    What about when the available bandwidth is taken by people on your network uploading seeds who don't know or haven't taken the trouble to eliminate or at least limit them?

    Well, there would some justification for an extra rule to move "seed" uploads out of your P2P (default) class, to allow QOS to work on the downloads as a separate issue. Your normal P2P rule (which should be the default class) would then control only your downloads, and the extra rule controls the uploads. This is in theory. In practice, I haven't found a good way to differentiate seeds from normal P2P operation, so I no longer try to use class E for this. If you find a way that will work, please post it for us all to benefit.

    While on this subject of classes, do make proper use of the available classes to keep applications separated for clarity. For example, many people place WWW and DNS traffic in the same class. But this slows up DNS response. There are ten classes in Tomato from Highest down to E. There doesn't seem to be any noticeable performance hit involved with using them all. When you have split things into different groups it is much easier to see what is going on using the QOS pie charts and "View Details".

    Here's an example of how you might use them - replace with whatever suits your situataton:

    Highest---DNS, NTP
    High------Game Control Ports
    Medium---IPTV Control Ports (RTP, RTSP, etc)
    Low-------WWW, HTTPS, Web Proxies
    Lowest----Shoutcast/IPTV/Messenger Video etc. data streams
    A----------Mail POP, SMTP, IMAP
    B----------IRC/Chat/Messenger text
    C----------File Uploads/Downloads (HTTP) (FTP)
    D----------Default (P2P and anything unidentifiable or annoying!)
    E----------P2P Uploads/unwanted UDP/anything really annoying (as suggested above - a "crawl" class...)

    As you can see, most of the QOS settings interact with each other and don't have precisely defined points at which things can be seen to be “pegged”. You should perhaps think of the QOS system as a mechanism that "steers" your overall traffic distribution in the direction that you would like to achieve. There is nothing precise about it, no "quick fix" when you have many connections that you have to keep in their place. It may take you some days of staring at your monitor to properly evaluate the results of even a small tweak in your parameters!

    So does QOS really achieve it's aim? The answer is that it is never 100% successful but can do a “best effort” job which minimizes the delays and dropped packets. A point is reached where no further twiddling with parameters will provide an improvement - and the result, for better or worse, is what you’re stuck with. Compulsive twiddlers will eventually go insane trying to achieve the impossible :wall:

    One big piece of advice I have is for those who would like to make maximum use of their bandwidth for downloading P2P, keeping both inbound and outbound links pegged at close to 100%. Doing this will make your QOS slow and ineffective. Web (HTTP) response will not be snappy and users will complain about it. You cannot have it both ways. If you wish to do it, set up QOS as above and then just increase the amount of outgoing and incoming bandwidth available to the P2P class. You can often go up to 80% - Oh, and be absolutely sure to uncheck the "prioritize ACKS" box. As to how well it works - this is a matter of personal opinion. It does slow things down, which is unacceptable to me personally. It may be acceptable to you.

    (original graphs thanks to Jared Valentine)

    Attached Files:

  3. Toastman

    Toastman Super Moderator Staff Member Member

    Limiting numbers of TCP and UDP connections

    If your router crashes or becomes unstable due to P2P applications opening large numbers of connections, try to limit the number of ports that a user can open. Here is a collection of useful scripts: Put one or more of the following in the "Administration/Scripts/Firewall" box. Check that function before adding another rule. You may list the iptables rules by telnet to the router and issuing the command "iptables -L". If you are running a recent tomato mod, you can also do this from the "system" command line entry box.

    These are just things for you to try, they will be found in many places on the web. Wheteher they work for you, you need to test.

    #Limit TCP connections per user
    iptables -I PREROUTING -p tcp --syn -m iprange --src-range -m connlimit --connlimit-above 80 -j DROP
    iptables -I INPUT -p tcp --syn -m iprange --src-range -m connlimit --connlimit-above 100 -j DROP

    #Limit all *other* connections per user including UDP
    iptables -I PREROUTING -m iprange --src-range -p ! tcp -m connlimit --connlimit-above 20 -j DROP
    iptables -I INPUT -m iprange --src-range -p ! tcp -m connlimit --connlimit-above 50 -j DROP

    #Limit outgoing SMTP simultaneous connections
    iptables -I PREROUTING -p tcp --dport 25 -m connlimit --connlimit-above 10 -j DROP

    The next script is to prevent a machine with a virus from opening thousands of connections too quickly and taking up our bandwidth.

    #Limit UDP packet opens from all users - UDP to Router
    iptables -I INPUT -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT

    #Limit UDP packet opens from all users - UDP out to WAN
    iptables -I PREROUTING -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT

    QOS still works and show all of the affected users in the appropriate class in the graphs. If you test the above scripts with a limit of say 5 connections in the line, you will often see that it doesn't appear to be working, you will have many more connections than your limit, maybe 30-100, that you can't explain. Some of these may be old connections that have not yet timed out, and waiting for a while will fix it. Be aware that often these may be TEREDO or other connections associated with IPv6 (windows Vista, and 7) which is enabled by default. You should disable it on your PC by command line:

    set state disabled


    If your router becomes unstable, perhaps freezing or rebooting, apparently randomly, then it may have been asked to open too many connections, filling the connection tracking table and running the router low on memory. Often this can happen because poorly behaved applications (usually P2P clients) can attempt to open thousands of connections, mostly UDP, in a short space of time, just a few seconds. The router often does not record these "connection storms" in the logs, because it runs out of memory and crashes before it has time to do so.

    Obviously, there is a flaw in the firmware, which most definitely should never allow this situation to happen. Until such time as we can correct this situation, we must resort to some means of damage prevention and control. Setting the timeout value of TCP and especially UDP connections is necessary.

    Setting the number of allowed connections high (say 8192) makes the situation worse. In fact this number is almost never required. Most connections shown in the conntrack page will actually be old connections waiting to be timed out. Leaving the limit low, say 2000 to 3000 connections, gives the router more breathing space to act before it crashes.

    The following settings have been found to help limit the connection storm problem without too many side effects.

    None 100
    Established 1200
    Syn Sent 20
    Syn Received 20
    FIN Wait 20
    Time Wait 20
    Close 20
    Close Wait 20
    Last Ack 20
    Listen 120

    Unreplied 10 (25 has necessary by the odd guy, for some VOIP applications to work, otherwise reduce it to 10)
    Assured 25 (Sometimes this needs to be increased up to 300 for VOIP use. Choose the smallest number that is reliable for your own VOIP system)

    VOIP is a bit of a mess. If you have problems, there's actually quite a lot of help on the web. Google for it.

    10 for both

    ADDIT .. NOVEMBER 2009

    Teddy Bear is now compiling Tomato under a newer version (2.6) of the Linux Kernel. The "NONE" and "LISTEN" settings have been eliminated. There are two new settings, "GENERIC" and ICMP. The ICMP is self explanatory, the "generic" timeout which is used for all TCP/UDP connections that don't have their own timeout setting.
  4. Toastman

    Toastman Super Moderator Staff Member Member

    Another relevant thread here:


    More about outgoing "rate and limit"

    Now, let's take a look at a different scenario. Let us suppose that we had defined P2P in class D with an initial rate of 10% rising to 100% if no other class has priority. This is a very common thing for people to do, but it introduces a problem which many people don't think of.

    Suppose our hero P2P user gets some good seeds, or you have several users all determined to download the Lord of the Rings in record time. Nobody else using the router, so our upload bandwidth rises to close to maximum. Now, let's suppose you wake up, and try to access the morning newspaper online while swigging your coffee. OOPS - you've got no bandwidth :mad:

    Tomato has to do something. It will try to increase the bandwidth for your HTTP class, but it has to wait until some becomes free. There's a significant delay while tomato shuffles things around. In other words, an application in a lower class, if allowed to take a significant part of your upload bandwidth, CAN indeed slow down response time of higher priority classes. So don't set your rate TOO low....

    At the same time, incoming bandwidth due to the P2P is full or near full speed. There's probably a queue of P2P waiting at the ISP. And your newspaper headline page is stuck in the queue. [And of course, the longer the queue, the longer you have to wait for something that's stuck at the END of that queue. So the lower you set your incoming bandwidth, by definition, the faster the response time to those packets at the end of the queue becomes].

    Therefore, for best speed it is important that not only do you prioritize a class but you must also remember that the other, lower priority classes, can nevertheless have an effect on that class.

    You can see from this, that it isn't quite the cut-and-dried setup that we dreamed of. Everything is a question of "balance" and you have to get that right - and you have to do it yourself. That is why commercial routers with the nice big button marked "fix everything for me" don't actually work. If the slowdown is acceptable to you, then it's OK to let your lower classes consume a lot of bandwidth, but if the delay is unacceptable, then you must limit them - until it does become acceptable. That is usually around the 20% to 50% outgoing limit. Initially set 10% and then go upwards, 20% is usually very safe. I find 50% to work reasonably well most of the time while allowing a reasonable amount of bandwidth to be used by P2P customers.

    And you can prevent the big queue at the ISP by setting a limit on incoming P2P !

    You can't have it both ways. If you are a VOIP user or games player, or you want the best possible speed for browsing etc., you will not be able to let your P2P take all of your bandwidth just because nobody else is using it. Period.

    EDIT - Toastman Tomato now has an IMQ based ingress system, which has a rate and limit similar to the outgoing section. It allows much better control of bandwidth then ever before, and also sets a real overall incoming bandwidth limit.
  5. Toastman

    Toastman Super Moderator Staff Member Member

    Classifying applications

    It may appear relatively easy to classify an application, but that isn't always the case. FTP, for instance, is listed as using port 21. So do you just stick it in a rule and forget it? No. There is more to it than that!

    Some applications and protocols, FTP being one of them, use more than one port. Port 21 is actually a control port - the actual data transfer, upload or download, is usually done on a separate port 20, but occasionally on a port which is unknown and therefore cannot be easily classified.

    A variation of FTP known as "Passive FTP, uses port 80 to set up the link, using dynamically allocated ports for the data. You need to learn about this in order that you might try to address those ports. Priority based on a range of ports and size of the transfer may be necessary. You must research your applications carefully before you act.

    IPTV likewise has a control and setup port, but the data is usually carried on a separate port, often HTTP port 80! The connection tracker in Tomato has "helpers" for some applications such as FTP, GRE / PPTP, H.323, and RTSP, but this doesn't give any priority to the control or data channels - you have to do that yourself.

    In the case of IPTV using TCP over port 80, the downlink will get the priority that you use for HTTP. But you usually won’t be able to predict what port data links will be on. It may be possible to find some information on Google, but often the best you can do is to prioritise the control port. You might just try to cover most of the commonest data ports used by the popular applications like Messenger services. Often you won't ever know if you have really been successful. But at least, even if you are not successful, the worst that can happen is that the application falls through the rules into the default class, which it then has to share with your p2P and any other "bulk" traffic. If sometimes your rules work badly and some traffic ends up in a higher class than you intended, that may also not be the end of the world.

    You can control file downloads reasonably well by classifying any very large data movement as a file transfer, and stick it in an appropriately limited class. If you can’t make this work, then take comfort in the fact that luckily, most file transfers don't last very long, especially if they are done on HTTP port 80 - which usually has a high priority. And here, you can move it out of your WWW class by making a rule to treat any large outgoing data transfer on port 80 - say 512k plus, as a file upload - and again stick it in the appropriate class. (The default tomato rules which I hate, put it into "bulk" along with P2P).

    MSN and other messenger services can be very complex.

    Sometimes they are able to tunnel through firewalls using port 80. Some browser-based file downloads and video applications are done in this way. Audio and video applications often have ports used for opening the connection and for control purposes, but the data itself is carried on another port or even a set of ports in the range 1024-65535. Again, it isn't easy to classify them properly, and very hard to give them the necessary priorities. There are many services broadly referred to as Messenger - white board, file transfer, chat, video, VOIP. In an apartment complex it isn't necessary or desirable to cover them all. VOIP and personal video (webcams), for instance, we felt should not be given particular priority over other users, but if there is spare bandwidth then we try to accomodate it. Because we have a lot of foreign residents who like to watch news and TV from home, we give IPTV some priority when bandwidth is available. Actually, in practice, it seems to be fine, and most of the time there is no excessive IPTV traffic.

    Many video streaming protocols exist and have control and setup ports. Hence you may want a "Control" class near the top of the list to make sure they set up speedily and any control signals are given priority so that the application does not "stutter". Again, you may give priority to whatever you feel is important!

    Multicast TV has to be supplied as a service by your ISP – it is his decision whether to allow it on his network or not. It is not normally available. The “Enable Multicast” button in “Firewall” settings, is therefore best left unchecked as it rarely does anything useful. [For your interest, I hear that the IGMP in DD-WRT is also unuseable as it can transfer multicast onto your LAN, and completely swamp it!] It seems that the multicast protocol support in Tomato is broken anyway. So for the moment, I believe it is useless trying to use multicast with Tomato.

    UDP traffic is harder to control as the protocol is a connectionless one and no receipts are sent for incoming data. You can do a limited amount of shaping only. Priority can be given to UDP by reducing TCP and other traffic with QOS rules to give it increased bandwidth. Incoming limits on your other classes may be necessary to unsure that this happens.

    All my rules are port-based unless absolutely necessary, with some size variations set. IPP2P rules are very broad and not much use. L7 filters are likewise, but even slower and more processor intensive. Most of them don't work very well, and some don't work at all. Address and Protocol/Port are the fastest and most efficient ways to match. If at all possible, use Address and Protocol/Port before resorting to IPP2P or L7. UPDATE - There is a little good news - recent issues of the L7 filters have been much better and some are now useable (August 2010).

    For stability with high useage, it's best to completely avoid them. For example, four such filters when used on an ASUS WL500gP v2 slowed the router almost to a crawl. Too many L7 or IPP2P rules can also cause your router to crash or restart. If you are experiencing frequent crashes and restarts under heavy load, these may be the cause. If you have an ASUS RT-N16 router running at twice the speed. then use of a few L7 filters is much less lilely to cause problems.

    I have a mix of Linux, Windows, Apple MAC users, with all sorts of odd applications that need to be covered. When I see something new going on, I try to find out what it is and then consider whether it is covered by an existing rule, whether that is sufficient, and if not – does it require to be addressed, or just left to fall past the filters into the “default” class.

    Get yourself a few lists of common port numbers, and learn what each port is used for. Think carefully before making assigning it to a class, or does it need a new class? Do you need to cover it at all?

    QOS rules are processed from the top down. The first rule that matches will apply. You must be sure that all your rules co-exist, that the order is correct, or things will be siphoned off into the wrong class. It is not always obvious when this has happened. Think carefully and re-examine ALL your rules when you make a new one, or even alter an existing rule. Make sure that your new rule is in the correct place in the list.

    Especially, don’t try to make a "rule" for P2P - because you will fail. Note that the L7 filters are almost useless for P2P, and not so good for a lot of other things too (which is why the QOS on most SOHO routers doesn't actually work). Use the following strategy to deal with P2P and you won't need any special filters at all.

    Set your default class to something suitable for P2P - usually this will be the lowest or penultimate class. Don't be afraid to use classes A to E. Then just make rules for all applications that you want to give priority. Anything NOT in those rules, including P2P, will fall through them - into the default class.

    I used to set all small packets to get priority. That included ACKS - many people would recommend that this box is unchecked. If you do that, my thought was that you will delay traffic unnecesarily. However, there is a problem here which is not mentioned anywhere in Tomato FAQ's or Wikis. The "small packets" in the check boxes use the "Highest" class to prioritize them. So, if you check the ACK box, any ACK packets for e.g. P2P will move out of the P2P class into the "Highest" class. The data stream however will still be identified and correctly classed as P2P, and will still respond to limits. The problem may not be noticed if you have set up QOS for best latency by limiting outgoing P2P severely. But for most people, checking the box will slow down their QOS rules by giving an unfair advantage to P2P by effectively giving P2P downloads a high priority. This is because most outgoing traffic for P2P is actually ACKS.

    The moral of this is - if you run P2P on your network, and wish to limit it, uncheck the ACK box. If you want the best P2P speeds, check it.

    Don't use DHT or uTP (methods used by uTorrent and others where UDP packets are used for transfer of data) as it takes up a ridiculous amount of of bandwidth for no significant gain. Get your users to turn it off if you can. If this can't be done, make sure that you use firewall scripts to limit the numbers of UDP connections that can be opened per user. Bit Torrent DNA is similarly useless and spends a lot of it's time using your computer and link for someone else's benefit - Google this and see how it is screwing up your network before you kill it. A newer system called uTP is now being used by uTorrent, but so far it has also caused a lot of UDP traffic for no observed benefits at this location. Keep your eye on it to see how it turns out.


    DO get everyone to use UPnP or NAT-PMP. If P2P is allowed to run without any incoming ports being forwarded, it will take up a LOT of your bandwidth but users only get a very, very low download speed. Users aren't generally aware of this, and they will not know what is responsible, so some guidelines at the time the users signs up for internet (a user guide) is the best way to educate them.

    UPnP is often regarded as a security risk and many people are so paranoid they will recommend it is not used. That's your choice, but with no UPnP, most of your users' applications won't work. Is that acceptable? I think not. The simple fact is - if the residents can't use their PC's to do what they want, they will often move out. If it's any comfort, as far as I am aware none of our apartment blocks have ever had any security problems from using UPnP/NAT-PMP.

    Try to set up QOS so that bandwidth is not unduly wasted, but latency is still reasonable low. Do the best you can. To recap - set up a ping session to your ISP gateway on your PC. With ACK priority OFF, start with incoming P2P limit 50%, outbound P2P limit 20% and increase that slowly while watching ping times. You should be able to get downloads to around 80% of max bandwidth with response times still around 50mS. Experiment with outgoing and incoming limits to quickly gain experience. The higher your available bandwidth, the less critical your settings need to be, so for those lucky enough to have 16Mbps or higher, your task will be a lot easier than someone with 1Mbps.

    A lot of software does not always close UPnP forwarded ports on exit, P2P clients and sometimes Messenger are particularly bad for this. Tomato originally supported only 25 UPnP port forwards. If the table filled up, Tomato did not automatically delete entries. You had to cancel them manually or use a script to restart the service periodically which would reset all of the rules.

    A better option has been added to Tomato from February 2009. A rather nifty UPnP client called "miniUPnPd" has been integrated by modders. This UPnP daemon has no limit on the number of portforwards (excepting memory limitations) and also automatically releases unused rules after a configurable period. It also supports NAT-PMP (Apple's version of UPnP). This has now also been adopted by Jon Zarate in the official Tomato release.

    Online Games Enthusiasts

    A word to owners of apartment blocks. From time to time you will get problems with games players. No matter what you do, games will never work as well as if they had a dedicated line, more often than not they don't work very well at all. There are thousands of games and you can spend all of your time fiddling with the router QOS and forwarding ports every time someone changes his preferences. It just isn't practicable. It is very important to make games players fully aware of this fact and that they are sharing the line with many others. Human nature being what it is, though, they usually never stop bitching about everybody else on the network, so you may have to make it quite clear before anyone signs up. Fanatical downloaders also give a lot of trouble, it is common for them to want to take all of the bandwidth for themselves and they will accuse everyone else in the block of downloading so that you, the owner, will cut them off. Sorry to be blunt about it - games players are usually very immature and obsessive. Often if they can't play games they will lock themselves in their rooms and cry - we've seen dozens of those.

    Be firm and let the QOS deal with it, don't overreact. Please also take note - in our experience, disgruntled P2P and Games users will often roam the floors late at night attempting to find your AP's and routers and fiddle with them, often cutting cables and removing power to them, hoping they can get more speed or sometimes revenge against the management. Don't underestimate the damage they can do - this isn't a joke, it is just what we have experienced with several thousand users in our apartment blocks. You must always make sure your AP's and switches have nothing exposed that they can screw up, no exposed cable, and the main routers should be in locked steel cabinets with all cabling inside a conduit, and preferably in the reception area where the staff can see it.

  6. spicoli

    spicoli LI Guru Member

    Question: According to this post, which I hope was written according to the results of some experimenting, if you have a class set (lets go with Highest) to 100%-100%, QoS will filter said rule in its class to be "unhandled", as in it will use whats needed and will be immune to bandwidth reduction? Sorry if that came out worded wrong or horribly. Lots of turkey, drowsy enzymes kicked in.
  7. Toastman

    Toastman Super Moderator Staff Member Member

    Unclassified connections

    I get a lot of people ask about unclassified connections which they can't seem to get rid of.

    Any connection that terminates at the router, is not classified by QOS. For example, when you look at your router's web gui, you open connections to the router - usually about 10-20 or so will be seen.

    P2P is notorious for generasting masses of "unclassified" connections. You downlaod a file and at the same time notify the "cloud" that you have part of it for seeding. Machines will therefore know how to connect to your P2P client. But what happens when you disconnect? An external machine that tries to open a connection to, say, a P2P client on one of your machines that is no longer listening, will NOT BE classified as it is terminated at the router! Remote machines may continue to connect to your clients long after they have been switched off. You can't stop it - just don't worry about unclassified connections.

    NOW - a lot of people keep asking me why their QOS doesn't work between LAN machines. Please note that it is supposed to work on traffic between the router and the ISP - i.e. on the router's WAN port. NOT on the LAN.
  8. az2008

    az2008 Addicted to LI Member

    One of the downsides to uPnP is that you don't know what's opening ports. For example, I installed BitTorrent which, without me knowing, it installed something called "btDNA". This is something which is supposed to work in the background, downloading common things to speed up your downloads. (Something like that.).

    Well, after I was finished using BitTorrent for a week, I uninstalled it. A month later I was puzzled over traffic I (by accident) saw in my Wireshark trace. Places like Portugal and Brazil.

    It took awhile before I realized that uninstalling BitTorrent didn't do the same thing as installing it. I had to specifically uninstall btDNA.

    So, that's a practical example of the downside of uPnP. it wasn't uPnP's fault that BitTorrent wasn't very straightforward about what it was giving me. But, *a lot* of stuff isn't straightforward (i.e., malware. And, if you google for btDNA, you find a lot of people discussing it in terms of being malware.). If I hadn't done a Wireshark trace for unrelated reasons, no telling how long that thing would have been running without my knowledge. Ports open. Etc.

    Stupid me for installing BitTorrent. But, from now on, uPnP will always be disabled unless I specifically need it to be enabled. That's the easy solution.

  9. Toastman

    Toastman Super Moderator Staff Member Member

    az2008, that's a coincidence, as this afternoon I had just downloaded and installed Bit Torrent to see what DNA actually does. As far as I can see at the moment, it isn't doing anything useful at all.

    DNA is malware, as far as I am concerned. So it's a bin job ...

    Anyway, there was a clear warning when I downloaded it that one "had" to also install DNA, and also that DNA would "have" to be uninstalled separately. It's a clear policy of trying to leave a trail of machines running DNA to assist in this global "helper" scheme - helping somebody make a huge profit by using our bandwidth for their own purposes, that is - regardless of whether people want to or not. However, at all times, I could clearly see that DNA was active because it listed in the UPnP connections list. Of course that's only because I have access to the router, most people would never know why their computer was so slow, especially on low bandwidth ADSL lines etc.

    A standalone home user may not need UPnP because he can open his ports manually on the router. On bigger networks like this one, where we have no idea what people are using and it is impossible to police, some method of opening ports automatically is essential. Without UPnP/NAT-PMP, most people just can't use their applications. If they are really miffed about that they will move out and a large part of our income is lost.

    Having said that, it's about time that Microsoft etc. fixed the problem and replaced UPnP with something a bit more secure. No that that would help in the case of Bit Torrent/DNA bandwidth theft, of course.
  10. az2008

    az2008 Addicted to LI Member

    That sounds about right. It doesn't surprise me that I overlooked requirements like that. I just wanted to get Ubuntu, and figured BT is widely used by people, so there wouldn't be any "subtleties" to concern myself with.

    It didn't sound like I'm the only person who overlooked the subtlety. When I googled for btdna, I found a lot of people asking "is this a virus?"

    To me, this is a good example of what's wrong with uPnP enabled as a default. uPnP may serve a valuable purpose when someone wants it to. But, even reputable software can employ some subtelties to get something on computers, taking advantage of uPnP without the user knowing it (or approving of it).

    What's funny about this is, after I found all the activity in Wireshark, I poked around in Tomato and found the open ports. I deleted them. Not even knowing uPnP was the problem. A week later, I saw the traffic again. Deleted the open ports. Disabled uPnP. Then when I upgraded to 1.23, I saw the ports open again(!). That's when I started looking around, and realized I'd never seen "btDNA" in my task manager process list.

    Even though it was my own ignorance, I got a bad taste about uPnP (especially being enabled by default).

    I agree that it could serve a valuable purpose, and that it's too bad there isn't a better way.

  11. spicoli

    spicoli LI Guru Member

    What I meant in an earlier post is that, for example, you start a certain app configured to get 100%-100%... like a game. Will it always get the amount of bandwidth desired to run?
  12. Toastman

    Toastman Super Moderator Staff Member Member

    Spicoli - online games are usually low bandwidth control streams, and require a fast response, low latency. 1% initial allocation may be enough, but to be certain I'd put something of the order of 2-3% rate and 5% limit. The reason for not allowing the "limit" higher than this is to limit the damage if and when a torrent application uses the port and attempts to take that bandwidth for itself.
  13. spicoli

    spicoli LI Guru Member

    Maybe gaming wasn't the best example to use, hehehe. Anyways, I suppose what I think goes on to QoS according to your post is correct, so, I'll just fiddle around.
  14. baldrickturnip

    baldrickturnip LI Guru Member

    Hi Toastman - thanks for your effort

    could you post a screenshot of your QoS basic settings and classifications - as your settings might make a good starting point.

    do you have multiple tomato routers in your network , and do you only have QoS switched on for the router that is connected to the modem ?

  15. Toastman

    Toastman Super Moderator Staff Member Member

    Hi BaldrickT

    I admin several buildings. I'll give an example of the current biggest, to encourage people not to think of this little router as being quite as limited as one would think.

    The building is of reinforced concrete, 120m long and 25m broad, 12 floors, about 350 residential rooms, 9 meeting/recreational rooms, 12 shop units, several offices, some of which occasionally use the wireless internet. There are 2 x 16Mbps ADSL lines to ASUS RT-N16 routers running modded Toastman Tomato. We label the classes differently to make it easier to understand. These enable the staff to see what the classes are doing much more easily than with the rather silly "Highest" to "Lowest" and then a further 5 classes, which are actually lower than "lowest"!!!)

    The routers are CAT5e wired via 16 way 100Mbps switches to a total of 34 WRT54GL AP's. These are located on alternate floors and are interleaved so that no room is more than 16m from an AP. The AP's are spaced about 25m apart on those floors. Coverage from floors without routers is through the concrete floor or ceiling - penetration has been adequate up to now, but many newer laptops and netbooks, and especially the now popular (and deaf) wifi enabled phones, have very poor wireless and it is possible that the number of AP's will have to be increased in future. If such a laptop client has difficulty connecting I recommend they use the TP-Link TL-WN422G USB adapter. This uses a higher power than most and also has a 4dB gain antenna on it. There are some Apple laptop models with metal cases which are so bad that they need to be used almost right underneath an AP! My experience is that Apple devices generally have really awful wifi.

    I interleave channels in groups, several AP's close together will be on the same channel. This hopefully prevents AP's from transmitting over the top of clients which they cannot hear. The same channel is re-used by another group as far away as possible from the first as necessary. We try to minimise interference between the AP's - but there will inevitably be interference with so many AP's on so few channels. But we can't do anything about that. The whole wifi thing is a mess, unregulated, and with not enough channels. Having said that, it works surprisingly well in practice.

    Here are some good explanations of how to ensure roaming works between AP's:


    WEP is used on all AP's - which use the same SSID. I've found WEP to be relatively trouble-free. While offering some encryption so that data is not transmitted "in the clear", there is little slowdown due to the CPU overhead caused by the encryption. More secure methods of encryption caused between 15% and 30% loss in speed, depending on the client wireless and the usage of the wifi at the time. Encryption as a means of access control is completely pointless regardless of the method used, because all residents freely pass the encryption code to all their friends both within the building and outside it. WPA2 Personal + AES would be the better choice for increased security.

    Loss of an AP causes user's PC's to switch to another - the process is automatic and transparent to the user. It is possible to walk around anywhere onsite watching streaming video on a laptop with no breaks while switching AP's.

    Sometimes the client wireless software will for no apparent reason jump from a good channel to a weak one and throughput will drop. Some manufacturers of wireless adapters are worse than others. Windows "Wireless Zero Configuration Service" also has been accused of causing this. So we've tried a few methods to try to ensure users can always get a gtood signal. Different SSID's have been tried for each router, but the experiment failed for many reasons, not the least of which was that users were just not capable of selecting the best AP for their apartment. And if an AP failed it would require them to have all of the alternatives set up on their machines too. Roaming is not quite seamless and sometimes not fast enough to prevent a stutter in video. This idea is not practical, although it would seem to be a good idea to allow users to select their own AP. This may still be done through the connection software of some wireless adapters, but the choice is rather limited, especially if you have a 64 bit version of Windows.

    The network is arranged so that half of the building is served by each switch, each of which is connected to one of the two routers. The two networks thus created can then be joined by a patch cable between the two routers. This gives me a potential choice between a single network, or two separate ones. I use manual load balancing via DHCP, as described in a later article, to share the two internet connections between users. Currently all my users are issued IP's via DHCP from a single router, which also allocates the default gateway by DHCP according to rules in the ADVANCED/DHCP-DNS text box.

    The AP's are all WRT54GL's running Tomato.

    220v AC Power is distributed to the AP's and routers throughout each building from a common, central UPS (a cheap computer device). Power Over Ethernet has been tried but the implementation and reliability of this was so poor we gave up on it. It was also too expensive.

    Simple is best.

    The 2 routers wired to the internet have wireless disabled, for best speed, and both run identical QOS. One router is the main gateway, which runs DHCP/DNS, and also allocates the second router by DHCP as an alternate gateway to half of the supported PC's, as a form of load balancing. Limits are placed on individual UDP and both individual and total TCP connections, normally keeping a router's total maximum connections in the region of 1000 to 2000. A much better UPnP client, MiniUPnPd, has now been included in Teddy Bear and Victek versions of Tomato and takes care of port forwarding for client applications. [Now also included in original Tomato versions].

    I am going to post screenshots of some setups. I hope this will assist people but, PLEASE - I do not need a deluge of mail asking me to tell every user why his QOS is not working. The information is here will help you to solve most problems - but the only way is to learn how to do it yourself. I will answer sensible requests but many will go unacknowledged.
  16. spicoli

    spicoli LI Guru Member

    *cough*how about sending a screenie in PM?*cough*

    I'm still trying to master this tomato QoS. It works fine now, don't get me wrong, but I'm a perfectionist when it comes to stuff like this. If it ain't broke, I tweak it. :(
  17. Toastman

    Toastman Super Moderator Staff Member Member

    QOS example screenshots

    A few caveats:

    The aim of the QOS here is to provide web access for the majority of normal users in such a manner that they are not aware they are sharing an internet connection. The setup as shown works well enough so that nobody here ever complains about anything, but it is not, and cannot ever be, perfect. Some games are covered, latency is not wonderful however. VOIP isn't particularly encouraged, IPTV is allowed and works quite well unless everyone comes on at once. It is possible to watch DVD quality IPTV streamed from Desync in the USA by Shoutcast. Average "good" web pages open completely in about 3 or 4 seconds. Queries to Google are usually answered in 0.5 to 1 second.

    I have deleted some entries that apply only to my own situation, what is left should be taken as an example which works in my particular circumstances, but will probably work well in other condominium blocks - just add your own specific details.

    FYI - I compile my own version of Tomato which labels the classes, because Highest to Lowest then A to E is very hard to remember when you are setting things up and for untrained staff to interpret. How the hell can you expect anyone to know that classes A to E are "lower" than "lowest" ??? It's stupid. [My opinion!]

    However, the examples are shown using standard tomato labels HIGHEST through to E.

    Firewall script examples - choose the ones you need:

    #Limit no. of TCP connections per user FORWARD=to WAN INPUT=from WAN
    iptables -I PREROUTING -p tcp --syn -m iprange --src-range -m connlimit --connlimit-above 80 -j DROP

    #Limit no. of all *other* connections per user (inc. UDP)
    iptables -I PREROUTING -m iprange --src-range -p ! tcp -m connlimit --connlimit-above 20 -j DROP

    #Limit no. of outgoing SMTP simultaneous connections
    iptables -I PREROUTING -p tcp --dport 25 -m connlimit --connlimit-above 10 -j DROP

    #Limit new UDP packets opened by users - UDP to Router
    iptables -I INPUT -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT

    #Limit new UDP packets opened by users - UDP out to WAN
    iptables -I PREROUTING -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT

    If you test the above limit scripts with a limit of say 5 connections in the line, you will often see that it doesn't appear to be working, you will have many more connections than your limit, maybe 30-100, that you can't explain. Some of these may be old connections that have not yet timed out, and waiting for a while will fix it. Be aware that often these may be TEREDO or other connections associated with IPv6 (windows Vista, and 7) which is enabled by default. You should disable it on your PC by command line:

    set state disabled

    Split traffic between multiple gateways with these scripts - enter in Advanced - DHCP custom configuration box:

    dhcp-mac=red,00:0D:87:2D:1C:7A #(one example MAC SHOWN)
    #This entry must be duplicated for each MAC address to use the second gateway


    #This entry sends anything in this address range to the second gateway


    #wildcard used on MAC header can be used to split if the above IP address range does not work

    dhcp-option=net:red, 3, #Assigns the tag "red" to the second gateway
    dhcp-option=net:red, 6, #Assigns "red's" DNS server to the second gateway

    The second gateway is merely an Access Point set up as normal but with an attached ADSL PPPoE modem, which provides the second connection to the internet. Switch off the DHCP/DNS on the second gateway. The script assigns gateway details along with the IP by DHCP). Once it is verified that your split is working, you can add access restrictions (if needed) and QOS to this router also. "red" is just a tag referring to a particular gateway. e.g. 4 gateways could be red, green, blue, yellow or whatever you wish.
  18. Toastman

    Toastman Super Moderator Staff Member Member


    Connecting two devices by cable remains the best method in terms of speed and reliability.
    The router connected to the internet is known as the "gateway".
    The secondary router is called just an "AP" (access point). Please don't refer to it any more as a router, as this just confuses people. Set it up as follows.

    "Basic-Network" page:
    • Set WAN/Internet to "disabled"
    • Router IP Address - something different to your gateway IP address - e.g.
    • Subnet Mask
    • Set "Gateway" to the IP of the gateway e.g.
    • Make a Static DNS entry for the IP of the gateway e.g.
    • Disable DHCP
    • Enable wireless
    • Set wireless mode to Access Point
    • Use the same security settings and SSID as the main gateway (best for roaming)
    • Set your preferred encryption method etc. as normal
    • There is no need to change advanced/routing setting, leave the router in "Gateway" mode.
    • Connect a cable between LAN port on the AP and a LAN port on the gateway router (LAN not WAN !)
    That's it. Don't do anything else, If it doesn't work recheck what you did. Be assured it works, (I have several hundred access points set up in exactly this way) it isn't complicated, and don't go changing anything else.
    *** If you want ultimate stability on your gateway machine, you may choose to turn off the wireless and just use the access points for wireless access. Without the additional complications of wireless, these routers are almost 100% stable and will usually stay up for years.

    I would also encourage anyone NOT to use WDS or any other form of wireless connected access point, as they are inherently very slow and unstable. Use CABLE to connect devices wherever possible -and save yourself a lot of tears! The philosophy I follow is "Keep It Simple". If you really want to bog down a router with Virtual Access Points, VLANS, WDS, Torrent Downloads, VPN, TOR, etc. your router will become as useful as a chocolate teapot. It isn't a PC - it's a little underpowered chipset that has been made as cheaply as possible to do the minimum required of it. If your web pages don't open in less than 1 second, then you should take a good look at everything again.

    KEEP YOUR SETUP AS SIMPLE AS YOU CAN and you will not regret it.
  19. baldrickturnip

    baldrickturnip LI Guru Member

    thanks for that Toastman

    I will try the settings and tweak from there - I have to deploy 4 AP's to share 1 internet connection with maybe 30 users.

    how do you control wifi access to the AP's ?
  20. Toastman

    Toastman Super Moderator Staff Member Member

    Access Control

    Now for a topic that's especially important in an apartment block. How do we stop unauthorized people from gaining access to the network? This is especially important when we want to make a charge for use of the internet.

    Firstly, the method used for Wireless encryption does not matter at all as far as access control is concerned. Some people may think I am mad for making that statement. But the reason is very simple. Human nature being what it is, within a week of changing the access code residents have given it to all of their mates. In the blocks I have upgraded, it was commonplace that there were more freeloaders than paid-up users. As a method of controlling access in residential buildings, wireless encryption is completely useless.

    The true function of the encryption methods is, of course, to stop people seeing your traffic.

    I have in the past used WEP - more as a deterrent and to hide plain text from view, than as a serious attempt at security. Why WEP? Because it's simple and reliable, whereas some of the other options seem to have various problems which have rather put me off using them, and there is also a speed issue involved. However, nowadays WPA2/AES would be the preferred option as it is supported by everything.

    So far I have never noticed that anyone outside the buildings has ever bothered to hack into the networks, and I have a very large number of sites active 24/7. There are, however. many people who are so totally paranoid about security they have crippled their setups so badly as to be unusable. To each their own: biggrin: KEEP IT SIMPLE!

    I use STATIC DHCP to issue fixed IP addresses for residents. So each resident has to give the reception desks their Wireless MAC - and some cash.

    I use the "Hostname" field to enter a unique code for the users - actually it's their room number - so that I can easily see who is doing what. Without this it would be very difficult indeed to set up QOS, deal with customer complaints, and to maintain the network. Tomato originally only supported a small number of users in Static DHCP so I changed things to allow 250. That should be more than enough.

    Average takeup of wifi is about 30% of the rooms, incidentally.

    The main router's DHCP is set to issue only one address - which is actually already assigned by Static DHCP to the system administrator's PC. Anyone requesting an IP, who is not in the "Static DHCP" list, can therefore only be issued with .100. BUT - as this address is already in use by the administrator - the freeloader gets the message "address not available" and is thus blocked. Hence, nobody can get into the system unless he is in the STATIC DHCP List.

    Next, I use an Access Restrictions rule to allow those in the list to use the internet. You should be able to figure it out from this:

    Access Restriction

    Rule description: "Allow access to this list" / All Day, Every Day / Normal Access Restriction / Applies to: All Except / Blocked Resources: Block all internet access/

    Tomato originally only supported 50 entries in Access Restriction. I've increased this to to 250 - but some NVRAM space issues do need to be addressed in some routers as this is the limiting factor. You may be able to enter hundreds if for example your QOS rules and other stuff is small. BTW - there is a trick to allow the use of more entries than are allowed by the GUI which may help if you want to do this with any older Tomato version. Use iptables -L to see the chain used by the rule - it will be something like "rdev01". Then you can add more addresses to the rule with e.g. :

    iptables -A rdev01 -m mac --mac-source 00:00:00:00:00:01 -j RETURN

    Add them to one of the script boxes if necessary. When there is no space left Tomato will become unstable and do strange things, so be very careful with this. If you have NVRAM space problems you can just use the restriction MAC addresses and forget about assigning static DHCP to clients. You do lose some of the convenience though. This way you may be able to add several hundred MAC address restrictions.

    The Wireless restrictions aren't of any use in my situation, because they apply to each individual AP - and not to the whole network including LAN users. However, up to 500 are allowed now if space permits, for people who want to run an open system.

    Unauthorized users could still gain access to the network by using a fixed IP and changing their MAC address with utility software, but so far nobody in any of my residences has ever done so, as far as I am aware. You can use Static ARP binding to assist. But if someone is determined enough, there is no way to stop them. If someone does this, I would block them using netcut until they got fed up.

    [Toastman mod also allows Static ARP binding, google for info on how this can help prevent someone assigning himself an IP address].

    At the beginning of each month, resident's payment records are checked and the router updated. Two entries have to be maintained, the Static DHCP Table, and the Access Restrictions table. MAC address from Static DHCP in one browser window can be easily cut and pasted into the Restrictions list in another window. The lists can be remotely maintained quite easily over the internet. [See note below].

    Upgrading Firmware via WWW

    I was forced to upgrade 24 AP's and 2 routers over the web, some 200kM from here. I made the discovery that actually flashing over the web was quite reliable. The secret is to WAIT and not panic if the router does not accept a flash in what you might think is a reasonable amount of time. It is quite normal for a remote flash to take up to 10-15 minutes and sometimes longer. If no flash after 15 minutes, I would wait an hour or two before I gave up because once the connection is closed, then you have a big problem!

    TIP: You can check the remote router's GUI in another browser window to see if it still responds - if it does, then the router is obviously not accepting the flash and it is safe to disconnect. If it doesn't, either a) it's accepting the flash b) it has rebooted and DDNS not yet updated, or c) it's dead Jim ....

    I have now flashed remote sites thousands of times with no failures whatsoever.
  21. spicoli

    spicoli LI Guru Member

    Couldn't have said it better myself.

    A note, when using BT, I noticed the flood of SYN Sent packets when counting the connections, so I reduced the time to 3 seconds after a good Google search and I must say its helped speed wise in all aspects (BT speed & router speed).

    Also, do you have the Prioritize Ack box ticked?

    Thanks for the thread. I'm only a few minutes into totally reworking everything and already I noticed good speed improvements using more classes (I usually stuck to Highest-A, and no lower) and bunching similar (i.e. p2p) classes together.
  22. Toastman

    Toastman Super Moderator Staff Member Member

    I no longer prioritise ACKS. The theory was that if you don't - you slow everything up. Kind of throwing the baby away with the bathwater.

    However, checking the box prioritises ALL acks. It does so by moving them into the "Highest" class. The hidden problem with this is that it also prioritizes ACKS for the P2P class, if used. Of course, the P2P outgoing traffic responsible for controlling the incoming TCP data consists mostly of ACKS. So if your rules do not otherwise restrict P2P, it will negate some features of the QOS system and P2P will gain an advantage over the rest. The P2P data stream is still correctly classified as P2P and can be restricted in other ways, such as a limit on incoming bandwidth to slow it down by TCP/IP backoff mechanisms, but on reflection it is probably safer to leave this box unchecked unless there is a special reason. Even then, if any P2P is used on your system, do not check it.

    DHT and BT DNA

    With P2P, e.g. BT & uTorrent, I never notice any improvement in downloads, nor any extra sources, by using DHT. It seems to generate a lot of completely useless traffic and is best turned off - if you can get your users to do so. [I just spent a week examining my uTorrent downloads, and never saw a single download using DHT]. Kademlia, on the other hand, is a pretty good system, but eMule and the ed2k system is not used much here these days, and is a bit redundant.

    Bit Torrent DNA is a real pest - it can hog your bandwidth while resulting in no benefits for you at all. You should always uninstall this. It is a sneaky attempt to use YOUR bandwidth for commercial gain. If you google for DNA you will see what I mean. Be warned, nowadays many companies are stealing your bandwidth to use it to spam others or to upload crap. Even Microsoft do it to upload Windows 10 to other PC's without your knowledge.

    Since I can't tell hundreds of users what they should or should not turn off, I restrict the numbers of their connections for them, particularly UDP, using the scripts in the previous post :biggrin: The average number of connections at any one time is now around 1000, but on occasions it does reach 2000 or so. There are usually between 5 and 11 P2P users online at any given time in the blocks I administer, and about 3 or 4 of these are what I would call "power users".

    Whatever you do, there will be occasions when it seems that tsome routers will reboot for some as yet unknown reason - because of this I switched to using ASUS WL500gP v2, which has 32MB, memory as the routers. My hope was that they would be more stable, but they are a little sluggish due to the "budget" chipset. The probable cause of the rebooting with 16MB routers is connection storms from a client, usually from P2P, and more often from a virus on an infected machine. A better choice than the WL500gP v2 would be the WRT54G-TM.

    Nowadays, there are many faster routers such as the ASUS RT-N16, E3000, RT-N66 etc.

    A later post will discuss strategies for dealing with UDP "connection storms" etc.

    @spicoli, I think 3 seconds is a bit sharp for SYN timeout - there must be many occasions when a remote server cannot respond within that time period. I'd make it a bit longer. Normal Windows behavior as I recall from my MS training sessions several years ago is for initial SYN timeout of 3 seconds, if no ACK/SYN is received, then the connection will back off, and try again - up to a limit of (approximately) 20-25 seconds. So no less than 10 seconds would be my choice. While on the subject, if you use VOIP be careful with the UDP assured timeout setting - up to 300 seconds may be necessary in many instances.
  23. spicoli

    spicoli LI Guru Member

    Alright I've pretty much nailed my classification settings. My list is definitely alot smaller now, but its speedy and it works.

    What about TCP Vegas? Should I use it simultaneously with QOS with/without limits? Or should I just knock off QoS and use Vegas? Vice versa? I haven't had any luck with Vegas tests, as they seem to be coming up the same across the board. Reading the posts in threads things seem to be spotty with some people. I figured I got a pretty active network as do you, so any results you might have would help myself.
  24. callous

    callous Network Guru Member

    QOS and vegas arent replacements for each other.

    Enable both and see if there are problems. Ive found they work well together. Enabling QOS actually gives me a very smooth flat maximum and sustained upload graph (look at Tomato's bandwidth graph for realtime).

    That for me is what it should be about - a sustainable maximum upload speed for all applications.
  25. callous

    callous Network Guru Member

    Could u tell me what's wrong with using Vegas in your experience, toastman?
  26. spicoli

    spicoli LI Guru Member

    Isn't that the purpose, to try to clear/avoid the congestion? I don't know too much here, sorry for the bombardment.

    As for the tests, I don't see any difference between using and not using TCP Vegas. Maybe my network isn't bogged enough to see a clear difference (or have something such as IPTV to have an easier look), but I put torrents at full speed and didn't notice much difference between having it on or off.
  27. Toastman

    Toastman Super Moderator Staff Member Member


    It doesn't do anything, basically. It does not work at all on connections passing "through" the router. It *may* do something on connections from the router itself, browser, ftp, etc... but even then, does it depend on BOTH ends also running similar protocols?

    My tests show no improvement, of course. This subject has been flogged to death on the forums .. let it rest...

    ANY such transmission control or congestion avoidance algorithm running ON THE ROUTER will have absolutely no effect on connections that the router is merely passing through.

    **** SNAKE OIL ****

  28. Toastman

    Toastman Super Moderator Staff Member Member

    Accessing bridged modem via the router:

    Give your modem an IP in a different subnet to your router. Normally it's easy to use which is probably the one in most common useage, as the router's default is

    Enter the following scripts into these sections of ADMIN/SCRIPTS page:

    init: or wan up: ip addr add dev $(nvram get wan_ifname) brd +
    (you may need a "sleep 5" line first to give a delay)

    firewall: iptables -I POSTROUTING -t nat -o $(nvram get wan_ifname) -d -j MASQUERADE

    The first allocates an IP ( and a different subnet to the appropriate vlan interface (WAN port) for your router. Normally this port doesn't have an IP so we have to give it one, so that we can use it to add a route. There's nothing special about the .13 - choose what you want. Since we wish to acess your modem, the subnet is chosen to be the same as that of the modem.

    The second sets a route for that subnet via that vlan interface (WAN port) to the modem.

    The scripts will discover the correct vlan for your router from NVRAM.

    You can check your routing list in Advanced-Routing menu to see if the route is there.

    Now you should be able to access your modem just by typing its IP into the browser. Please note that not all modems seem to respond to this, though, even if they work just fine when connected directly to a PC.

    Exactly where you put these scripts is up to you. The first usually is put in init, but it may cause you to lose access to the modem when a new IP is issued on lease renewal. Placing it in the WAN UP box should cure that. Or, you can stick them all in the firewall. It's up to you!


    For those reading through this thread - as of late 2011 Toastman tomato has modem routing built into the GUI so you don't need to use these scripts. You MUST however give the modem an IP that is different to the ones used by your router and LAN clients.
  29. Toastman

    Toastman Super Moderator Staff Member Member


    So why do we use "ping" to measure effectiveness of QOS? Well, it's because it is one easy way to find out if the link is saturated. When the link is saturated, the reply to the ping may be stuck at the back of the queue at the ISP waiting to be sent to your router, so we get a long ping time or a dropped ping. The more congested the link, the bigger the delay could be. If QOS is working correctly then the ping response will be faster because we should have eliminated the queue at the ISP altogether. The fastest time will be with no queue or a small queue - the best times will be when the queue is being cleared faster than it fills up. A lousy ping time and dropped pings occurs when the queue fills faster than it can be sent to the router, and a queue then exists. Once it exists - we have no way of knowing how BIG the queue is.

    The best way to improve ping times, or latency, is therefore to leave plenty of free bandwidth on the downlink by use of QOS outgoing and incoming limits. Mostly, that means limiting P2P to no more than 70% of incoming bandwidth. Usually that happens when outgoing P2P class is set to about 10%-20% - and I also set an incoming limit on P2P to 70% for additional insurance (you should regard this as essential). If you wish to allow more P2P and suffer a little worse latency, then outgoing limit up to 80% or so can be set. But my setup probably isn't typical - it's more of a worst-case situation, because of the large number of active users and especially P2P-ers. Without any QOS the link would be saturated most of the time and especially in the evenings.

    So, in general, if my settings will work for me, they should work for just about anyone, and would make certainly a good starting point. Once working and studied, start disabling things that don't apply, change some limits to more suit your own environment, etc. Be careful to understand what a rule did before you get rid of it. Don't change the order without thinking if the order of the rule is critical to it's operation.

    I agree wholeheartedly that the system we use on routers is not "QOS" in the usual *engineering* sense, but the term has come to be in common use, and we're stuck with it. Nevertheless, the function of the "Router QOS" is indeed to provide some "Quality of Service". Some posters in the forum measure the success of their QOS by pointing out that their upload bandwidth is full all the time. I, on the other hand, sometimes consider that a complete failure, because it will always result in congestion. If filling the available bandwidth was the goal, just turning off QOS and letting all hell break loose is sufficient :eek:

    From January 2012 - A much better QOS ingress system using IMQ has been added to "Toastman" Tomato. The overall limit works as it should and the classes now have priorities. This should allow better utilization of our precious bandwidth. At last the QOS is complete!
  30. bigaperture

    bigaperture Addicted to LI Member

    Toastman, thanks for putting this together. I've been struggling with how QoS works in our little routers and how I should set it up. Your thread has been a big help. I've already read it through it twice.

    How would I figure-out my real low speed with Comcast speed boost affecting any speedtest?

    I have a PC doing a VPN tunnel back to work. Does the traffic in the VPN get affected or is the VPN seen on the whole as one kind of traffic?
  31. az2008

    az2008 Addicted to LI Member

    Speed boost affects inbound only, right? In that case, you'd have to specify the inbound QoS bandwidth to be the conservative, lower value you normally obtain over a sustained period (not the immediate "boost").

    This is where it may depend on what your goals are. I.e., if you're trying to preserve sensitive traffic like VOIP, you'd want to be conservative and sacrifice the optimal throughput that may be available at times (the "boost"). If you're trying to get the most speed, you might be less conservative and accept some congestioon.

    I use a VPN which is over SSL. It's seen as 443 (secure http) traffic. QoS doesn't apply very well to that (differentiating the different protocols that are "tunneled" over that SSL web connection).

  32. spicoli

    spicoli LI Guru Member

    Powerboost goes both ways. You can either do large file dl/ul tests or have a look at your modem's config file to determine your speed tier. Non DOCSIS3 tiers are something like 6/1, 8/2, 16/2. I've heard of 6/2 and there are other lower tiers that go unadvertised but thats the gist of it.

    Anywhoo, thanks for the Vegas explanations. It also occured to me that I can't use various applications to test Vegas, cause, well, I assume its TCP only. I'll leave it off... for now..
  33. Toastman

    Toastman Super Moderator Staff Member Member

    bigaperture, you're getting better advice than I can give from the other guys here. If you can't come to any other arrangement, of course, at the end of the day, you have to use the lowest figure for outbound limit, more's the pity. As far as the incoming limit goes, if you put in the low figure as the overall limit, and put NONE as a limit for a class, (such as WWW) that class will ignore the limit and reach the higher speed of the boost - which will at least give Web browsing a nice turn of speed!. By juggling with this you may find a balance.

    In more recent versions with better QOS, the "No Limit" choice has been removed.
  34. spicoli

    spicoli LI Guru Member

    err, just noticed the majority of my torrents are on UDP, and I doubt TCP Vegas applies to UDP, so off it shall go. All that searching and whatnot and I didn't bother to check for the protocol on my torrent users.

    Anyways, I've gotten my QoS working pretty well. Thank you Toastman and others for your input and suggestions.
  35. az2008

    az2008 Addicted to LI Member

    Is that true for all cable ISPs? I don't have Power Boost. But, my cable ISP says that (for the tiers that have it) it improves download speeds. I've never known them to miss an opportunity to overstate things. :) So, the lack of any mention for upload speeds made me think it only applies to downloads.

  36. az2008

    az2008 Addicted to LI Member

    That could be another factor explaining why there are so many mixed reports about the benefit of TCP Vegas. If someone's trying to improve the quality of UDP traffic, they might be disappointed as TCP Vegas improves the quality of TCP traffic *at the expense of UDP traffic* (avoiding TCP congestion, only leaving more bandwidth for UDP to consume.).

    If someone's trying to improve TCP performance (movie downloads) they might notice an improvement. If they throttle some UDP applications (like torrent) down to trickle rates, they might find that TCP Vegas improves other UDP applications (like VOIP) be the fact that TCP applications are prevented from congesting the pipe.

  37. spicoli

    spicoli LI Guru Member

    For Comcast Powerboost it goes both ways.

    EDIT: Noticed http uploads was bogging down my connection. ARGH! Back to the drawing board.
  38. bigaperture

    bigaperture Addicted to LI Member

    Thanks for the help. I was having some problems with my old DSL line but now that I've been on Comcast Blast I haven't. It's only been 5 days though. My high is 27778/8451 kb/s and my low is 14836/3704 kb/s. I work from home and during the day I have full bandwidth to myself and even through the VPN I can get the 14mb/s.

    But at night I could have 5 systems plus VoIP going on. Wost case would be:
    xbox360 or PS3 online
    PC doing online game and Skype
    my Wife's Mac on-line
    PC streaming a Netflix movie
    me downloading something big on VPN or not
    and a VoIP call.

    For now I'll leave QoS off.

    If I run into problems I was thinking of this priority?

    VoIP = highest - 2 to 100 ( Since it's low bandwidth should this be more like 2 to 10?)
    www and ack - 20 to 100
    streaming video 40 to 60
    VPN - 25 to 100
    on-line games 13 to 100

    Excuse my ignorance, I went from using Linksys firmware on my 54gl with default settings a week ago to high speed cable running Tomato and trying to figure-out QoS.

    Thanks again guys.
  39. az2008

    az2008 Addicted to LI Member

    I checked with my cable provider and they said it affects both directions too.

  40. Toastman

    Toastman Super Moderator Staff Member Member


    I have to be quick because very busy today, but a few comments to help.

    QOS needs to be on before any of the limits work.

    Streaming video is often on HTTP port 80, if it's on other ports you need some QOS entry for priority. You also may need to prioritise the control channel somewhere - look at the examples I gave which attempt to cover those I know about and some I don't. By and large, IPTV works quite well with those settings. Shoutcast needs ports covering - again, look at the examples. I chose mine by looking at the ports used in Winamp.

    With online games, mostly the traffic consists of small packets, no need usually to give them much bandwidth. It wouldn't seem to matter if you did, but if some P2P gets into that class by using the same ports, then it can take over - so it's best to set the limit small unless you need to do otherwise.

    Look at the game examples - these cover some MS games and a few local ones - it's impossible to cover all of them - there are so many. A serious gamer would make mods here I'm sure. Again, there will be help forthcoming from others if you get stuck.

    You'll get the hang of it quickly, just keep experimenting!

    I am very envious of all of you high-bandwidth hogs. Where I am we are living in the past, anything remotely approaching high speed costs an arm and a leg and comes with so many conditions you have to be mad to sign the papers....

    VOIP users need to prioritize UDP for VOIP without also doing the same for P2P...

    Spicoli - are you sure that most of your P2P is using UDP? I have never found UDP making much difference to torrents - most of it caused by DHT which generates a lot of UDP traffic but results in no downloads at all. It is true that if you enable DHT, and later, uTP, the majority of the peers are UDP. But the actual contribution of these peers to your downloads is very small compared with TCP!!!

    EDIT: This has all got even worse with the issue of uTorrent v2.0. This version uses a new protocol which is carried over UDP. So from now on, expect even more P2P UDP traffic than TCP. Since uTP is now hammering ISP's equipment with huge quantities of small packets, all over the world, many of them are responding to what they consider to be a malicious attack by banning uTP. Not good. Anyway, as far as I can see, no change in QOS setup is necessary. See posts below.

  41. Toastman

    Toastman Super Moderator Staff Member Member

    Confusion about the term "QOS"

    Unmanaged switches, and many routers, just forward traffic without looking at it and without doing anything special to it. Some switches and routers have several priority queues for network traffic (e.g. Tomato has 10 - which are Highest/High/Medium/Low/Lowest/A/B/C/D/E). These provide a basic kind of QoS by giving priority treatment to certain types of network traffic.

    However, anyone searching the web for "QOS" will find that in engineering circles, QOS means something quite different to our simple little router's so-called "QOS".

    Okay, so all these QOS systems do use the idea of priority queues to give some types of traffic an advantage over another. How do they classify which network packets go to which queue?

    The methods used in small SOHO routers can be quite effective but rather limited.

    Simple Router "QoS" can classify by, for example, the UDP port number. ie. SIP traffic for VoIP is often on port 5060. So we usually have a QoS rule that reads the destination port in each packet and sends all packets destined for port 5060 to a high priority queue. Traffic in this queue will be sent before that in a lower priority queue.

    Another classification method could use the physical LAN port on the switch. ie. all traffic going down the wire plugged into socket X gets priority.

    So, what methods are being referred to in the engineering papers and articles we often find on the internet?

    There are methods which tag each packet with a code that can be read by hardware along the traffic route, to tell that hardware how to send the traffic (assuming the hardware is configured to obey the codes). The idea being that all routers across the internet would recognize these tags and give priority to the marked traffic as needed.

    Two common methods, ToS and the newer DSCP, refer to packet marking schemes that use the packet header.

    ToS uses only 3 bits of the packet header and with 3 binary bits this gives you 7 different possible priority codes other than 0. Namely: 001, 010, 011, 100, 101, 110, 111. DSCP uses 6 bits of the header giving 63 possible codes. These two marking schemes overlap. Packets marked using DSCP can also be understood by hardware that only reads the 3 ToS bits. Using these methods to mark packets traversing the internet should, in theory, give priority to latency-sensitive applications such as VOIP (as long as every router on the internet actually reads and acts upon the tags).

    You can, for example, purchase little adapters which mark packets they send, such as the popular Linksys PAP2. These plug between an analog phone and an ethernet jack, allowing use of the phone for VOIP. Traffic marked by these adapters will therefore give priority to your VOIP traffic as it traverses the internet.

    VoIP calls via SIP in fact consist of SIP traffic that set initially up the call, and RTP traffic that actually carries the voice. Some devices can mark these two types of packets differently - so you could prioritise them differently if you had the hardware to do so.

    But now for the big problem. You see, routers on the internet mostly ignore any of these marks, so it doesn't actually work. Even if it did, everyone and his dog would immediately just classify all of their own traffic to give priority and since EVERYONE has "priority", NONE actually have it. It is quite well known that even Microsoft did this! Microsoft Windows 2000 always tags its traffic with IP precedence 5 [citation needed], which fact alone would have made the global traffic classing completely useless. So let's stop even thinking about it - OK?

    Next for all the pundits who keep muttering on about how router QOS doesn't accomplish anything for incoming traffic except drop packets when the link is saturated. Yes, of course it does you idiots! Thats exactly what it's supposed to do, that is how it works (for TCP anyway). That forces the distant server to back off and retransmit thus slowing down the link. That is exactly what happens anyway with TOS and DSCP marked traffic! What difference do they think marking can make ? If the downlink to your router is saturated - it can't be delivered and will be dropped, marking or not - causing the distant server to back off. It's exactly the same. So now please shut up about it and think.... you know that soft spongy stuff in your head? Use it.

    Our simple router QOS as used in the vast majority of SOHO routers does not mark traffic in this way. We have to classify each protocol/application by ports, L7 filters, etc. which can be a challenge. Packets are then marked, sure. But this marking is only used internally by the router.

    This thread tries to show how this may be achieved.

    I hope this clears up some of the confusion.
  42. davidehue

    davidehue Addicted to LI Member

    FTP QoS?

    Hi.. I'm using Tomato 1.23 with SUB mod (teddy_bear), and it have a built in FTP Server. The question is, How to set the QoS for FTP Server access from WAN so I can manage the upload B/W of my router for FTP?

  43. Toastman

    Toastman Super Moderator Staff Member Member

    Hi dh

    Assuming that QOS works on connections from the ftp server, go look at the screenshots above, you'll see entries for FTP on ports 20 and 21, which do the same thing in the apartments here. I believe the source of the FTP connection will be the main address of your LAN - for most of us - so a new rule could be made to prioritize anything being sourced by the router. Let us know if it works, I have a rule for 8080 to speed up remote access and that does work OK.
  44. davidehue

    davidehue Addicted to LI Member

    Thanks for the info, finally it works. I created some rules using L7-FTP, and set the limit on QoS-Basic Setting. I gave the upload limit of the FTP 50%-60% of my total upload B/W.
  45. kabar

    kabar Addicted to LI Member

  46. Toastman

    Toastman Super Moderator Staff Member Member

    Adding more bandwidth with extra gateways

    There comes a time in a residential system where more bandwidth would be desirable. In some countries, only low-bandwidth ADSL lines are available at any price. We have to use more than one line and share the available bandwidth somehow. Commercial DUAL-WAN routers are expensive and don't really do what is expected of them. They are unreliable. There is a chinese version of Tomato which looks very promising, but unfortunately the author has stated that he has no intention of releasing either an English version or the source code. It is also extremely unstable.The fact is, Multi-WAN routers are probably going to cause you more grief than you ever had before.

    What happens when your Multi-WAN router goes down and you lose ALL access to the routers? You can't use any kind of remote access to fix the problem. All of your customers are cut off.

    This articles shows another way to share the load between two or more gateways, and discusses some of the implications.

    Adding the new gateway(s).

    This is the easy bit - just set up another router on your wired LAN with an internet connection as normal - e.g. Treat it exactly as an AP, don't turn on DHCP/DNS. Just add the ADSL (or whatever) line. The AP has now become the new internet gateway.

    How to use it

    From your list of allowed users, set up rules to assign a group of them (whatever split you wish) to the new gateway, using the following method on your main router (often

    Add these lines to your Advanced - DHCP/DNS custom text box (three examples shown).
    OR a MAC wildcard:
    OR for an IP range example:

    dhcp-option=net:red, 3, #Assigns "red" group to the second gateway
    dhcp-option=net:red, 6, #Assigns "red" DNS server to the second gateway

    NB - "red" is just a tag. Change it to whatever you like. You may find it necessary on some builds to change the word "net" to "tag".

    Anyone in the list will now receive instructions from the DHCP server to use the second gateway. You could even use this to assign the second gateway to troublesome P2P users and let them all strangle each other :)

    Now for the implications of this:

    Anyone using the second gateway will have normal access to the web, all incoming and outgoing new connections will take place via that second gateway after a new lease has been obtained. However, what about the port forwarding necessary for some applications to work? We need to enable UPnP on that gateway also.

    When it is working, add QOS rules to the second gateway if necessary. I did not add any restrictions to the second gateway yet, but the MAC addresses should be added to any restriction rule to prevent use by unauthorised persons. Generally most people are completely unaware of what other gateways are used in a network, and so restrictions have usually not been duplicated on extra gateways.

    You will sometimes see ports opened by UPnP on both routers - don't worry too much about it. You may also occasionally see an odd packet apparently on the wrong router - reason unknown. Again, don't worry about it, it all seems to sort itself out.

    Addit info:

    1) uTorrent appears to renew UPnP port assignment every 20 minutes.
    2) eMule UPnP is broken, look for modified versions such as MorphXT
    3) Vuze (ex Azureus) will open ports on both gateways if required.


    [The downside to a Dual-WAN router is that when it goes down, it takes down both lines anyway. The method described here will allow users who already have a lease to use the surviving gateway to continue as normal. If you have web access by DDNS to both routers, you can use the surviving router to access the failed one and sometimes bring it up without a site visit.]

    ANOTHER METHOD - two gateways for redundancy

    You can manually enter IP and Gateway details in Windows TCP/IP properties.

    You can actually enter more than one gateway in Advanced Setup. The second gateway will be used if the first fails. However, it will not reset if the first gateway comes back on line. You need to release/renew the lease, or reboot the computer. However, it's hard to get a large number of users to implement it, so it is of little use for residential networks. But for you personally, remember it is a way to get automatic "failover".


    With the introduction of Tomato 1.25/6/7 there is a new feature allowing your router's DHCP to allocate the default gateway set in the BASIC/NETWORK/LAN section when the WAN connection is disabled. This may allow Tomato to act as DHCP server even when not connected as a gateway. You may relieve some of the load on your router by running DHCP on one of your Tomato AP's. I haven't tried it yet.


    A different method that you can use in a busy apartment block with many access points, set half of the access points to use the second gateway. That gateway must also be running DHCP set up excalty as the first. Now if someone loses access on his AP, he can switch to another and probably be reassigned to the second gateway. I haven't tried it, but it is a though ...
  47. fun.k

    fun.k Addicted to LI Member

    toastman: u da man!

    thanks a ton for taking the time to do this,

    u just demystified QoS to an artist that made friends with Tomato fw while looking for a router to handle ad-blocking (@ router level)...

    mraneri's post Auto DL Hosts File and Install...
    was my first nirvana and now yours allows me to understand and do decent QoS...


    u da man :)
  48. Toastman

    Toastman Super Moderator Staff Member Member

    Incoming maximum limit being exceeded

    In spite of everything you do with your QOS setup, on rare occasions you will be sitting happily contemplating the infinite when suddenly you will notice the incoming data in the 24 hour chart has been pegged at your maximum. Oops.... you say.

    When you look to see why your incoming maximum limit is exceeded, you can't find a reason. Usually there will be a mix of P2P and HTTP file download. Restricting either P2P or HTTP download class outgoing limit will reduce the incoming data rate. Reducing the incoming limit individually doesn't bring down the incoming rate as much as it should. Reducing the inbound maximum limit to 1000kbps is also ignored although the rate does slow down a little.

    If anyone has any ideas on what is happening it would be very useful. In normal operation this does not happen, I've seen it a couple of times and I am still puzzled. Since these downloads have always been from users and not under my control, I've no way to know what is going on - and I can't reproduce it with my own machines.


    "Porter" has since pointed out that the inbound limit set in tomato is only used for percentage calculations of the individual classes. It is NOT an overall limit. This totally explains the phenomenon! See this post and the discussion that followed:


    From January 2012 - A much better QOS ingress has been added to "Toastman" Tomato. The overall limit works as it should and the incoming classes now have priorities. This should allow better utilization of our precious bandwidth. As an added bonus, we now have an incoming pie chart !
  49. davipiero

    davipiero Addicted to LI Member

    Thanks for your explanation above.. It's still confusing for a newbie like me :(
    In fact, I just need my tomato do one of the following (for example):

    1. I don't care whatever the client do. Downloading, Streaming, Online game, etc. I need their traffic to be shaped everytime the exceed the file transfer rule. For example, a client can use the internet at a rate of 1 mbps but when it has transferred 25 MB of files, their internet rate will be shaped into 500 kbps. They can be back to a rate of 1 mbps after they are idle for some time, or after they go offline for some time.

    2. I want web surfing using web browser and another light traffic to get the highest rate and priority. While downloading, from http://, ftp://, P2P, and Torrent and another Bandwidth consuming activity to be placed at lower rate and priority.

    Both of those conditions above would be better if I can set per IP or MAC...
    Could you teach me to make the QoS and Classification that meets at least one of above conditions?

    Thank you very much.. :)
  50. Toastman

    Toastman Super Moderator Staff Member Member

    OK, the rules shown for my apartment blocks do most of what you require. You can also add IP/MAC addresses to the rules to limit them to certain people.

    I think your other post will elicit some response for limiting by functions, what you need is someone good at scripting to assist...


    Now - there's something I need to say. I am getting all kinds of mail telling me (and other users) that something doesn't work or that it can be done better "their" way.

    Now listen. This stuff WORKS. If you can't make it work, then you need to learn how to use it. There's no point in having a tutorial if people read it and then do the opposite because they think they know better.

    Here's a typical example:

    Whether or not they believe it works for them some or most of the time is immaterial. The fact is that unless the settings are set correctly then QOS will not work correctly. It may work some of the time if you're lucky. Posting this shit on the forum just confuses everybody and gets even more of them complaining because they follow the idiotic advice. I am bored with answering so many stupid questions when the posters have not followed the instructions even in the most basic form. In future I will simply delete such posts.

    You might gather from this just how many *really* stupid mails I get every day. Go figure why I am getting annoyed :)
  51. Toastman

    Toastman Super Moderator Staff Member Member

    Overclocking the WRT54GL

    One of the favourite subjects that keeps coming up in the forums is that of overclocking the WRT54GL (and now also the TM and the later routers such as the RT-N16). Specifically, what is the maximum safe speed, does it overheat your router, and what is the point anyway?

    Well, let's firstly define "overclock". It means to run the device in question beyond its designed speed.

    The data sheet for this BCM3302 processor seems to have disappeared from public view, but the short product description states quite categorically that the design frequency is 264MHz.

    So at 250MHz it is not actually being "overclocked" at all!

    So please don't believe that crap that you will read elsewhere in the forums. LOOK AT THE MANUFACTURER"S DATA SHEETS!!! They designed the thing, they made them, and they know best how it is supposed to work. It is a good idea to believe they know more about their product than some spotty kid on blahblah.com forum.

    If you have so much traffic that your router is struggling, then quite obviously having more speed would help. Such a situation is common these days when many people have more than 20Mbps internet connection.

    So why do *I* "overclock", because I have only quite feeble 5Mbps ADSL links?

    The answer is simple.

    Overclock by 25% and the speed of most things increases by 25%. That is actually quite noticeable. If you do get everything working properly, and as fast as you can possibly make it (this includes modem/MTU/router/ethernet cards/switches/PC/Sufficient Memory/Decent Web Browser/Half-open Connection Limits/Good Operating System etc..) then the reward is an almost instant response to Tomato's web GUI. Plus, faster response from web browsers, more throughput for your P2P, lower latency, better gaming, etc.

    Individually, a 25% increase doesn't seem like so much. Now add up the effects of 25% increases in all of the above - and you will be amazed how crap your internet was before you began. Does downloading a movie in 3 hours as opposed to 4 hours make sense? That's what you can get by overclocking, sometimes more. Pay attention to all of the other things I mentioned? You just might find that movie arrives in 2 hours. I really want to emphasize this - every customer who has ever seen my internet access working has said it's much faster than theirs. Yet I use the same router that they do! The usual reason is either they are using laptops or PC's which are poorly set up or have insufficient memory.

    25% extra speed is normally easy to achieve with the WRT54GL running at 250MHz. At that speed, most GL's are stable (I have only ever had one that wouldn't run at 250Mhz). And yes, it's very noticeable. The GL is way ahead on speed of the ASUS WL500gP v2's which I also use, even at 200MHz. Subjectively, the 500, which is supposed to be faster, "feels" SLUGGISH.

    The maximum "safe" speed is normally 250MHz, the fastest in the cfe table! You don't have to worry about overclocking because it's not possible to "overclock" this router without changing the cfe. There is a very small possibility that YOUR router may not clock to 250. And I mean small. That's because you may have a faulty "below-spec" processor. But I've only ever seen one. Actually I only say this to cover my ass. But it's up to you to decide whether to do it or not. Feedback from those who have changed the cfe to allow higher clocks indicates that the majority of CPU's will easily run at 263 MHz and usually at 275MHz.

    I have no idea where it started, but there are hundreds of posts in the forum warning people about this so-called "overclocking", saying it will trash your router and cause it to overheat. That is complete nonsense and I really wish people would not talk such crap. Just stick to a known working clock frequency, and do not try to flash an odd number "off the top of your head". Usually the nearest known frequency will be selected, but just in case ... don't do it.


    **NB** There is a known problem with selecting 215MHz which will brick your router (which is odd because usually selecting an invalid frequency will just select the nearest lower value from those listed).

    These are the valid clock frequencies for the WRT54GL - 183, 188, 197, 200, 206, 212, 216, 217, 225, 238, 240, 250. Note that although 264MHz is the design spec, it is not supported in this list.

    *** [For the ASUS RT, the list is very long. Most RT-N16's would not run stable at 532MHz, and Broadcom, the makers of the chip, now state the maximum operating frequency is 480MHz. I find mine all run OK at 500MHz. So a nice selection would be, say, 266, 300, 354, 400, 453, 480, 500.

    Additiona - 532 MHz, which was previously unstable, may now be an option - later versions of TomatoUSB MIPSr2 now seem to be fine at this frequency

    [Versions of Toastman firmware after September 2010 will have a safe frequency selection drop-down list, which hopefully will reduce the number of bricks caused by entering incorrect frequencies. Please set your router's clock to the speed you desire and save. If you do not do this, the router may occasionally revert to 133 MHz or similar lower default clock speed.]

    It is *possible* to go above 250 but then you really do start to tread on thin ice. Firstly, software modifications are usually necessary to enable any support for higher clock speeds. Then as you approach the true limit (said to be typically around 270-300MHz) the chip may well need extra cooling. There's a real danger of bricking the router irrecoverably. In fact 250MHz is the fastest overclock that can be achieved without modification of the table in the cfe. Anything higher than this, will show a setting corresponding to what you selected, but the reality is that it doesn't really change at all. Those interested in clocking higher than 250Mhz, take a look at this link:


    (At May 30 2011 - this link is to a different page and is no longer relevant).

    So, at 250MHz, does it overheat? Does it explode? Does it somehow end up with a shortened life? No, the evidence is that it does not. Let me expand on this. I live in a country where the temperature is always over 30 degrees and often 40+ degrees. Because I have many routers running in many remote locations it is very important to me that they are stable. I've used these routers under full load over more than three years in full sun in enclosed cases with no ventilation, fans, or additional heatsinks. They don't overheat and there's really no need to add a heat sink, though if it makes you feel better and you know what you are doing, there's no harm in it. Adding fans is really quite pointless. Adding a fan means having holes in the enclosure, access by insects and lizards, problems with dust and dirt, and the need for frequent maintenance on the fan itself. Kind like shooting yourself in the foot.

    However, there are a lot of zany people out there who want you to believe your 30 dollar router is some kind of advanced supercomputer and needs to be run by a nuclear reactor and cooled with liquid nitrogen :rofl: One guy even has heaters and coolers and all manner of temperature sensors on his externally mounted WRT54GL. Another idiot has an Arctic PC CPU cooler towering out of a huge hole cut in the case. If you really like to impress your friends I can recommend this fan:


    especially if you strap one to each ear.

    Well, it's a free world, and if you have nothing better to do but to make a prat out of yourself ...:biggrin:


    There have been many posts in which people have claimed that their router must be overheating because the router has "slowed down" and an instant cure has been to add a heatsink when all was magically restored to normal. Let me just point out right now that this is also absolute crap. Overheating microprocessors do NOT slow down gracefully when they overheat. They fail to work properly. That results in:

    system crashes
    random lockups
    memory errors
    application errors

    The exception to this is when the manufacturer of the particular microprocessor has deliberately implemented some scheme to protect the chip. This is for example done by Intel in their PC processors. They have onboard temperature monitoring and onboard circuitry so that when a threshold is reached (and also please note that this is at around 90 - 100 degrees C) - they reduce their clock speed.

    As the microprocessors used in current SOHO routers most definitely do NOT have this kind of protection, it is NOT POSSIBLE that a router has "slowed down" due to overheating. Be careful what you believe when reading forum posts. Most of these claims are just assumptions by non-technical people who read it on a forum .... and here we are back at the beginning again ...... Please don't accept everything you read on forums as gospel ! Look at the background of the poster, are they engineers? Do you go to your local plumber if you need a tooth extracted?


    Just wanted to put this in here -

    November 24 2009. Today is 1 year since I installed around 200 more routers/AP's in another four condo blocks. All running 250MHz at 150mW, all in enclosed boxes with PSU, no ventilation, no fans, no liquid nitrogen tanks or cooling towers, about 20 of them are in direct tropical sunlight, and NOT A SINGLE FAILURE. I haven't even had a single electrolytic capacitor failure. I now have about 300-350 routers and AP's (December 2009) - many of them have been up for over 2 years.

    October 10th 2010 More than one year has passed with not a single failure of the existing WRT54GL's, and a lot more now pressed into service, total now is around 450. Plus several ASUS WL-500gP and RT-N16's, which I now use for the main routers.

    Of course, immediately the RT-N16 came out there were those who insisted it overheats. More fan mods, holes in cases, etc. Yawn ....

    April 5 2011 and we have a new series of routers from Linksys, the E3000 is proving popular. And of course, people insist it's overheating.


    The two WRT54GL's that I have bricked myself due to experiments that went wrong have been successfully recovered by JTAG. That is a 100% success rate with a very large number of very cheap SOHO routers - so I take my hat off to Linksys!

    After 5 years some of the capacitors in the early WRT54GL's are beginning to fail. They are easily replaced.
  52. Toastman

    Toastman Super Moderator Staff Member Member

    "New Driver" transmit power level setting is not working?

    It's just been discovered that the transmit power setting does not work correctly when using the "ND" versions of Tomato. Read this thread for details:


    For a information on increasing transmitter power, overclocking, etc. and an engineer's explanation of why increasing your transmit power can often markedly increase performance (in spite of all of the absolute nonsense often seen in the forums to the contrary) follow this link:

  53. chadrew

    chadrew LI Guru Member

    Thank you Toastman, this thread is super helpful. I myself love Tomato QoS since it has helped a lot in balancing the load on two PCs here. I can browse the net and play games without any lag while other PC is downloading stuff or p2p'ing. The only problem like you mentioned is that if downloads max out my line, there's still lag despite QoS. The only solution I've found is to actually limit the download for certain classes.

    So, how exactly would I go about applying a separate rule for p2p uploads? Would I first add port 1024-65535, 0-10k for downloads and then add a lower rule for port 1024-65535, 10k+ for uploads? Limit the "downloads" one to 5%-20%, and the uploads one to 5%-100% so it would get good upload?
  54. Toastman

    Toastman Super Moderator Staff Member Member

    Greetings chadrew!

    Don't try to define anything for normal P2P. No rule is needed, anything not covered by normal rules will automatically end up in your "default" class. Set that to 1% - 10% to begin with. Limit it's incoming bandwidth to 60%, again to begin with.

    Hope this clears things up a little![/I][/COLOR]
  55. chadrew

    chadrew LI Guru Member

    Thank you for the tips. I'm curious about the part where you say to give P2P uploads 1% - 10% though. I can certainly understand the low "rate" (is that what you call it?) of 1%, but I don't understand limiting the maximum upload ("limit"?) to 10%. Since class E is the very lowest there is, even if I set the maximum outgoing bandwidth to 100%, it will only use the bandwidth that's available, isn't that correct? It just seems kind of a shame to use only 10% of my upload since normally programs like browser, etc. don't use much upload at all :)

    Also regarding p2p download class, I'm interested in what upload limit it should have to effectively limit download speed. You said 5% in the first post, does that mean download speed = ~20x of the upload, or so?

    Why I don't prioritize ACK: I read somewhere on this forum that P2P uses a lot of ACK packets, so prioritizing them would in fact give some priority to P2P.

    Thanks again for your help :thumbup:
  56. chadrew

    chadrew LI Guru Member

    Just do add, I've always thought QoS "rate" was the minimum guaranteed bandwidth (and the sum should amount to 100%), but you say it's just the initial bandwidth. If the sum is >100%, will it have some negative effects when Tomato will try to use more than 100% and start reducing the speeds dynamically?

    Maybe Tomato should have a separate "initial", "minimum" and "maximum" boxes? :biggrin:
  57. Toastman

    Toastman Super Moderator Staff Member Member

    QOS in the presence of P2P again...

    Hi chadrew !

    Some things are indeed not quite what they seem. The big problem is that there is not much genuine information about QOS operation, and most of what you read is actually just somebody's assumption. Even the wikis have misleading data. Because of this, it's really necessary to test things out yourself and try to see if the claims hold water. The reason I had my arm twisted to begin this thread was to help make people aware of the reasons why they were not being successful and offer some suggestions and explanations as to why. I searched this forum and others and found that there was so little factual information that thought it would be a great idea. I'm still learning too, and whenever I find something I have written is incorrect or badly explained, rather than leave it on the forum to spread even more misinformation, I go back periodically and update it.


    Proritising ACK's, which, incidentally, places them in the Highest class, is often done to improve speed of normal connections.

    If P2P is used on your network, however, a lot of the outgoing traffic consists of ACKS, and the rest is usually tracker traffic. Prioritizing ACKS for P2P can therefore effectively give a high priority to your P2P by moving most of its traffic that QOS is able to control out of its own class into Highest. That is often exactly the opposite of what we intended to do. It is better on reflection to uncheck this box. Unchecking this box places ACKS back in their appropriate classes.


    Your last comment on the P2P: This is the area which is least understood and the cause of most of the complaints about QOS not working, so it is very important for me to get this across correctly. I will *try* to explain more clearly but you need to re-read my earlier again posts in conjunction.

    The bottleneck that causes poor performance is generally the download link from the ISP to our router. If too much data is sent to us, it cannot be delivered to the router quickly enough, and a queue will build up at the ISP. Many packets will time out before being sent to us, and are lost. The remote server receives no acknowledgement for them, so it "backs off" and resends the data. Again some packets will be lost, so the server backs off again (exponentially), and sends the data yet again. This time, let's say the delay is sufficient to slow down the data stream, and the queue is reduced just enough so that our PC receives the packets, and sends acknowledgements. Now the server proceeds at the new data rate, and the bottleneck is cleared. That is a normal TCP mechanism and there is plenty of information if you google for it.

    The problem is, we are not just dealing with a simple case of ONE server sending data to one application. In my case, with residential complexes, I always have several hundred to several thousand open connections. And of course, the link NEVER stabilises, because new connections are opening and old ones closing all the time. Over a period of time, the "average" effect of our QOS system is what matters, but it is very difficult to step back and see the "big picture".

    You can see that something else is needed to prevent the buildup of incoming data at the ISP. Generally, we want to allow fast WWW browsing, so we give that priority and unlimited bandwidth. It is the P2P applications that are the worst, so let's concentrate on that.

    Let's go back for a moment to the analogy in the introduction:

    Suppose there are a thousand people out there who will send you letters or parcels in the mail if you give them your address and request it. Until you request it, they don't know you and will not send you anything. Send them your address and a request for 10 letters and 10 parcels and they will send you 10 letters and 10 parcels. Ask for that number to be reduced or increased, or ask for only letters and no parcels, and they will do so. If you get too much mail, you stop sending the requests or acknowledgements until it has slowed down to a manageable level. Unsolicited mail can be dealt with by ignoring it or delaying receipt and the sender will send less and give up after a while.

    The amount of mail you receive is usually directly proportional to the requests you send. If you send one request and get 10 letters, that is a 1:10 ratio. You've controlled the large amount of letters you receive with only the one letter which you sent. Sending 1,000 requests at a 1:10 ratio would result in 10,000 letters received - more than your postman can deliver. So based on your experience, you can figure out the ratio of letters you are likely to receive from a particular request, and then LIMIT the number of your requests so that your postman can carry the incoming mail. But if you don't limit what you ask for, then the situation quickly gets out of control.

    OK, so we need to understand how to prevent too much incoming P2P from saturating our link. Now the above analogy introduces the concept of an approximate "ratio" between what we send, and what we typically receive for a given application. We send out a *small* amount of data in the form of download requests and acknowledgements, but we actually get back between 10 and 50 times what we send - this ratio varies with protocol and application, amongst other things.

    With P2P applications we get back about 20 to 25 times what we send (my estimated figure, but this varies all the time so you need to find your own figure). So if we send out 100kbps of P2P data, (which are mostly ACKS) we will get well over 2Mbps of data back! That strangles our connection immediately and leaves almost no chance for other applications to work.

    1) Restricting outgoing P2P traffic

    So if we have a 1Mbps download bandwidth, we must send out LESS THAN 50kbps - or the returning P2P will begin to saturate the link. That is why I suggest a 1% rate and no more than 10-20% limit on the P2P class uplink. This will immediately place a check on the amount of P2P files (data) sent back to our client PC's. This is the point at which P2P will not usually saturate your link badly, but will do so from time to time. This will give best latency. You can increase this to allow more downloads, but the latency will suffer.

    2) Restricting Incoming P2P Traffic

    NOTE this is quite different to sending out 500k of data and then dealing with the resulting incoming 10Mbps+ of data by purely trying to place an incoming limit on it, which would not succeed immediately. There would be a delay, while packets are dropped, the server sends them again after a backoff delay, dropped packets again, another backoff before sending again, and so on - before the connection becomes stable at the new lower speed. At the same time, new connections will open and the process never ends. The latency will be worse.

    Both ways of adjusting the throughput have their place, but the incoming limit is best used in conjunction with an outgoing one, as a "backup" in case the "ratio" that we have guessed at is wrong for a given user/day/seeder or whatever.

    By way of example, by limiting outgoing P2P to 10%, and INCOMING LIMIT 66% I can reduce ping times to 21mS - 30mS most of the time. By increasing outgoing limit to 80% to allow more P2P the ping time goes out to 100mS plus. You must "juggle" both figures.

    I always set an incoming limit on P2P of around 50% - this will help to stabilise things by forcing the normal TCP backoff mechanisms to cut in IF the incoming P2P does rise too high. It also ensures that there is always SOME room for other applications to get a look in. (NOTE - although I want to emphasize the importance of limiting outgoing requests, you will almost certainly find this incoming limit on P2P bandwidth to be essential).

    10-20% outgoing seems like a *very* small figure, and many people are extremely reluctant to set this limit, or even 50%. You will see people insisting quite vehemently that they want to allow uploads to fill their bandwidth - which is why you will see so many complaints in the forums that QOS doesn't work. They have no BALANCE. Just imagine what would happen if we let our P2P applications send out a total of 512kbps of data? The returning P2P file downloads might be of the order of 50 times bigger, say 25Mbps, and the router cannot accept it, thus a huge queue forms. The reply to any ping we send out will often even time out before it gets back to us. In this eventuality, an incoming limit of say 80% does allow the whole link to stabilize by normal TCP backoff mechanisms, BUT it takes time - and new connections are being opened all the time, so this isn't the best approach. The latency is better by setting a lower limit on outbound traffic AND a lower incoming limit.

    [NB - If you are using uTorrent, the best way to get files fast is to prevent seeds altogether, or limit them to say 10kbps. Just try it and see. By doing this I am able to max out incoming P2P if I wish at 8Mbps.]

    So, let's examine what happens if we set our QOS to allow P2P at a rate that actually does allow it to reach 100% of outgoing bandwidth when no other application is using it. Let's suppose we then PING the gateway at our own ISP. The returning ping reply is held up in the queue of P2P data files waiting to be sent to our router. There is no priority on it, it has to wait it's turn in the queue. And there will ALWAYS be a queue. Our aim is to try to make sure that the queue is small enough that the returning ping is not held up longer than is necessary for our application to work. Now, if that application is a game requiring fast response, or VOIP audio, then we have to make a choice between it and the p2P. Reducing other traffic will improve your response time, using an "upload" or "crawl" class MAY help, but problems still occur. A compromise is necessary!

    And this answers the last part of your questions. Into the "P2P Upload" or "crawl" class we dump UDP, uTP, P2P uploads, and anything else we don't want to influence our normal data. By increasing the limit on this you can see other applications slowly lose their edge. You have to strike a balance which suits your own useage. It may be difficult to generate enough P2P traffic to actually see the effect of much of the above, so be careful with drawing conclusions quickly.

    To recap - if the P2P class is running amok, always REDUCE the outgoing P2P class data to something small, like 5%, and you will usually see an immediate reduction. Remove any incoming limit on that class just so you can see the effect of your changes. Then slowly increase to a safe figure. Keep experimenting until you are sure you see the relationship. Now reinstate an incoming limit. Use the client monitor feature in the latest Toastman compiles to see what is happening to your traffic.

    So, let me be quite brutal about this. YOU CANNOT HAVE YOUR CAKE AND EAT IT TOO, as my father used to say.

    In a large multi-user environment there has to be an emphasis on speed or customers will complain. We aren't so concerned about P2P or file downloads and their throughput figures.

    But, what setting would allow good P2P downloads with reasonable latency and good response for WWW browsing? Try P2P outgoing 5% rate 80% limit - incoming limit on that class 80% to 100%. Overall INBOUND LIMIT set to the 80% of measured speed. WWW class outgoing rate 40% limit 100% OR 100 - 100 and incoming limit NONE.

    As a home user, you may not need to go quite as far as I have, I need to make sure that everything works sufficiently well that most people do not even realize they are on a shared connection with around 100 other users.
  58. QSxx

    QSxx LI Guru Member

    Well, first of all Toastman THANK YOU for wonderful guide.

    Just implemented this on my small home network and i must stay it works better than IP/MAC BW limiter (i'm using Victek's 1.23 ND on WRT54GL)

    I would ask you something ofcourse if it isnt too much. I read thru all of this and well, while it's really well done, i didnt find any particular mention of small setups. Here, for example i have 6 clients on my WRT54GL.

    - Four wired and two wireless.
    - Two of them are heavy torrent users and would like to get most bandwidth when there's no demand for it
    - Three of them are heavy on youtube and dailyshow - video streaming sort of stuff
    - All of them use MSN
    - All of them want snappy web
    - One of them is power user that sometimes uses remote desktop, ssh sessions, ftp, ... in short not regular stuff and would like that to have priority over video streaming but not over web...

    Since i configured my QoS following your screenshots EXACTLY (page 2 of this thread), i wonder are there any "tweaks" or recommendations that might improve my particular situation (since you based your tutorial on your own experience which is "loads of users").

    I'm kind of guy that will eventually break something experimenting but well - to be honest it's little too much to understand just by reading it all - i'm basically asking someone (you? :) ) to show me the right direction - so i can understand from applied example.

    P.S. We are talking about 4mbit/256kbit ADSL line (currently Outbound max is set to 200 and inbound to full speed - 4000)

  59. Toastman

    Toastman Super Moderator Staff Member Member

    Sorry I have to reply very quickly as I have to go out shortly, but if you look at the setup, you should see a class for remote access. Medium?? That is the one I use to access remote sites, it is in about the right place? That has the ports for remote access already in, if I remember rightly. You can also put ftp ports 20 and 21, to this medium class instead of C. Get yourself a list of common TCP/UDP ports from the web, there are a lot so get a nice one, and see what ports you need to give priority. If you have any specific questions send me a PM. But hopefully most things should be working pretty well as it is?
  60. QSxx

    QSxx LI Guru Member

    Yes they are, actually working very fine so far. It does the job wonderfully. I'll be experimenting with giving priority to specific IPs later.

    Thx for quick answer.


    Why so many connections to 443 (it's tomato web interface but still) - and why is it unclassified inside LAN? Is there a way to hide LAN to LAN connections?

  61. phuque99

    phuque99 LI Guru Member

    lan to lan connection, especially to your router is not subjected to qos. if you don't wanna see them, they can be turned off in the Administration debugging options.
  62. Toastman

    Toastman Super Moderator Staff Member Member

    A Basic QOS Example

    I've had several people ask for a basic QOS setup which they can examine to see how to set up different applications. It's similar in principle to the one I use normally but much simplified. Most people can use this, and it will work quite well. Because it uses all of the available classes, it's very easy to stick your own rule into one of the classes or change one that's already there. The pie charts and graphs will quickly show you how your rule is working. Using ALL of the classes makes it easier to see what is happening. Change as necessary to experiment and suit your setup. Disable things you don't need - add more ports for applications that aren't covered. Read the full thread for notes.

    The best way to learn how QOS works is to change things and watch what happens.

    To get enough P2P to really show what is going on can be difficult - most people just aren't aggressive enough. Try this. Use uTorrent etc, switch off DHT. Switch off or limit uploads to as low as possible say 1 kbps. Set max number of half-open connections in Windows to around 500-1000. Set uTorrent to allow large numbers of connections (1000), and a high number of allowed torrents (500). UPnP will take care of ports for you. Go to a torrent site and search for recently posted HD movies, just load 20 to 50 from the top of the list with largest numbers of seeds/leechers. Now let it rip. Within 10 minutes or so you should be able to get up to 5Mbps+ of downloads. You can switch off QOS to begin with and make sure that you do have sufficient incoming files to test. I recommend HD movie files because they will give you several hours to experiment.

    No rule is needed for P2P as it will bypass all other rules and end up in the "default" class set at the top of the QOS/Basic page.

    The maximum speed settings here are typical settings for 4Mbps/512Kbps ADSL links here in Thailand. You'll see the upload set quite low to 250Kbps. This is intentional, because it is 85% of the lowest speed I EVER see on that link.

    The thumbnails seem to be readable so I won't upload anything bigger. I've just done this in a hurry, hopefully I didn't make any mistakes :dunce:

    This QOS allows P2P to take most of the bandwidth that isn't being used by other applications with higher priority. WWW response is generally quite fast (my test page at 4.5 seconds). Streaming video is possible even under full load from P2P and everyone else, but only rtsp is covered in this simple example. (Messenger text of course always seems fast anyway). HTTP and FTP downloads will take over from P2P and are also fast, while WWW browsing speeds and streaming TV should be unaffected. Games and maybe VOIP with these settings will probably be poor, as sufficient bandwidth needs to be immediately available at all times to ensure low latency.

    To limit P2P severely, set class D rate and limit to 1% and 10%, incoming P2P limit to 30%. Now you should see ping times come down, average to my ISP gateway is about 35mS. WWW speeds improve (my test page now 3 seconds). Plug in your own port numbers for your favorite game and test. It should be better. Now increase things one at a time and watch the results.

    If you are a regular VOIP or games user, you will find that the best ping figures are obtained by deliberately setting your "inbound limit" to about 66% of your maximum bandwidth. My best ping times from my ISP gateway are 21-25mS. The variation of the latency of a 1.5Mbps link with traffic load is shown in the graph below (original thanks to Jared Valentine). The big disadvantage with limiting your incoming speed to this level is underutilization of the link. You can have one, or the other, but not both.

    Short conntrack timeout values are great for control of P2P, but they can kill a VOIP connection. An assured UDP timeout value of e.g. up to 300 seconds may be necessary. Find the smallest number that will work reliably for your own VOIP system.

    Attached Files:

  63. fun.k

    fun.k Addicted to LI Member

    thanks for enhancing your (already perfect) thread t-man :)
  64. Toastman

    Toastman Super Moderator Staff Member Member

    The first screenshot below is of the above QOS working in normal state, several P2P clients, about 2000 P2P connections - plus all normal traffic. That system usually runs like this 24/7...

    The second shot shows Tomato with the classes labelled for ease of use.

    If this would be helpful to you when setting up QOS, you may download it here (please note that this server is not always running). The .bin and .trx files can both be flashed by Tomato's GUI..

    http://firmware.mooo.com/Toastman B...P Builds/tomato-ND_4515.5 based on 8515.5.trx

    Here a useful article:

    ADSL Bandwidth Management HOWTO - A good article on Linux HTB (Here you'll see that for low latency the author (Dan Singletary) also had to set his inbound limit to about half of the maximum speed....)


    Attached Files:

  65. Toastman

    Toastman Super Moderator Staff Member Member

    Remote Access to AP's

    I had some mails asking how to remotely access Tomato AP's.

    For those wishing to access their AP's remotely, you have to set up a DDNS server to gain access to your system. There's plenty of information on this on the web - use google for help. Enter the details of your chosen DDNS service into the tomato DDNS page. After that, you must port forward a suitable TCP port - say 8001 for AP no. 1 - to the AP's internal IP number, port 80. The AP must have a route back, set the default gateway on the AP to the router's IP address.

    You can then access that AP by e.g. http:myrouter.sillydns.org:8001

    If you are using access restriction rules to allow access to the internet, then you must also add the LAN MAC address of the AP to that rule, or it won't be able to reply to you. This is why many people have had problems making it work!

    Finally, to get the time correct on an AP, you have to set the gateway IP as your main router and ALSO set it as DNS, so the AP can look up the timeserver's IP address. If you forget this, you won't get time updated.
  66. BXCracer

    BXCracer Addicted to LI Member

    I have few problems in my qos.
    First of all no matter what upload limit i set for my games class, latency frequently goes up to the 200 ms. and then get back to normal 50 ms.
    And the second thing is that limiting speed by setting up an inbound limit does seem to work at all. I tried setting inbound and outbound limit for my p2p class and the speed is always higher then set in inbound limit.
  67. Toastman

    Toastman Super Moderator Staff Member Member

    It's not so important to limit the games class, it is other classes that must be held in check to allow the latency to be kept low. Post the details,QOS basic and Classification, and I'll take a look see if there's anything wrong. If there are any games ports, just a quick explanation of what they are.

    If you're checking latency by ping - try setting your outgoing HIGHEST class rate to 100% and limit to 100% - and set incoming HIGHEST limit to NONE. Drop your Outgoing maximum bandwidth to around 25% of your maximum, this will make absolutely certain that QOS has some bandwidth to work with - and that should bring your pings down close to normal (as long as other factors don't swamp the router). You can play with the settings later - but doing this immediately brings my pings down to between 23mS and 48mS with no dropped packets.

    Short conntrack timeout settings will control P2P well, but may break VOIP. An assured UDP timeout setting of up to 300 seconds is often necessary for VOIP. Note - the long timeout isn't for the actual UDP voice data, it seems to be necessary for link registration, something to do with the timing of setting up different parts of the link. As I don't use VOIP here, I can't offer much help on this.

    BTW - I'm not a game player and we don't encourage it here particularly, it always causes nothing but trouble every time someone's game server is down or overloaded - so I'm not certain what you would call a good response. Pinging my own ISP gateway while router is under full load (with 32 bytes from Windows 7) gives between 23 and 48 mS with QOS ON, as stated, and no dropped packets. With QOS OFF it rises, anything between 240mS to 1700mS, with some dropped packets. What goes on after that, especially international links, is beyond anyone's control, and the laws of physics being what they are some games will always have excessive latency. The best cure is to either alter the speed of light or move to a different universe.
  68. BXCracer

    BXCracer Addicted to LI Member

    I am attaching the screenshots of my basic and classification windows.
    27015 port is default port for most of the valve games. Currently I play day of defeat : source

    Well your advice doesn't seem to work. The ping is stable know but it's twice as high as the average ping before reconfiguring qos. Please note that there is one user that is constantly using p2p. Maybe i can do something with p2p to decrease the latency for higher classes ?

    You say to reconfigure the HIGHEST class, do i have to reconfigure it even if my game is in the HIGH class ?

    Attached Files:

  69. Toastman

    Toastman Super Moderator Staff Member Member

    I'll look at your QOS. But yes, the HIGHEST class is used for ICMP and other services, so it needs to have super priority for DNS and PING response if you're using it to measure latency. Don't prioritize ACKS if you want best latency in the presence of P2P. The problem usually is not so much the game class needing priority, it is preventing other classes from wrecking the QOS performance. I'll get back to you later... but do try the mods I suggested and see if they do anything.

    What is your default Class?
  70. BXCracer

    BXCracer Addicted to LI Member

    Default is D
  71. Toastman

    Toastman Super Moderator Staff Member Member

    OK... if you follow this exactly you will have a simplified replica of a large building QOS with 70+ mad users - but tweaked for low latency and to suit your useage. With all the evening users on (there are usually about 50) my latency was 23-48mS with occasional jumps up but always below 75 (32 bit continuous pings to ISP Gateway). At the time there were 1300 connections tracked.

    Good luck!


    First, make sure that the "small packet" boxes are ticked at top of BASIC page except for the ACK box.

    Default Class D
    Max bandwidth set to 200kbps for now
    Highest 5-20
    High 5-20
    Medium 5-20
    Low 50-100
    Lowest 20-50
    A 20-90
    B 5-50
    C 5-90
    D 5-90
    E 1-None

    Inbound Limit 2000 (60% of bandwidth for best latency figures).
    Highest 10%
    High 10%
    Medium 10%
    Low NONE
    Lowest 30%
    A 70%
    B 70%
    C 90%
    D 80%
    E 1%

    T/U Dest 53 0-5K Highest DNS
    T/U Dest 27015 High Games (DOD)
    (addit for xbox live - 3074 T/U 88 UDP, src and poss dest - High Games)
    T/U Dest 554,8554 0-5K Medium RTSP
    TCP Dest 80,443 0-256K Low WWW
    T/U Dest 52860,20978,4070 Lowest Spotify
    T/U Dest 25,110 A Mail
    T/U Dest 6667-7002 B IRC
    TCP Dest 20-22,80,442 256K+ C FTP/HTTP file transfers
    T/U Anything you don't want on your network - E, Crawl Class

    Maximum Connections 2000
    TCP Timeout

    UDP Timeout
    10 - Some people have reported they need to stay above 25 here for their VOIP
    10 - Some VOIP users may find connection problems and raising this figure up towards 300 may help. Choose the smallest number that is reliable for your own VOIP system.

    and tick all boxes below

    These settings will expire unused connections quite quickly and you'll see the numbers drop from the default settings, usually stabilising at less than 1000 connections.

    In ADMIN/SCRIPTS/FIREWALL - paste these lines - change the IP's to suit your own if necessary.

    #Limit UDP connections per user
    iptables -I PREROUTING -m iprange --src-range -p ! tcp -m connlimit --connlimit-above 10 -j DROP

    #Limit TCP connections per user
    iptables -I PREROUTING -p tcp -m iprange --src-range -m connlimit --connlimit-above 100 -j DROP

    #Limit outgoing SMTP simultaneous connections
    iptables -I PREROUTING -p tcp --dport 25 -m connlimit --connlimit-above 10 -j DROP
    (limits any mail virus activity)

    These are designed to throttle P2P'ers from hogging bandwidth. You might need to adjust them.

    If you test the above limit scripts with a limit of say 5 connections in the line, you will often see that it doesn't appear to be working, you will have many more connections than your limit, maybe 30-100, that you can't explain. Some of these may be old connections that have not yet timed out, and waiting for a while will fix it, but others may appear instead. Don't worry, you can see it's actually working. Be aware that often these may be TEREDO or other connections associated with IPv6 (windows Vista, and 7) which is enabled by default. You should disable it on your PC by command line:

    set state disabled

    Also, if you use Windows XP, set the maximum half-open connection limit to 1000 using the usual patcher (google it).
  72. BXCracer

    BXCracer Addicted to LI Member

    Well thank you, that worked really well except that I had to increase the upload rate for E class.

    Thanks again.
  73. BXCracer

    BXCracer Addicted to LI Member

    its between 30 ms and 60 ms to my ISP's gateway

    Edit: well it seems that I was happy too soon. Sometimes the clients that uses p2p can overcome all those outgoing and incoming bandwidth limitations and blow the torrents in full speed. Naturally this causes ping to be sky high. Any advices ?
  74. Toastman

    Toastman Super Moderator Staff Member Member

    P2P can and will find ways to screw everything up. Try to find out what it's doing, probably using a port that you have set for a higher priority. Look in QOS - Details for all connections, and see if you can tie anything up with your P2P machines that shouldn't be there.

    I've stopped using the E class to try to classify seeds. It does not work as intended. I recommend not doing this. I'm using it to throttle those damned UDP and uTP connections that consume bandwidth and don't result in any downloads.
  75. BXCracer

    BXCracer Addicted to LI Member

    Well you are right again :)
    I see p2p connections with S ports 4832, 2066 and D ports 45442, 19166 and lots of other random ports that fall under UNCLASSIFIED. I quess that where my problems come from. Any way to solve this ? Maybe i should just apply l7 filter for p2p keeping in mind that there is only 2-3 users online at a time.
  76. Toastman

    Toastman Super Moderator Staff Member Member

    This is the problem. Anything not covered by your rules *should* bypass them all and end up in D. The unclassified area is usually cluttered with other unexplained stuff, but I will post an explanation for this soon [see #93]. This isn't the problem.

    Some degree of "bleeding" through of p2p to other classes is permissible and usually doesn't do too much harm, but if your aim is fast latency, then it's more critical. Try anything you think might work in a new rule and post any finding! Don't pin much hope on L7 filters though, I find them next to useless!

    The really important things are to get UDP under control and limited, DHT is a big problem, loads of traffic but resulting in no downloads. Bit Torrent DNA bandwidth thieving program is another headache, always get rid of it.

    EDIT: This has all changed with the issue of uTorrent v2.0. This version uses a new protocol which is carried over UDP. So from now on, expect more P2P UDP traffic than TCP. As far as I can see, however, no change in QOS setup is necessary. See posts below.

    EDIT: Many items in the QOS monitor pages are actually ports that haven't been closed yet and are waiting for the various timeout periods to be expired in conntrack. Click on "drop idle" in conntrack, you have a 17 second count until they expire to navigate back to your graphs to see the result, often almost all of the unclassified connections are dropped. If you see them start to come back again, then read post #93 below.
  77. Toastman

    Toastman Super Moderator Staff Member Member

    Unclassified Connections - an explanation

    One of the biggest puzzles for many users has been the plethora of connections cluttering up the "unclassified" section of the QOS page. The wiki states that this is traffic destined for the router, and offers no more explanation. Users can easily see, however, that most of it has a source and destination port that is not the router, and is usually associated with p2p traffic. They find it impossible to classify this with any rule.

    OK, here's are some clues. See the source (=local client) port in most of these offending connections? Using that port number, let's make a new QOS rule, let's make it for both source and destination port, put into some class we can monitor, it doesn't matter what as long as you know what it is. Commit. Now all of this traffic should be classified, right? Er .. no, it's still there.

    Now switch off our p2p client. Go to ADVANCED/CONNTRACK and click on "drop idle" connections. Switch back to QOS graphs. The junk is still there, right? And it keeps changing, some disappear, some new ones open, right? Wait for some time, even after several hours. Some are still there, yes? Wait a week, they're (mostly) gone...

    Now, force a reconnect to your ISP to change your WAN IP address. Immediately they all stop.

    So what is the explanation?

    They are usually incoming connections from remote computers that have been contacted by your p2p application (etc) and are attempting to connect TO your assigned incoming port to download your shared files. Your p2p client may not acknowledge some or all of these, as they may have already been timed out by the router, the port is probably no longer open. Or - you may in fact have switched off your computer and gone to bed, or maybe you restarted your p2p client with a new randomly selected port number. But the remote site will not know that and will still keep trying to connect to your p2p application. Depending on what P2P software it is using and how the remote computer is configured, it can do this for a very long time. And there can be many of these connections which don't go anywhere - i.e. they stop at your router. I often have over 100, some of them are days old. To check this, I changed the p2p port assignment, but many incoming connections were still trying to use the OLD port several hours later. Often, these are machines trying to contact your P2P client because the tracker registered that you have a file part that someone wishes to download.

    So, they are not classified because they were unsuccessful attempts by a remote server at making a connection - and they STOPPED AT YOUR ROUTER because there was nothing listening for them to connect to. They will time out, but new ones will keep being opened until the remote site(s) gives up altogether.

    They do no harm in the "unclassified" section, that is what it is for, it is just information for you.... so just leave them alone....

    NB - There are many other kinds of connection that also stop at the router, you should examine the source and destination ports very carefully for clues. Don't assume your QOS is faulty and screw it up by fixing things that weren't broken! If you can't see any reason for them, my advice is to just ignore them.

    Some applications themselves cause connection storms. Take a look at these posts by Planiwa, before you enable any software that does things without your knowledge or consent :)



    Be aware that many P2P programs will cause connection storms that will crash your router. It's up to you to find ways to protect it.
    Duniker likes this.
  78. Kiwi8

    Kiwi8 LI Guru Member

    I see, thank you for the detailed explanation. :)
  79. Toastman

    Toastman Super Moderator Staff Member Member

  80. QSxx

    QSxx LI Guru Member

    Here's the quickie for you Toastman:

    What's the practical use of having QoS and IP/MAC BW limiter working simultaneously?


    How should one setup IP/MAC BW limiter if QoS is configured following your guide (which works great btw, in my case at least)?

  81. Toastman

    Toastman Super Moderator Staff Member Member


    I have not really used the IP/MAC bandwidth limiter in earnest because my personal feeling is that it's better to use QOS to allow fuller use of facilities when nobody else is using them.

    MAC/IP limiter and Robson's script generator are supposed to work on LAN based traffic before it passes to the router's own QOS. It is said that they work independently and can work together.

    However, I did experiment at one time and found that using it in conjunction with QOS was unpredictable, and in my case, did not seem to work properly (neither did Robson's generator). There seemed to be some conflicts with QOS. Since both use iptables, this seems to be quite possible. Other people stated they did use it and had no problems. This may be because their useage is light, whereas most of my sites are very heavily used. Or they may have been lucky and just chosen a configuration that didn't conflict with QOS. I pass.... but this is why I have never really made much of it in these posts!

    I find that you can achieve the same or better results with normal QOS, if that is what you want to do.

    A TIP when using IP/MAC Limiter - Some people have found that the TC Tag setting of 10 gave problems, and the limiter worked correctly when they changed it to a higher number.
  82. menses

    menses Addicted to LI Member

    Well, I was just thinking how to make both QoS and BW limiter work together. But maybe I should just give up :)

    I recently set up a tomato box for a three person shared flat. Without any shaping/limits the connections became slow due to P2P... Initially my plan was to set QoS, but it required too much time. Setting up is not that hard, but I think it also requires time to analyze the traffic and change rules accordingly. And I don't have the time to do that, especially when I don't live in the flat my self (I haven't found a convenient way to do it remotely).

    So this is why I ended up using the IP/MAC BW Limiter and divided the bandwidth in three equal slots.

    However I have one question... According to Victek's BW Limit manual the Download and Upload Bandwidth should be tested with some speed test "service" and then put these exact (100%) values in the fields.
    But according to your QoS manual, the bandwidth should be around 85% of the lowest maximum speed tested.
    Maybe I should have asked this in Victek's thread but what's your take on using 100% of the max bandwidth value in the BW Limit QoS?
  83. Toastman

    Toastman Super Moderator Staff Member Member

    I don't recall that that was in the BW limiter, but remember, that isn't Tomato's QOS, but an addon and since I don't use it I can't make much comment. But for any QOS, you MUST set less than the minimum you measure or QOS can't work properly. It is explained quite in depth in an earlier post in this thread why. You can test this statement at any time by keying in the maximum value, when you will see that QOS immediately fails under heavy load. This statement refers only to the "OUTGOING" limit. The incoming maximum figure figure is covered elsewhere - it isn't actually a limit at all. For your interest, most of my connections are nominal 512k UP but the QOS max limit is set to 250k or 350k. This ensures that, for my particular setup, that QOS always works as intended, no matter what.

    Setting up QOS is not so time consuming if you start from a known working setup. You would almost certainly find that an exact copy of my QOS rules would *work* for you. It works for many of my clients, and they probably do much the same as yours. I also use exactly the same rules in any site I administer, no matter how many users, 3 to 200+ ---- go ahead and key it in! I assume you can get remote access by a dynamic ddns server? Once you have it working, then you can improve it to suit your own setup.

    UPDATE: This is a good one to try: http://www.linksysinfo.org/forums/showpost.php?p=357556&postcount=135
  84. cr48202

    cr48202 LI Guru Member

    Hi Toastman,

    I have followed the information in the above posts. For the most part everything seemed OK. One problem that was encountered though was that my kids were getting messages in Windows Live Messenger when they sent messages to others that

    "The following message could not be delivered to all recipients:"

    If you could please help me in tweaking the settings above so that these messages do not happen anymore, it would be appreciated.

  85. Toastman

    Toastman Super Moderator Staff Member Member

    The QOS rules you are using were tailored for one specific user who did not use Messenger (important port 1863). You might be better off using the rules suggested in later posts - always find the latest one in the thread.

    Using the following rule would address most IRC/Messenger etc protocols such as IRC,MSN,WLM,ICQ,Yahoo,AOL:

    T/U Dest 94,1503,1863,3389,5000-5010,5050,5100,5190-5193,5222,6005,6667-7002,6901,8000-8010 B - IRC,MSN,WLM,ICQ,Yahoo,AOL etc

    However, I'm not sure if that message is due to a fault, maybe the problem isn't with your router at all. I also often get this message lately. It probably means the users is offline, no more. So if this line doesn't fix it, search Google for some other reason.
  86. Dashiell

    Dashiell Network Guru Member

    I have a question on the outbound E class setting... does setting X-None for the outbound bandwidth limit it to the value of X, or, as I understand it works for the inbound bandwidth, does setting none tell the QoS service to "ignore" the ceiling for E class? For instance, if I place "3%-None" and get a value of 130k, is it locked to 130k?

    Also, I'm assuming the "5-5-" for the outbound B class is actually a typo... I'm assuming you meant "5-50."

    Thanks, Toastman!
  87. ctrl_alt_id

    ctrl_alt_id Guest

    WDS - Speed and reliability

    Hi Toastman,

    I've done a search in the forum looking for further detail about the issues you mention with WDS, but the forum does not show any results when I search for WDS. Can you give a little more information on the issues with WDS?

    Also, you mentioned you run an apartment block. How do you have each device setup? I've got a couple of WRT's covering my house and it is nice that the clients don't drop connection when moving from one device to the next. This includes vpn connections and iphones (which seem to not roam very willingly even when another app offers a better connection, but they will switch when using WDS).

    Thanks for your QOS article. I still have more to read, but I'm making my way through the article.
  88. Toastman

    Toastman Super Moderator Staff Member Member

    ctrl-alt-id - If your AP's use the same SSID and encryption then the connection should not drop when you roam. In these blocks you can pass 28 AP's without noticing any switchover while streaming audio or video. The AP's do not have to be on the same channel.

    Hmm. The forum search utility seems to be almost useless, maybe it's broken, or I don't understand how to use it either. I recommend just paging down through the thread lists and you will see many discussions about WDS.

    WDS slows down your speed because the connected AP has to receive data, switch to transmit, send it to the router, switch to receive, wait for a reply, switch to transmit, send it to you. It cannot receive while it is sending, so the bandwidth is halved. If you put another AP on WDS, then it is halved again because all other AP's have to wait for it to finish sending - then they do the same thing themsleves. And so on. If you have several, plus other AP's, say 20-30, then of course the whole thing becomes unuseable - this may not apply to your situation though. Also, WDS is, in my experience, very unreliable - lots of dropped connections. In short, I personally view it as somewhat of a total pile of poop, and I would always recommend hard wiring AP's to the router, if at all possible.

    In fact, in my building obviously we use wireless to distribute internet to people in their rooms - laptops, PC's. Whatever. It is cheaper than wiring every room with cable. But wireless can never be as fast as cable. If you don't need to use wireless, then avoid it. Wifi is a half baked and unreliable connection medium and will always be so. It's getting worse with "N" - not better.

    All AP's in all my blocks are hardwired to switches and then to the central router(s). On occasions when we have outside activities I used WDS to provide an extra AP out in the grounds. It is rarely a trouble-free event.
  89. bangkokiscool

    bangkokiscool Addicted to LI Member

    First time poster on this forum, and a big THANK YOU and thanks to Toastman for sharing all this information.

    I'm not a network expert, have zero formal network training, so my setup has been largely on trial and error. I manage a 12-unit apartment building, about 30 residents, in Ohio. There are 6 Proxim Orinoco AP's in the building, running over PoE to a netgear PoE switch, and then to a Netgear router, then to a cable modem. This setup has worked fine for the last three years, but this year many residents have complained about outages and slowdowns. I suspect torrents are to blame (isn't it always?). So I installed an Asus gp500 v2 with tomato 1.25 today, and I've implemented the QOS that Toastman recommended. I hope it does the trick!

    A couple of quick questions.

    1) according to my speedtests the connection ranges from 4.5 MB/s to 10.11 MB/s, and the upload is always 0.48 MB/s. I've set the QOS max bandwidth on the outbound at 250 kbits/s and the inbound limit at 4000 kbits/s. Does that sound about right?

    2) Your sample QOS (on page 2 of this thread) lists 2 L7 classifications, rtsp for video control, and shoutcast. There's another for a Thai tv service (I think). I assume these are unique to your setup?

    3) If I just paste the firewall scripts, do I need to reboot the router for the scripts to take effect?

    4) I manage remotely, as I don't live on the property. How do I know if QoS is working? I don't know how to tell, through Tomato, if any users are experiencing slowdown on web traffic and they don't always tell me, they just get frustrated and move out.

    Sorry for the newbie questions, and thank you again!
  90. Toastman

    Toastman Super Moderator Staff Member Member

    Hi and thanks for the post! You'll probably get experienced quite quickly with Tomato, the remote monitoring facilities are pretty useful. It does help to have someone who is able to give feedback from the apartments though.

    1) That's a big change in measured speed with time. But your setting suggestion is fine. However, it does waste the higher 10mbps bandwidth that is there from time to time.

    OK, here's a way to deal with it. You could set your outgoing limit (the most important thing) to 250 initially (when it's working, try increasing up to about 350). Set "inbound class limit" at 4.5 Mbps. (Note that this box is not an overall limiter - it is just a figure used to calculate the percentages for the individual class limits).

    Then limit the individual classes (P2P etc) on the basis of your LOWER speed. Say 3Mbps for mail, downloads etc - 2.5mbps for P2P and so on. But - set the limit for HTTP etc (the high priority class) to NONE so that web access etc. will take advantage of whatever higher speed becomes available (your 10Mbps). That way things should work fine even at the lower throughput times of the day but HTTP will become more responsive as the bandwidth increases. After a while you can experiment a little, and maybe set other classes to have no limit also - but be careful.

    2) The TV service is a special local service (buddhist temple). No need for you to use it. The L7 classes were used to try to trap some services that I could not obtain port numbers for. Generally, it is better to use port numbers. Currently I use only destination ports 554,1755,1935,5004,5005,5060-5063,1719,1720,3478,3479,15000 at 0-50k - and I stopped using the L7 filters. The Shoutcast ports were mainly gleaned from Winamp shoutcast. See what works for you. I found that running more than a couple of L7 filters slows down the router, while the filters were not even particularly effective. As we get faster and better SOHO routers, L7 filters will be easier to use. Some of them are becoming quite good nowadays.

    3) After pasting the firewall scripts, they won't take effect until the service restarts. You can reboot, but you can also just disconnect/reconnect to the ISP from the "Overview" page - which also restarts the firewall. If you gain access via SSH etc you can also start and stop services via the command line (along the lines of "service iptables stop/start/restart") but personally I stick to the GUI.

    4) The pie charts are your best source of information coupled with the realtime/24 hour charts. You need to get experience in using these, by making changes and then monitoring. With your users online, you have to be careful with this, but there's no other easy way to learn. However, in general, whenever you see your download bandwidth below say 60% of your maximum, you can assume everyone's connection will be quite fast. Above that you really need to get some input. If you see your download charts pegged at maximum, and flat topped, then you can safely assume your link is saturated, that there is a big queue waiting to be sent to you from the ISP, and that pretty much everyone will be experiencing slowdowns and timeouts. You can ping the ISP's gateway from the router's GUI and see what the ping time is, which will tell you if there is a holdup.

    Problem Users

    Some of my remote sites also have the same problem with clients moving out. So let me tell you what I've found in my apartment blocks and hope it may be useful to you (and others) as part of this thread.

    Generally, the majority of our users want only to browse the web, use email, text messaging etc. Some want to use audio and video messenger services (generally these work OK if not too many people). The problem users - the ones who hog all the bandwidth trying to download large files with P2P - get very angry if they are not allowed to do exactly what they want. These have always been the ones who moved out when thwarted. I'm afraid this is the same anywhere, the immature, spoiled brat syndrome.

    So, P2P is usually your greatest problem. It's true that there are a lot of people whose main reason for wanting the internet is to download files, but in the end, it's not ever going to be very fast on a shared connection. Providing they understand that, it's OK. But most of them are unreasonable, I've found, and they expect to get exclusive access to the ADSL line, and refuse to accept any limits. We generally find it's a great relief when they leave, we miss their money but not their person :biggrin:

    When they have clogged up the internet with their P2P they will often try to accuse other users of being the culprits and try to get the management to ban other users. They will usually say they know who is doing it, it is usually someone in a room near theirs that they see watching a movie on their computer, or similar, and try to get them banned or kicked out. They are usually paranoid. They will also try to get access to the AP's and routers, and often pull out plugs and cables, for some odd reason I've never understood, they especially like cutting the power. Probably they find that resetting the routers/AP's like this can sometimes give them an advantage when they reboot. All this usually happens late at night when nobody sees them creeping around. So I found it absolutely essential to put all AP's in enclosed boxes with no visible cables or access. Placing CCTV cameras on the floors also worked wonders.

    To stop them messing about, I use fibreglass electrical housings to enclose the AP's, with the power cable from the central UPS and CAT5 well protected by conduits/cable trays. Routers are in steel locked electrical cabinets, generally close to the reception areas well out of sight. I generally have a minimum of 2 ADSL lines and routers so as to a) double up bandwidth b) I have 2 routes for remote access should a single router go down. Most of the time it is possible to telnet on the internal LAN from one router to the failed one and bring it back up via command line. I also port forward to all AP's to allow remote access for diagnostic and admin. purposes.

    Some condos here actually have one ADSL line per floor, which is not only unprofitable, but can become an admin nightmare, and the P2P users are still not satisfied with sharing even with a few others. The performance, with normal users, isn't actually significantly faster, and the solution is very expensive because one building can end up with over a dozen ADSL / telephone lines! For most normal users, a single ADSL line of 5Mbps upwards shares well between 100 users, with very few people being aware they are sharing at all. If you really must cater to P2P, it is often possible to add a second router/ADSL line to one of the AP's and divert the worst P2P users to that new gateway, which can help the others who were suffering. Oddly enough, it doesn't usually improve the happiness of the P2P users though. I am a big torrent user and the only thing that satisfies me is to use maximum bandwidth - this is human nature. But that of course means a dedicated line. Try to get them to understand that isn't possible :eek:

    BTW - While setting up QOS you may find that labelled classes are helpful. I compiled a version with the following class labels corresponding to the normal Highest to E.


    Games Control
    Media Control

    This is a modded Victek's 1.23 RAF version that works fine:

    http://firmware.mooo.com/Toastman Builds/Toastman 1.23 Builds/Toastman 1.23 ND_MiniUPnP Builds/tomato-ND_4515.5 based on 8515.5.trx

    [EDIT - in later versions I include the ability to change the class names to suit your own taste]

    Also, you may find it useful to know something about connection storms, mainly from P2P applications. Planiwa in particular has been trying to focus people's attention on this problem. Read this thread:


    Limit UDP

    (These scripts may be useful to slow down connection storms - just trying to evaluate them now. Be warned - they don't always appear to work. Try if you wish and keep an eye on the thread for updates). Use iptables -L to list chains and look at the rules to verify they are there.

    #Limit UDP opens from all users - UDP to Router
    iptables -I INPUT -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT

    #Limit UDP opens from all users - UDP out to WAN
    iptables -I PREROUTING -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT

    Conntrack Timeout Values

    Keeping connection count low by expiring connections faster prevents a lot of router crashes and resets, particularly caused by P2P. Here are settings I am using now which seem to work OK with no reported problems. Just enter them in the Advanced - Conntrack page in this order:

    TCP Timeout - 100/1200/20/30/30/20/20/30/30/120
    UDP Timout - 10/10

    With these settings you will find the connection count rarely goes above 1000. Be careful with UDP timeouts if you use VOIP. Use 10 and 25 as a base - note that some people have found that the assured value needs to be increased to keep connectivity, increase towards 300 but choose the smallest number that is reliable for you. In fact reducing them ALL except TCP established to 10 works fine but you may have to tweak some of them if you have VOIP problems.
  91. bangkokiscool

    bangkokiscool Addicted to LI Member

    Thank you again, for being so helpful. Your idea of setting http traffic to "none" is a great one, and I'll try that.

    As I understand the QOS rules you set up on page 2 of this thread, the default class is D, and you've used ports to define the vast majority of traffic types your building typically sees. You've left Class D undefined, so typically, Class D traffic is p2p traffic. You've allocated Class D 90% of the incoming bandwidth, presumably so that some p2p traffic can still come through (although limiting the upstream, presumably, means there is some throttling of p2p going on). Is that accurate? If I do get further complaints about web browsing slowdown during peak time, would reducing that 90% down to 40% or so be your first recommended step? Thanks again.
  92. Toastman

    Toastman Super Moderator Staff Member Member

    Yes, that would be good. In fact I would set it lower (maybe 2.5) especially at first - but the amount you allow is up to you to determine by experiment. Higher than that allows more P2P but gradually chokes everything else. Back then I allowed more P2P here, but after a while I cut it right back, nowadays I allow a maximum of about half of my max incoming bandwidth.

    I used class E to try to classify uploads (seeds) from P2P but I have since stopped using that because it did not work as intended. Now I use it to kill off anything I wish to ban.

    To give you an idea of what response to expect - when QOS is working well, with typically up to about 50 PC's online including maybe 10 or so P2P users and incoming traffic running at about 80-90% of max, a reasonably complex web page should open in about 4 seconds. Google searches take about half a second. (I have 5 to 16 Mbps ADSL lines). Local IPTV works with no dropouts. Yesterday Victek and I had Messenger video running both ways (he's in Spain, I am in SE Asia). At the same time, my other machine was maxxing out the link downloading 40 or so videos with uTorrent. Plus all the other users. Web pages still responded in about 4 seconds.

    You may be interested to know that flashing new firmware over the web is almost always successful. The trick is - wait a LONG time before assuming a dead connection. Ten minutes is pretty average, but can be much longer on dodgy links. I maintain some relative's routers which are 8,000 miles away.
  93. bangkokiscool

    bangkokiscool Addicted to LI Member

    Hi Toastman,

    I'm monitoring from less than a mile away, actually. And from this distance, flashing over the web has been a snap, less than 3 minutes per router!

    Is pinging from the GUI the best way to determine if the connection is "speedy" when I'm monitoring remotely? What's a good response rate? Thanks again!!
  94. Toastman

    Toastman Super Moderator Staff Member Member

    Pinging from the GUI is a good way, but I usually use Windows ping in a box on my monitor where I can see it all the time. You need to do it when the bandwidth is pretty well full if you want to check your rules work. I leave the 24 hour chart running and watch it to choose the time to ping. My lowest response to pinging my ISP's gateway is 21mS and normally the average hovers around 35mS. Occasional spiking up to 100mS which is fine for my use, response to WWW is fast. It is when it starts to go out above 100mS - to the hundreds, that things slow down considerably.


    ADDIT - of interest, with an ASUS RT-N16 router the fastest ping response dropped from 21mS to 16mS. That's not bad, uh?
  95. bangkokiscool

    bangkokiscool Addicted to LI Member

    I am still getting client complaints about slow WWW browsing. It happened last night (when I was not monitoring, unfortunately). Looking at the 24 hour chart, it looks like there is sustained traffic hitting the system from about 2130 to 2330 (see attached).

    How do I tell what is causing this? Do I look at QOS charts to figure out where the bulk of the traffic is coming from? Is a chart like this almost certainly caused by p2p, or is that too far of a conclusion to reach?

    Thanks again for any insights!

    Attached Files:

  96. Toastman

    Toastman Super Moderator Staff Member Member

    Can't tell what it is. Might be an HTTP download. Can you take some screenshots of your QOS Basic, Classification, Conntrack files and put them in a PM to me? I'll see if I can spot anything.
  97. Toastman

    Toastman Super Moderator Staff Member Member

    After a lot of experimentation with different scripts, these really work (individually) to limit UDP packets. They are for pasting in the firewall script box. NB - These scripts prevent too many packets being generated too quickly - they don't limit the NUMBER of connections.

    Thanks to ntest7 in this thread: http://www.linksysinfo.org/forums/showthread.php?t=62860&page=3

    Adjust the settings to suit your circumstances. 10 and 20 seem reasonably OK here for normal use - you will need to increase this if your VOIP stops working. Both may not be needed. The first acts on the INPUT chain and the second the same thing on the PREROUTING chain.

    #Limit UDP opens from all users - UDP to Router
    iptables -I INPUT -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT

    #Limit UDP opens from all users - UDP out to WAN
    iptables -I PREROUTING -p udp -m limit --limit 10/s --limit-burst 20 -j ACCEPT
  98. ferox

    ferox Guest

    Not Classifying

    Did Toastman's 'Basic Settings' as a starting point for my setup. However, my connections doesn't seem to be classifying itself:
    It's like it only recognizes DNS? This with torrent on, WOW, skype and web running. What am I missing?

    Update: Reflashed firmware. It's working now.

    Attached Files:

  99. Toastman

    Toastman Super Moderator Staff Member Member

  100. michse

    michse Addicted to LI Member

    Hi Toastman,

    great thing. it helps to understand a lot. but really, I lost the red line on site 2 or so because it's not my language :)

    2 Questions.

    Simple one, I wonder you drop via iptables all INPUT and PREROUTING udp and after that you allow something. Did I missunderstud iptables or why can you first drop and than allow something what should be dropped??

    Second question. I seperate the ports via vlan, wlan and one port is still on br0 ( so friends can surf via wlan. me on the vlan2 ( works well, but qos seems to be not work correct. all Traffic from vlan2 is directionmissmatch, source is dest on ports and ip. My extern IP is almost the destination, my connection from pc to router is always the pc the destination and router the source. I think, its not only a gui problem but dont know. Can you or somewhere say something about that? My idea intern it works with br0 but vlan2 is not on br0 and so the missmatch begins. friends connection are correct displayed.

    thank you

Share This Page