Weird tomato wireless behavior on RT-N66U (possible bug)

Discussion in 'Tomato Firmware' started by alpovs, Apr 18, 2013.

  1. alpovs

    alpovs Networkin' Nut Member

    This has been an ongoing problem on my RT-N66U since its day one in my possession. I immediately installed tomato on it. I tried Shibby and Toastman builds 3-5 versions back that would include the previous wireless driver version in those builds. I erased NVRAM several times when I was changing the versions and applied configuration changes manually. I even decided to update the CFE bootloader to version using the method described on SmallNetBuider forums. In order to do that I installed the latest non-beta version of the ASUS firmware, then I installed the latest 64K Shibby AIO version of tomato. I observed this problem on ALL versions of tomato I used on this router.

    I have been using tomato since its first version, when WRT54G was considered a modern router. I did NOT observe this problem on any other router running tomato (my previous router was RT-N16). I did not run the ASUS firmware on my RT-N66U long enough to observe this problem.

    OK, now the problem itself. Have a look at the attached picture. After a while after reboot, sometimes 2-3 days, sometimes several hours, I see this information in the device list. The quality is above 100, the RSSI's are well above 0 dBm. The 2nd and 3rd devices are webcams that are always on. The first device is my Android 4.2 phone that connected to wireless 15 min ago. The 4th device is an Android 2.3 phone that left the network about 4 hours ago! Its owner left with the phone. Normally such devices disappear from the list after a while (minutes). This is all 2.4 GHz band. The problem is that in this state new wireless devices cannot connect to the router while connected ones stay connected. However, devices still can connect on the 5 GHz band. So, the problem is limited to the 2.4 GHz band.

    Rebooting the router definitely helps. Later I found out that clicking "Measure Noise floor" button on the Device List page also helps. The RSSI and Quality levels go to normal/reasonable and new devices can connect again. By "new" I mean newly turned on. So it seems that the problem is related to the 2.4 GHz band driver or how tomato interacts with it. There is absolutely nothing about it in the router logs.

    I googled, searched various forums but never came across any posts describing something similar. A lot of people run tomato on RT-N66U. This problem would bug somebody. Now I am thinking that maybe my router is faulty but its hardware seems to function. This problem happens randomly and disappears after measuring the noise floor or reboot, so I blamed the firmware. But how come nobody else seems to have it? Any thoughts?
  2. gfunkdave

    gfunkdave LI Guru Member

    What Tomato version are you using?

    Have you cleared the NVRAM recently?

    I have a RT-N66u running Toastman and have never seen this.
  3. alpovs

    alpovs Networkin' Nut Member

    I answered both of your questions in my original post. I used several versions of Toastman and Shibby tomato as they progressed. I normally cleared NVRAM after upgrading. Couple of times I didn't do it to see what happens. It didn't make a difference. Several times I cleared NVRAM without changing the firmware. I always used 64k versions. What version are you using?
  4. RMerlin

    RMerlin Network Guru Member

    These info are reported by the (closed-source) driver. Not much that the Tomato devs can do about it. I have seen it occur myself a few times on my own firmware, mostly when I was sending too many queries to the wireless driver about wifi status, so it's most likely a bug with the driver itself.
  5. Monk E. Boy

    Monk E. Boy Network Guru Member

    I've found that the only times I have a connection problem like this is when a neighbor decides, in their infinite wisdom, to change the channel their router is using and plop it smack dab in the middle of the channel I was using. Its especially bad with 2.4 since you effectively only have 3 channels (1, 6, 11) so a neighbor putting his router at, say, 8 ends up screwing up both 6 and 11. Using a wireless sniffer, like inSSIDer, to determine what channels are in range and choosing a range that's free un-gums up the works... until the next time they do it.
  6. koitsu

    koitsu Network Guru Member

    The issue described by the OP sounds like a wireless driver quirk/issue, and there isn't anything anyone can do about it (or even look into it by trying to debug it) because the wireless driver is a binary blob (closed-source) driver provided by Broadcom.

    What I would like, however, is this "list of several [Toastman and Shibby] versions" that were tried. I want actual filenames of the firmwares you used. The wireless driver has changed, especially in the most recent couple of builds. What changed in the driver we do not know -- again, closed-source binary blob. We just know the driver has a newer build date (it's stored in the driver binary itself).
  7. alpovs

    alpovs Networkin' Nut Member

    Hi koitsu,

    Right now I am using the latest Shibby build. Filename: tomato-K26USB-1.28.RT-N5x-MIPSR2-108-AIO-64K.trx. Before that I used probably 4 versions of Toastman builds. I was updating them as they were coming out because I hoped that problem would be solved in the next version. For example, I found two files on one of my computers that I used: tomato-K26USB-NVRAM64K-1.28.0501.3MIPSR2Toastman-RT-N-VPN.trx, tomato-K26USB-NVRAM64K-1.28.0502MIPSR2Toastman-RT-N-VPN.trx. The previous versions files are on other computer(s), they are from the same branch except for one of the earliest that I used - I tried the VLAN branch but later switched to normal branch because I didn't need VLANs and thought they could be causing this issue.

    I guess, 5 successive Toastman versions back I used a Shibby build whatever version was current at that time. I don't remember. I bought this router shortly after the 64k builds appeared. I immediately installed Shibby's tomato on it. Because I am so used to tomato and use features not available in ASUS firmware.

    As I mentioned in my OP I am aware that the wireless driver was updated in tomato. But it happened only once since I bought this router and it didn't change anything. The latest driver in tomato is old compared to that in the ASUS firmware. Look like ASUS updates the wireless driver almost every firmware release. I don't know how easy (or difficult) it is to extract this driver from the ASUS firmware but, Shibby, please try to do this for the latest version. You create builds specific to RT-N66U.

    I am aware that the driver is closed source and available only as a binary.
  8. alpovs

    alpovs Networkin' Nut Member

    Randomly browsing these forums I found another similar report:
    I would like to stress that it is not the weird levels that bother me, it is that newly turned on devices cannot connect to wireless. When this happens I open tomato's Device List page from a wired computer or from an Android phone connected on a 5 GHz band and see those strange levels. When I reboot the router or lately click on Measure Noise Floor, the levels go to normal and new devices can connect again. I would say that the weird levels I observe are a side effect of a more bugging issue.
  9. alpovs

    alpovs Networkin' Nut Member

    Monk E. Boy,

    My neighbors are quite far away. By using the built in tomato wireless survey I see 5 weak APs. The strongest of them is on channel 1 and I am on channel 11. There are two APs on channel 11 but their signal is barely detectable: -84 dBm. That is why I doubt that this is caused by interference. I also played with interference settings in advanced wireless to no avail.

    There is a slight possibility of interference from wireless electricity meters, mine is not far away from the router. After some attempts I could not find out which frequency is used by meters in my city. But my problem shouldn't be happening. Once wireless "gets stuck" it stays "stuck" and does not accept wireless devices until reboot or noise floor measurement.
  10. koitsu

    koitsu Network Guru Member

    Read everything I have written below. No skimming, read.

    The driver in the Asus firmware should not be immediately backported into TomatoUSB. This is a very complex and very painful task, and the risk of breakage is very high. You need to step back and think about the implications for others. If the stock Asus firmware works for you, then use that please. I cannot stress this enough.

    I'm still not exactly seeing what your issue is. All I see is some arms flailing and some sort of strange correlation between "the numbers and information I see in the GUI" with "I can't connect any 2.4GHz wireless device to my router any longer".

    The reason "Measure Noise Floor" """fixes""" the problem for you is because that actually resets the wireless driver's internals (and possibly part of the wireless chip inside of the router). Thus, if there is a bug in the wireless driver, then there is a bug in the wireless driver. We cannot fix or repair or accurately debug wireless driver problems because the driver is a binary blob (closed-source).

    You also haven't given any details about your wireless configuration -- specifically everything under Basic -> Network -> Wireless. I also want to know if you changed any settings under Advanced -> Wireless, and if so, what you changed.

    Did you also do a thorough NVRAM reset after switching between all these firmwares (you've been through at least 5 from your statement)? I'm talking about Administration -> Configuration -> Restore Default Configuration -> Erase all data in NVRAM memory (thorough), and afterwards manually re-entering all of your changes (do not use the Config Restore capability, as that will just reintroduce NVRAM variables that could be causing issues). There are known situations where "special" NVRAM variables can sometimes cause the wireless driver to act weird (e.g. firmware X uses wireless driver x.y.z which relies on NVRAM variable abc, but the driver uses it correctly. You then change to firmware Y which uses wireless driver x.q.q which also uses NVRAM variable abc but the driver has bugs and the existence of the variable causes anomalies).

    I also recommend you re-read Monk E. Boy's post above. What he's saying is that your 2.4GHz devices might suddenly stop connecting/crap out as a result of excess 2.4GHz traffic from neighbours and surrounding devices (that includes things like microwaves, baby monitors, etc.). The wireless driver inside of a router has to try and deal with this as best as it can -- when its inundated with a constant barrage of crap from outside sources (which you have zero control over), the driver tends to wedge in strange ways (existing clients might continue to work, but new ones might fail to connect; sometimes all clients stop working altogether). I've seen this on both Broadcom 802.11 chips as well as Atheros 802.11 chips. Welcome to the main reason wireless technology is utter, complete crap -- I'm serious, I'm not just ranting/bitching, it really is garbage.

    Read his post again, slowly, he tells you what to start with, re: diagnosing the issue) You should also try doing "Scan" under Basic -> Network -> Wireless, let it run for about 5-10 full minutes, and then write down/post here the number of networks/devices seen using each individual channel shown. It matters!
  11. koitsu

    koitsu Network Guru Member

    BTW, about the Asus driver -- if you looked at the ChangeLog Toastman provides, you'd see this:

    February 25 2013 - 1.28.0501.3 and variants
    recompiled March 7 2013
    Revised/recompiled BCM driver backported from Asus GPL (may 2012 issue)
    So if you tried 1.28.0501.3 before March 7th, then you were using the "older" RT-N driver, while from March 7th onward (including in newer versions like 1.28.0502 and up), the above Asus driver is what's used.

    I was involved in this discussion/came across this mistake/mishap myself, which is how I know about it.
  12. alpovs

    alpovs Networkin' Nut Member


    Read my post above in response to Monk E. Boy. I guess you were typing when i posted it.

    Read my original post. No skimming, read. I mentioned there that "I erased NVRAM several times when I was changing the versions and applied configuration changes manually" afterwards. Judging by your post I can tell you that I know much more about tomato, electronics, closed-source drivers etc. than you expected me to know.

    I tried changing all the obvious settings. In advanced wireless I tried all the settings under: Interference Mitigation, Bluetooth Coexistence, APSD Mode. In all cases the router suddenly stopped accepting new wireless clients at some point in time.

    My problem exactly is that my router suddenly stops accepting new wireless clients on the 2.4 GHz band. Everything else are observations that surround this problem. I made them while I was trying to find out why this is happening. I could not figure out why this is happening, so I posted here hoping that somebody came across the same issue and maybe figured out what to do. So far, I found one similar post that I mentioned above.

    The stock ASUS firmware does not work for me as I mentioned in my OP.

    Under basic wireless I have AP, Wireless Network Mode: Auto, SSID: broadcast, Channel 11, 40 MHz width, WPA2-personal, AES. Nothing out of the ordinary.

    I think that the driver in the ASUS firmware can be backported into tomato builds built specifically for RT-N66U without much harm. If the IO of the black box driver changed then wireless won't work at all, it will be apparent in the first build. If wireless works, it will probably work better in builds specific to RT-N66U.
  13. alpovs

    alpovs Networkin' Nut Member

    I know about that. I flashed the newer version later.
    Shibby started to use the newer version quite a bit earlier than Toastman. I tried Shibby's build even before Toastman's.
  14. alpovs

    alpovs Networkin' Nut Member

    RMerlin, do you know if these info are used by tomato or by your firmware in decisions whether to accept new wireless clients? The way it looks in Device List is not that important. The fact that the router stops accepting new clients is important.
  15. koitsu

    koitsu Network Guru Member

    Hmm, that isn't how I read this:

    To me, I read this as: "I didn't see this wireless-driver-stops-working problem on my WRT54G or my RT-N16, but on my RT-N66U I didn't bother to let the Asus firmware run long enough to determine if it too has the problem I'm seeing." Am I reading that correctly?

    If so, how can you tell me that the firmware "doesn't work" for you? Or by "doesn't work" do you mean "does not meet my needs?" If the latter: your needs seem to be that you need a reliable/working wireless driver that doesn't cause you pains like this. Surely I'm misunderstanding something here.
  16. alpovs

    alpovs Networkin' Nut Member

    A general comment regarding interference etc. I went through many routers running tomato at the same location. Only my RT-N66U has had this issue from day one. It must be something specific to RT-N66U.
  17. koitsu

    koitsu Network Guru Member

    The wireless driver is what's responsible for "accepting" new clients, as well as any kind of network I/O that goes on between the client and the AP. Tomato/TomatoUSB just tells the wireless driver what settings to use -- every other thing is handled by the wireless driver, independent of Tomato/TomatoUSB. Yes, even things like MAC limitation/etc. are handled by the wireless driver.
  18. alpovs

    alpovs Networkin' Nut Member

    Sorry if I was not clear. I meant the ASUS firmware does not meet my needs. I only glanced at it and installed tomato. I did not run it long enough to see if the router would stop accepting new clients.
  19. koitsu

    koitsu Network Guru Member

    Did you have the same problem/issue with the Asus firmware?

    Your answer so far: I did not run the ASUS firmware on my RT-N66U long enough to observe this problem.


    Please switch back to the Asus firmware on your RT-N66U and really give it a beating (I'm talking about weeks here, not hours, unless you can reliably 100% of the time reproduce the issue in a very short period of time. I do mean 100% of the time, not say 80% of the time). This is a proper/necessary troubleshooting requirement.

    If the problem happens with the Asus firmware: the issue is not related to Tomato/TomatoUSB and is related to the things Monk E. Boy and myself have eluded to, or possibly you have a broken router (there have been a few people who have gotten bunk/broken RT-N16 and RT-N66U units, where an RMA with Asus (or returning them to their place-of-purchase) got them a replacement that worked fine).

    If the problem does not happen with the Asus firmware (e.g. only with TomatoUSB): then your choices are:

    a) tolerate the problem (because we cannot do anything about the binary blob driver in TomatoUSB)
    b) go back to the Asus firmware (since it works for you and you seem to need this reliability in wireless)
    c) Get the TomatoUSB code (specifically the branch of firmware you like) and try to "work in" Asus' current wireless driver for their RT-N66U. You'll need to build your own firmware to do that.

    It's that simple. Really. :)

    Reminder: we have many people here on the forum using RT-N66U routers with TomatoUSB with reliability.
  20. alpovs

    alpovs Networkin' Nut Member

    Oh well, I guess we will never know what is going on. The lack of similar reports gives me a feeling that I should try to exchange the router...
  21. koitsu

    koitsu Network Guru Member

    Actually there are tons of reports of this "kind" of issue when it comes to wireless drivers and wireless chips. You'll even find these kind of problem reports on official support forums for vendors' products (i.e. no third-party firmwares).

    Monk E. Boy and I both explained how/why this can sometimes happen -- sometimes it is bad hardware (bad chip, antennas are broken to the point where the driver locks up/wedges), sometimes it's excess interference/noise causing the driver to wedge, and sometimes it's just the nature of wireless (wireless = crap).

    The 2nd item I listed off in my paragraph above is incredibly common -- look here for a problem report I had (I'm Jeremy Chadwick) with an Atheros-based wireless NIC on FreeBSD where the driver would just lock up (this was when the card was acting as an AP, not a client!). Adrian Chadd, a fellow who works for Atheros writing wireless drivers, responded and provided very low-level technical insights to the issue. I ended up mailing him the NIC (to Australia), on my own dime, just to see if he could figure out the problem. That was over 2 years ago. So far: no solution, no improvement.

    Are you ready for the kicker? Wireless chips on Linux, using Linux drivers, have the same problem. Hmm, two OSes with completely different drivers, across multiple wireless hardware chipsets, having the same issue... hmm... imagine that.

    Starting to get the picture here?

    The only thing reliable/stable is Ethernet, I'm sorry to say. Wireless is a "I SURE HOPE THIS WORKS!!!" protocol, and it isn't getting any better. 802.11n, for example, makes the situation significantly worse.
  22. Monk E. Boy

    Monk E. Boy Network Guru Member

    For what it's worth, the only way to determine what wireless frequencies your power meter is using is to get a device similar to WiSpy, which doesn't just see 2.4/5Ghz WiFi signals, it sees everything operating in the 2.4/5Ghz range. Because the power meter (or baby monitors, or security cameras, or cordless phones, etc.) don't use WiFi to communicate, they aren't picked up by any kind of WiFi scanner (like inSSIDer). Sadly, they're not cheap.

    Wireless is extremely flaky to the point that I'm surprised it works at all. Individuals who don't understand the technology involved and randomly change settings in the hope that they'll improve their situation, or outright replacing hardware, are half the trouble. Wireless signals travel a long, long, long ways, especially 2.4Ghz. My home is on a 1.5 acre piece of land and I see wireless interference from neighbors, so they're really quite distant.
  23. alpovs

    alpovs Networkin' Nut Member

  24. koitsu

    koitsu Network Guru Member

    That thread has the same analysis/statements in it that are in this thread. However, as I said above, do not conflate the two threads -- what that user may be experiencing might not be what you are. There is not enough hard technical data (result of a wifi spectrum analysis) in either thread to conflate the two threads. For all you know, that guy has bunk/broken antennas, and you have excess 2.4GHz interference.

    Sorry for sounding like a broken record, but I learned years ago -- from a FreeBSD kernel developer -- that every problem report/incident needs to be handled individually / considered unique.
  25. alpovs

    alpovs Networkin' Nut Member

    Every problem report is unique. Period.

    I have no reason to believe that I have excess interference. The interference level is always "acceptable". The houses are far apart, most people around are old and not tech-savvy. In my OP I described the other signals I see in the wireless survey. And my router is in the basement (ceiling of the basement). I use its signal "vertically" so to speak. Yes, there are microwave ovens; probably some neighbors still have 2.4 GHz phones; maybe the electricity meter operates on the 2.4 GHz band. But my previous WRT54GS, WRTSL54GS, Buffalo WHR-HP-G54 and RT-N16 had zero problems with tomato at the same location! I expected the RT-N66U to be better, not worse. The meter had been changed to wireless before I got the RT-N66U.
  26. koitsu

    koitsu Network Guru Member

    Please set Channel Width to 20MHz, not 40MHz, and see if the issue goes away.

    If not: stay on 20MHz, and then switch to "B/G Mixed" (rather than "Auto") for Wireless Network Mode.

    (Reference to where OP said he was using 40MHz)
    (Reference #1 (see item 5) to how 40MHz causes major problems; don't let this guy's hogwash sway you. Also, Toastman and someone else have a really good highly technical link to problems 40MHz causes, but I cannot find that link right now, I'm sorry to say -- it's excellent)

    Footnote: also, about signal levels for clients that suddenly "crap out" (but then might recover) -- this can be caused by power-saving operations in the wireless chip on the client side (and even more prominent on mobile/handheld devices, where every lick of power is saved at all costs). I saw this with lots and lots of laptops at my past job (Microsoft), and see it on my own home network (wireless printer). Always good to keep an eye on signal levels, but that's something to keep in mind.
  27. Toastman

    Toastman Super Moderator Staff Member Member

  28. koitsu

    koitsu Network Guru Member

    As always, thanks! It's this one:

    It's worth reading -- the entire thing, as in get some coffee/tea and toast and read it in full. It provides quite a nice insight to just how utterly broken the wireless protocol is, and the lack of proper/decent testing on behalf of the committee (Wi-Fi Alliance) responsible for the standards. I particularly like their response to Q4 at the bottom (sigh, this probably has to do with FCC regulations too -- the classic "this device must accept all forms of interference" sticker on everything; I'm surprised my cordless electric toothbrush doesn't have one).

    Bottom line is: yeah, we get it, 40MHz = more bandwidth = more speed. The trade off is destroying networks around you, being significantly more susceptible to other people destroying yours (since now you're using a lot more of the spectrum), and running into absolutely wonderful implementation/compatibility/design glitches.
  29. w11x22

    w11x22 Networkin' Nut Member

    Hey Alpovs
    This was my post and there is no solution to this, at least not yet, at the moment but in my case I only had problems with Shibby builds. I tried different builds from RAF to Toastman to Teaman and all performed well on my same Router (N66U)....I Googled and found nothing. IMO there are tons of people using Tomato and surprisingly there are no complaints so I think it is just an isolated problem related to couple of users. Why? I cant comment on that. Could be a couple of bad units. IDK....But a lot of people reporting that their N66U perform quite well with Tomato.....And if a lot people had this problem then I am sure that we might have seen more and more reports on this or other forums... My 2 Cents

    But I salute to all who built Tomato (including Shibby, Taostman, RAF, Teaman, original creator of Tomato and all others who contributed to this project) and continue to support and help others without expecting anything in return....Hats off
  30. alpovs

    alpovs Networkin' Nut Member

    I came to a similar conclusion. It may be a hardware problem, bad unit. In my case I had the same problem with all tomato builds I tried. I am going to RMA the unit and will report here later how the new unit behaves. ASUS immediately issued an RMA essentially after one sentence "wireless stops accepting new clients after some time".
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice