Troubleshooting encrypted Wireless Client on MIPS routers

Discussion in 'Tomato Firmware' started by cbunt1, Mar 1, 2019.

  1. cbunt1

    cbunt1 Serious Server Member

    I've been doing some troubleshooting of the Wireless Client Modes (Wireless Client and Wireless Ethernet Bridge) in the MIPS version(s) of FreshTomato. I've posted on a couple of other threads, and to keep from taking over another thread, it's time to start my own.

    I'm working with a small group of Asus RT-AC66U routers, all of which have been running one version or another of Tomato for several years. I'd like to figure out what's going on with the encrypted modes for Wireless Client (and by extension Wireless Ethernet Bridge) modes.

    I'm currently working with the latest release of Fresh Tomato (2009.1) since the glitch seems to behave the same way at least as far back as Shibby v.131. I think we can operate under the assumption that the code base is stable -- at least for the MIPS routers.

    To reproduce the issue: Start with a fresh firmware flash, with a through NVRAM clearing, followed by a soft-reset (to bring the 5G radio back to life). Then configure the WAN in Wireless Client Mode, under DHCP to your upstream access point. Start with an unencrypted link (as a proof of basic functionality). If the unencrypted link works (e.g. you get an address and can pass traffic), move to a WEP encrypted mode. Again, you should be able to get an IP and pass traffic.

    The problem shows up (for me) when I move to a WPA/WPA2 encrypted link. As best I can tell, the physical link is not occurring at all. This makes me think that there's either a problem with the wireless driver (a blob of precompiled code from Broadcom) or a problem with the way we're passing the crypto information to the physical layer (maybe something we can fix on this end).

    What I need at this point is feedback about which MIPS routers will and will not make the various connections.

    I'll be using this thread to post my findings as I go through the various versions and equipment that will and won't work, hoping to find the differences.

    So far I can positively say that the RT-AC66U will link up in unencrypted client and WEP client, but will not link up with WPA/WPA2 encryption enabled. It *will* receive clients in all three modes though.

    More to come. Your input is more than welcome!
     
  2. pedro311

    pedro311 Networkin' Nut Member

    WRT54G (MIPS) is working with both, WEB and WC mode on WPA2 (connected to RT-N18U) without problems.

    Bugs: on WEB mode, problem with dnslookup and no time (ntpd).
    Solution of the first one:

    Code:
    echo "IP of RT-N18U" > /etc/resolv.conf
    No solution (yet) for the second.

    //EDIT: I forgot to mention FW version on WRT54G: freshtomato-K26_RT-MIPSR1-2019.1-Mini
     
    Last edited: Mar 1, 2019
    cbunt1 likes this.
  3. Techie007

    Techie007 Networkin' Nut Member

    I am connecting an Asus RT-N12/D1 wirelessly to a Tenda AC15 using WPA/WPA2 encryption, and it's working fine as long as I use DHCP for the WAN mode. It initially didn't work because the Asus router kept the WPA2 Personal mode when I enabled Wireless Client mode. It would show that it was connecting and disconnecting every few seconds on the Status page (judging by the signal strength reading). When I was clicking through the settings on the Basic Network page, I noticed that the selected encryption mode wasn't one of the available options, and when I changed it to WPA/WPA2 Personal it connected right away.
     
    cbunt1 likes this.
  4. Sean B.

    Sean B. Network Guru Member

    As ntpc uses FQDN's for connecting to the time servers, wouldn't the time update failure be caused by ntpc being run during boot before the DNS server is echo'd into /etc/resolv.conf? If so, I would think this would fix for the time being:

    Code:
    echo "DNS SERVER IP" > /etc/resolv.conf && service ntpc restart
     
    Last edited: Mar 2, 2019
  5. cbunt1

    cbunt1 Serious Server Member

    Looks like my next compiler run will be the WRT54G and/or the N12/D1 versions...as soon as I finish going through the output of my last AC-RT66U build...it's interesting.

    I wonder what it would take to go ahead and upgrade the version of OpenSSL since the support on the current version dies off at the end of this year anyway.

    But...one thing at a time!
     
  6. pedro311

    pedro311 Networkin' Nut Member

    I will check it (remember that actually we have ntpd, not ntpc on FreshTomato).
     
  7. Sean B.

    Sean B. Network Guru Member

    Yes, I know. As ntpd is both client and server, I was referring to the (c) client operation as Tomato does with its function call ( ntpc_start , ntpc_stop ). Sorry.

     
  8. digixmax

    digixmax LI Guru Member

    I am using the same solution approach to fix the "no current time" issue I have with Wireless Ethernet Bridge (https://www.linksysinfo.org/index.p...net-bridge-no-current-time.74428/#post-302295).
     
    Last edited: Mar 3, 2019
    Sean B. likes this.
  9. cbunt1

    cbunt1 Serious Server Member

    Latest update: I have completed my development environment and been able to successfully compile images, and even confirm minor "easter-egg" changes are making it through the compile process. A potentially pointless step, but it would't be the first time I dabbled in projects that were working with multiple versions of the same library or something, and my changes/searches were all for naught.

    Anyway. I've been able to compile a working version of the RT-AC66U AIO firmware (based solely on the repo) and an N-66 version. Unfortunately I don't have anything to test the N-66 FW on.

    Next step will be to compile a N12/D1 version for a stare-and-compare of the compiler output. That was my original plan above, but I got distracted and built the N66 version. I'm not sure whether we have any confirmed reports of the N66 functioning correctly with WEB or WCM, but it gave me a nice 120,000+ lines of output to comb through! Even using some sophisticated diff tools, it's a LOT of output! Unfortunately, no "ah-ha" moments.

    So, today's job will be to build a MIPSR2 AIO image that hopefully will shed some light.

    I'll also be trying to dig into some of the Merlin code over the next few days, since the WEB mode is verified to work on the RT-66U...as I see it, the negotiation of WEB and WCM should be identical.
     
  10. Bird333

    Bird333 Network Guru Member

    I know my N66U works without encryption but I don't know if it works with WEP. It definitely doesn't work with WPA/WPA2. It works fine in WEB mode with WPA2.

    Also, WC wouldn't work on an old WRTSL54GS I had but DD-WRT did. I don't remember the details but I remember I had to install DD-WRT to get it to work in the hotel I was in.
     
  11. pedro311

    pedro311 Networkin' Nut Member

    See FW version.
     
    cbunt1 likes this.
  12. cbunt1

    cbunt1 Serious Server Member

    I find this quite interesting...WEB and WCM should be different implementations of the same protocols -- the differences being in the router's behavior rather than the radio/network's behavior. Or so I would think. But every time I start to think I understand what's going on under the hood, I dig a little deeper and discover just how little I know!!

    Not sure how far back or how long ago you're talking about, but I put a decent number of WRT54GS and WRT54GL routers in the field in client mode with DD-WRT in 2006 and early 2007. I remember it functioning quite well, but often requiring some fiddling to get it started...this was back in the days when not that many home-based units could even support WPA/WPA2, and we still thought it was funny to walk in and crack a customer's WEP as configured from the ISP.
     
  13. Bird333

    Bird333 Network Guru Member

    Oh no this was about a year ago. I still have the WRTSL54GS because it is slim which makes a good travel router.
     
  14. cbunt1

    cbunt1 Serious Server Member

    Indeed. I ran so many of the 54GS and GL routers back in the day it was ludicrous. I miss my little workhorses.

    I can confirm that DD-WRT v. 39031 (released earlier this week) does indeed work on my RT-AC6U in encrypted (WPA2-PSK/AEP) client mode, attaching to my RT-AC66U running Shibby v131. So there's that.

    I was going to pull down the source of DD-WRT to see if I can tell what's different about the way it interfaces with the physical layer and/or the drivers/kernels, but man that's a BIG code base, and makes the Tomato code base look simple and straightforward! We're talking somewhere around 10GB of source!!!

    As of right now, I'm still combing through the build outputs of the RT-AC66U, N12, and WRT54G versions looking for significant differences in the builds. So far, nothing really useful. What I originally thought might be crypto errors turned out to fails in "patch" (common if the patch is already applied), and are common across all the platforms - so are likely NOT the issue.

    Still hoping for some subtlety from the web interface, or as the PSK is injected from NVRAM to the radio/driver. Perhaps a glaring descriptor in the source code comments saying "HERE'S WHERE IT FAILS" ???

    Hey, a man can hope!! :D

    edited to correct a router HW version typo
     
  15. Sean B.

    Sean B. Network Guru Member

    DD-WRT uses a completely different wireless driver. One that was coded by, or primarily by, BrainSlayer ( I think, but don't quote me on that part ). That is why the overall wireless performance ( range, stability, speed ) are sub par compared to Tomato running Broadcoms wl driver. That also has its drawbacks, as we have no access to the drivers source, no detailed info on how it functions, and are locked to an ancient Linux kernel.
     
  16. cbunt1

    cbunt1 Serious Server Member

    That actually sounds familiar from my waaback machine. I don't remember how long ago or anything, nor how well (or how poorly) the wireless driver runs unit-against-unit, but I seem to remember reading some back-and-forth around developing a new driver.

    I do remember throwing my hands up over the versioning of DD-WRT on my newer hardware, searching for that right combination of stability and functionality and landing on Tomato builds back in about 2011...right after my last trusty WRT54GL released its magic smoke.

    The thing that gets me still is that Merlin's build works just fine (seamlessly in fact) in Wireless Ethernet Bridge mode...which leads me to believe that the core Station mode is in fact functional...which leads me to think that WCM can in fact be done effectively -- we just haven't found the breakdown yet.

    The intertwining of all the WRT/Firmware projects is enough to send me off on tangent after tangent chasing details and history. It's fascinating, but I can't help thinking we've got a ton of legacy code just hanging around doing nothing in all the projects -- and I'm afraid to start dumping files just yet because so much is subtly connected.
     
  17. Sean B.

    Sean B. Network Guru Member

    Wireless Client Mode is a different animal compared to Wireless Ethernet Bridge mode. It's close to comparing a router vs a switch. Ethernet Bridge mode doesn't require much change to the standard operation of the driver, or how it and the firmware interact. However Client mode requires the driver to "switch gears" so to speak and treat the wireless interface as if it were a WAN port. While the actual WAN port, by design, has the advantage of a completely distinct and segregated path to the CPU, this has to be simulated by the driver for the wireless connection. Firmware as well has to adapt to the change, there's plenty of room for things to go wrong. And as I stated before, considering the function is not present in Merlins firmware, my money would be that the issue is with Broadcoms driver and therefor everyone elses hands are tied in regards to diagnosing/fixing it.
     
    Last edited: Mar 5, 2019
  18. cbunt1

    cbunt1 Serious Server Member

    For those following along, (and offering advice, pointers, and experiences) I haven't updated this in a while. I'm still digging into it though, and I think I'm making some progress. Unfortunately, it's nothing tangible yet.

    I think I've figured out which sets of wireless drivers and the various files we're actually compiling at build time, and t this point I'm reading C code section by section to try to figure out what it handles differently and similarly between the various modes. God help me, I'm afraid I'm beginning to understand this gibberish!

    So far I know that the router works in WCM, either unencrypted or WEP. This tells me that the routing protocols and (generically) radio protocols are functional -- even "in reverse" (as WCM is). It's only when I add the WPA/WPA2 layer that the connection stops -- so I still feel like the problem is in the WPA/WPA2 negotiation, hashing, or ordering of the packets.

    Knowing that it works as an AP with WPA/WPA2 tells me that the main base of the WPA encryption is functional. What I don't yet understand is the difference between the handshakes as an AP vs as a STA (client).

    From a completely different angle, I know there were issues with early Linux kernel wireless driver (ieee80211) modules, some expecting the PSK to be passed to the driver in clear-text, and some expecting the PSK to be passed in a hashed form. I got all excited when I found the old versions of the Linux ieee80211 generic driver, but alas, it appears this is just unused code from a legacy version of the firmware.

    But with that old problem hanging around in my mind, I'm still wondering if there's a glitch in the way we're hashing (or not hashing) the wpa_psk as we pass it to the driver, or the order in which we're sending out the handshakes for client (STA) mode vs. AP mode handshaking.

    As of yet I've been unable to capture any of these handshaking packets with a separate machine and wifi card -- or if I have captured them, they're not complete enough to sort them out of the capture files.

    I desperately need some debugging info from the kernel and network stack, but haven't yet had the confidence to build a firmware with the debugging options turned on...that's probably next.

    I'm sure I'm boring the heck out of most, but it's both therapeutic and productive for me to communicate my progress -- it helps me get out of my own headspace, and often gives me new ideas and angles to chase down this dragon.

    Oh, and I've learned more than I ever wanted to know about the internals of encryption and low-level network interfaces. Trust me--it only LOOKS like we just plug in and things light up!! LOL :D
     
  19. amirx96

    amirx96 New Member Member

    Hey cbunt1,

    Thanks for the hard work on this.

    I have a TM-1900AC (Asus RT-AC68U) FW FreshTomato Firmware 2019.1 K26ARM USB AIO-64K. I can confirm the the same problem with WCM. The AC68U will get stuck on renewing, and never actually establishes connection to the host AP.
     
  20. Computz

    Computz New Member Member

    I've got a R7000 and have had to use dd-wrt explicitly for this reason. =( I'm not sure what changed in the code but client mode with encryption (WPA2) worked fine on a few different MIPs routers. Now seeing the the bug affects the MIPS builds as well makes me expect that he problem lies somewhere in the general networking code itself and not the specific chipset code.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice