"IP Traffic Monitoring" on WRT54GL

Discussion in 'Tomato Firmware' started by EpsilonX, Jan 3, 2012.

  1. EpsilonX

    EpsilonX Network Guru Member

    This thread serves as follow-up to my posts in...
    As it is OOT from that thread... :rolleyes:

    -1 wired PC, 1 PS3 WiFi, 1 HTC Desire phone
    -WPA2+AES and Wireless MAC Filter enabled
    -WRT54GL using Toastman 7628.1-Mini
    -No torrent, low amount of traffic, RAM around 50%

    What I did...
    -Set up router as always, enables "IP Traffic Monitoring" in "Administration"
    -Router always rebooted around ~20+ hours of uptime
    -Disabled "IP Traffic Monitoring" in "Administration", no reboot after 4+ days

    Then I update to Toastman 7630-Mini...
    -Set up router as always, enables "IP Traffic Monitoring" in "Administration"
    -Router always rebooted around ~20+ hours of uptime
    -Disabled "IP Traffic Monitoring" in "Administration", no reboot after 4+ days

    Teaman also experienced some random reboots on his own, but I think my setup is much more "simple" than his (more Free Memory?)... ;)
    As things stand, disabling "IP Traffic Monitoring" solved the reboots problem...
    Simple solution, but I'm starting to think I'm the only one with the problem since I haven't seen any report about this... :(
    I think a troublesome wireless can be chalked off (hopefully), since the reboots always happened after around 20+ hours, at day time or night time...
    Anyone with the same "problem" (reboots after ~20+ hours)..?

    Where did Tomato store the "IP Traffic" data before it is "saved" as history..?
    Am I right to assume the difference between enabling-disabling "IP Traffic Monitoring" is in the "data"..?
  2. teaman

    teaman LI Guru Member

    Let's begin with some clarifications about what is supposed to happen under the hood, shall we? ;)

    Basically, the 'Enable' checkbox on top of page Admin: IP Traffic Monitoring (admin-iptraffic.asp) only means the 'cstats' process won't be started (and a few things on the web UI won't work: Last 24h, Daily and Monthly history pages. All other pages should give a warning saying that IP Traffic monitoring is disabled, but should work just fine. But why?

    This is only possible since the ipt_account module is always loaded and configured to track any network packets being forwarded thru the router from/to LAN whenever firewall is restarted ;)

    Therefore, is something funky seems to be going on on your router... we can probably rule out the IP Traffic 'monitoring' part of that (the ipt_account module does all the heavy lifting) and we might wanna look at the record/history keeping part (that would be the 'cstats' daemon). Now that we're here, a few words about cstats:

    It started as a clone of rstats, but instead of keeping track/history of network traffic from/to the WAN interface, it has been patched/hacked/extended to handle the same kind of information about multiple IP addresses on LAN. Just like rstats, cstats can keep details of the last 24h (2 min intervals), the last 30 days (daily totals) and up to 24 months (monthly totals) for any number of IP addresses it has 'learned' about (see the "Enable Auto-Discovery" option on the Admin: IP Traffic monitoring page).

    Now, there's one key element here: cstats 'measures' things every 2 mins (so here's one frequent thing that could possibly cause problems). Also, it saves its data as per settings on the same Admin page (admin-iptraffic.asp). One of my routers is saving cstats data every 6h on the SDHC card, there's another router also saving data on this SDHC card (also every 6h, via on a CIFS share). This second router is configured as a wireless client, so... whenever cstats needs to save its data... it's best to have that CIFS share available (otherwise things might misbehave, hang for a while, etc... but the router shouldn't reboot /just for that/).

    With all that in mind, I wanted to ask: how many IP addresses your cstats database 'knows' about right now? How many are you currently monitoring? How frequently are you saving data? Do you have backups enabled? Any services being restarted? What else are you running on your router? Where are you saving your cstats/rstats data? Does such thing happen when you configure rstats to save its data at the same location and at the same frequency? Is it on another machine? Does it go to 'sleep' at any moment in time? How does it wake up? How long does the machine (or disks) take to wake/spin up? Please provide as much information as possible, otherwise it might not be possible for any of us to help you out...

    Now - a few more technical details about cstats internals (considering 'auto-discovery' mode /enabled/ to simplify things):
    * at startup, it tries to load its last datafile (see nvram var cstats_path)
    * if such file cannot be found, it tries to load previous versions/backups (on the same path) in reverse order (oldest to newest), then tries to load the 'last' datafile again (i.e. maybe a network share wasn't online when it first tried)
    * if none can be loaded/found, retry every 15 mins to find any valid datafile or backup
    * when a data/backup file is found, it's uncompressed (gunzip) and any valid 'records' are read
    * each of those records is 7920 bytes, all valid records found are stored/inserted in a AVL Binary Tree (i.e. as it was 'learning' about the traffic history of any IP addresses previously recorded)
    * assuming all went well (typical startup), cstats enters its main loop and updates its data/stats about all addresses it's supposed to watch and/or any new addresses it might 'discover' (with new traffic)
    * this is done by looking at the contents of pseudo-files under '/proc/net/ipt_account/lan*' (and those are created by firewall rules, BTW)

    Data goes 'out' from cstats in 3 ways:
    * Last 24h page: cstats walks our binary tree and dumps stats/details about traffic in 2 min intervals (last 24h) in a json file, which is served via httpd (see static void save_speedjs in cstats.c)
    * Daily/Monthly pages: same as above, but daily/monthly totals per IP address (see static void save_histjs in cstats.c)
    * 'regular' saving datafile is a 3 stage process (and each of these could also be a possible source of problems):

    1) dump all records to an uncompressed file on a tmp filesystem, then compress it with gzip (7920 bytes per 'known' IP so data/stats for 130 IPs ~= 1MB of uncompressed data)
    2) if backups are enabled, rotate them and drop the last/oldest
    3) copy/move the compressed datafile from step 1 to configured path


    Anyways - there's a few recent commits that might help things out in general:

    IPTraffic: K24 ipt_account module updated to v0.1.20

    IPTraffic: get rid of nested loop for realtime conntrack counting

  3. EpsilonX

    EpsilonX Network Guru Member

    Yup, noticed that...
    And the router always rebooted around 20-26 hours mark, and yet Save Frequency is set to 2 Days...
    And yup, thats why I'm curious about the history/data location...

    3-4 IP (100-103, I exclude
    Save Frequency 2 Days (Default), No backups, basically default setting, Save to RAM...
    And yes, rstats (BW Monitor) is at default settings, Save Frequency 2 Days (Default), No backups, Save to RAM...
    Tried cstats save to 6 hours (with rstats at default), also rebooted...
    I'm using the Mini version of Toastman's firmware, so pretty much minimum...
    DDNS, Static DHCP, Wireless Filter, NAT-PMP (For 1 mostly idle torrent client), QOS (11 rules), thats all...

    Is it on another machine? Does it go to 'sleep' at any moment in time? How does it wake up? How long does the machine (or disks) take to wake/spin up?
    What are you referring to if I my ask..? :)

    And wicked post... :D
    I'm more concerned at no replies from other users though...
    Either I did something wrong, my router went nuts, or no more old router around... :D
  4. teaman

    teaman LI Guru Member

    When cstats is configured to save to 'RAM (Temporary)', 'saving' means just writing/dumping data/stats into /var/lib/misc/cstats-history and compressing that file with gzip. Therefore, this part of the cstats code seems somewhat unrelated to the matter at hand. Since you mentioned you're /saving/ your data somewhere, I just assumed it was on some kind of persistent storage - to those other questions I asked about sleep/spinup times would be only relevant if perhaps you've using some kind of network share mounted on your router for saving rstats/cstats datafiles (i.e. CIFS volume shared somewhere on yout LAN, etc...)

    Since you're saving to RAM, you have cstats_path set to '' (empty string), right? What's the contents of all other cstats_* vars on your nvram?
    nvram show | grep cstats
  5. EpsilonX

    EpsilonX Network Guru Member

    Oh ok... :D

  6. teaman

    teaman LI Guru Member

    Unfortunately, I can't identify anything obviously wrong with your settings, except perhaps for one or two small things that just struck me as possibly-related (but still... more like remotely possibly-related): can you please try running cstats with cstats_exclude='' (empty) just to see if that might be causing problems in your particular case?

    Also... since you have a WRT54GL just like me, you might want to try running one of my builds, just to check if the same problem also happens with those ;)

    Let us know how it goes.
  7. EpsilonX

    EpsilonX Network Guru Member

    That's why its so weird... :D
    At first I didn't exclude anything...
    I exclude that because that stat is more or less viewable from Bandwidth Monitor, and to try to minimize the collected stats...

    Will do...
    As soon as I'm home, been away for a few days...
  8. EpsilonX

    EpsilonX Network Guru Member

    Flashed 1.28.0013 Teaman-VLAN-SNMP ND Std...
    Use the same setting as previous firmware...
    Enabled cstats without exclusion...
    Rebooted just now at +- ~12 hours... :confused:
    Gonna give it another 24 hours to make sure it's not just a one time only thing...

    EDIT :
    Another reboot just now after another +- ~12 hours... :confused:
  9. teaman

    teaman LI Guru Member

    Well, then... I guess we could consider another possibility: you router just might need some kind of maintainance and/or replacement...
    But why would I even suggest such idea? Here's what I got on my WRT54GL:
    root@amelia:/tmp/home/root# uptime
    19:25:52 up 5 days, 14:41, load average: 0.06, 0.01, 0.00
    It's been running fine for a few days, with no problems whatsoever.

    Best of luck!
  10. EpsilonX

    EpsilonX Network Guru Member

    An update...
    After 2 reboots in the first day...
    It runs fine for 4 days... :D
    Still can't explain the 12 hours initial reboots though...
    Right now I'm testing Toastman 7632 to see if your recent commits change anything for me... :D
  11. EpsilonX

    EpsilonX Network Guru Member

    OK, a very good news...

    Tomato v1.28.7632 -Toastman-IPT-ND ND Mini
    root@Tomato:/tmp/home/root# uptime
    18:13:50 up 3 days,  4:14, load average: 0.01, 0.01, 0.00
    It seems that the changes in 7631/7632 fixed it...
    Will continue to monitor things...
    Great work Teaman and Toastman, and thank you... :D
  12. teaman

    teaman LI Guru Member

    Thanks, EpsilonX! Just to follow-up, here's the current uptime of that WRT54GL:
    root@amelia:/tmp/home/root# uptime
    02:24:15 up 9 days, 21:39, load average: 0.08, 0.01, 0.00
    To be fair... I've been lucky - no power outages for the last few days - there's no UPS for that unit :)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice