1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

tomato - hourly data?

Discussion in 'Tomato Firmware' started by belbo, Dec 16, 2009.

  1. belbo

    belbo Addicted to LI Member

    Does anybody know if its is possible to access hourly data on tomato (or if it is possible to read the rstat file and extract the data) ? If not, is this intended as a feature at any point ?

    In Australia we have plans with peak and off peak usage allowances (my off peak is midnight to noon daily).

    I have had some issues with my isp about allocation of downloads around the changeover times being allocated to the incorrect period (on one occasion they admitted it and reallocated the usage). I would really like to have a record which I can just cut&paste/ download and send to them in case of another dispute.

    If anyone has any tips in this regard I'd really appreciate it.
     
  2. Planiwa

    Planiwa LI Guru Member

    I have been meaning to put together exactly this, so that others can use it. (I already have various versions of it for my own use). Even the minutest encouragement might make a difference at this point. I might do it in time to start with the new year -- as my 40th birthday gift to Unix. :)

    What I have in mind is MB/h, RX and TX, for each interface (e.g. ppp0 or eth1).
    This will require no external resources, and keep a whole year's data on JFFS.

    (There's quite a bit more to it. And it should not damage the memory.)
     
  3. belbo

    belbo Addicted to LI Member

    Well that would really be great. Would it also be possible to store the data on a cifs mount (or only jffs?). I for one would really like to encourage you - as I'm sure would all those users with different peak and off-peak limits - and I suspect it might attract a whole lot more such users.

    I installed tomato this week. I take it you are the (or one of the) developers. If so, thanks and well done for a great product.

    I am curious as to what file format the rstat file is and whether (or with what tools) it is possible to read the data. I can't seem to find anything about this on the web.

    Happy birthday and happy new year. If you decide to make the hourly stats available that would be a great present from you. Let us know when you do so.

    Thanks again

    belbo
     
  4. Planiwa

    Planiwa LI Guru Member

    Very briefly:

    It's being tested now.
    People can keep the data on JFFS or CIFS (but I can't test that), or wherever.

    Looks like this:

    Code:
    # head log/r*/*;echo
    ==> log/rxtx/eth1 <==
    20091216 21<97>246 22<163>266 23<132>272
    20091217 00<129>325 01<154>308 02<180>223 03<174>147
    ==> log/rxtx/ppp0 <==
    20091216 21<240>94 22<259>159 23<265>127
    20091217 00<318>125 01<300>149 02<216>175 03<142>170
    # 
    
    This is a log, not a report.
    Takes 150kB per interface per year.

    < means RX to the IF
    > means TX from the IF

    Note: for ppp0, RX means downstram,
    whereas for eth1, RX means upstream.

    There are two additional logs:

    "day" is a daily sampling of vital statistics.

    Code:
    # cat log/day
    20091216_21:59 br0:3 77 .3 2/27 F:724 C:4392 B:1692 A:4960 I:3076
    20091217_00:59 br0:5 277 .4 2/25 F:644 C:4508 B:1312 A:4680 I:3028
    
    "events" records the events that correspond to the AdminScripts.

    Here it is -- rxtx:

    Code:
    # rxtx [INTERFACE ...] # Logs hourly traffic & daily stats, run at HH:59; Planiwa 20091217
    # In Init Script: cru a RXTX "59 * * * * /jffs/rxtx"
    LOGDIR="/jffs/log/" # or use cifs if you prefer; may have to add .txt to filenames
    LOGDIR="$HOME/log/" # remove after testing
    TMPDIR="/tmp/"
    case $# in 0) INTERFACES="ppp0 eth1";; *) INTERFACES="$*";; esac
    DATE=$(date '+%Y%m%d'); HOUR=$(date '+%H'); MINU=$(date '+%M'); OUTLINE=""
    if [ -r ${LOGDIR}day ];then LASTDATE=$(date -r ${LOGDIR}day '+%Y%m%d')
    else LASTDATE=0; mkdir -p ${LOGDIR}rxtx/ # JFFS: first time ever, RAMFS: Rebooted
    fi
    cd ${LOGDIR}rxtx/
    if [ ! -d ${TMPDIR}rxtx/ ];then mkdir -p ${TMPDIR}rxtx/; fi # Rebooted
    
    for IF in $INTERFACES;do
      set -- $(ifconfig $IF |egrep 'bytes:|packets:') # counters reset at 4.3 GB (mod 2^32)
      case $# in
      0) ;; # IF down
      *) RXP=${2#*:}; TXP=${8#*:}; shift 12
         RX=$2; TX=$6; RX=${RX#*:} TX=${TX#*:} # in bytes
         if [ ! -r ${TMPDIR}rxtx/$IF ];then PRX=0;PTX=0  PRXP=0; PTXP=0
         else set -- $(cat ${TMPDIR}rxtx/$IF); PRX=$1; PTX=$2;  PRXP=$3; PTXP=$4
         fi
         if [ ${#RX} -le 3 ];then RX=0;else RX=${RX%???};fi
         if [ ${#TX} -le 3 ];then TX=0;else TX=${TX%???};fi
         echo "$RX $TX $RXP $TXP" >${TMPDIR}rxtx/$IF # in kB
    
         if [ $RX -lt $PRX ]&&[ $RXP -gt $PRXP ];then RX=$((RX+4294967));fi
         if [ $TX -lt $PTX ]&&[ $TXP -gt $PTXP ];then TX=$((TX+4294967));fi
    
         RX=$(((RX-PRX+500)/1000)) # in MB
         TX=$(((TX-PTX+500)/1000))
         if [ "$DATE" != "$LASTDATE" ];then
           if [ -r $IF ];then echo "">>$IF; fi
           echo -n "$DATE">>$IF
         fi
         echo -n " $HOUR<$RX>$TX"      >>$IF
         OUTLINE="$OUTLINE $IF<$RX>$TX"
         ;;
      esac
    done
    logger -p info -t RXTX "$OUTLINE"
    if [ "$DATE" != "$LASTDATE" ];then # daily stats here
      set -- $(cat /proc/meminfo)
      MEM="F:${22} C:${31} B:${28} A:${37} I:${40}"
      if [ -r /proc/sys/net/ipv4/ip_conntrack_count ]
      then CON=$(cat /proc/sys/net/ipv4/ip_conntrack_count) # Thanks, Teddy Bear!
      else CON=$(cat /proc/net/ip_conntrack | wc -l)        # This is very costly
      fi
      USE=$(arp|awk '/vlan1/{next};{N[$NF]++};END{for (i in N) print i ":" N[i]}')
      set -- $(cat /proc/loadavg)
      P=$4
      LOADL=${1%.*}; LOADL=${LOADL#0}; LOADR=${1#*.}; LOADR=${LOADR#0};
      LOADR=$(((LOADR+5)/10)); case $LOADR in 10)LOADR=9;;esac; LOAD=$LOADL.$LOADR
      OUTLINE="${DATE}_$HOUR:$MINU $USE $CON $LOAD $P $MEM"
      echo "$OUTLINE" >> ../day
      logger -p info -t Day "$OUTLINE"
    fi
    
    exit
    /jffs/rxtx       -- logs traffic + daily -- called by CRON at HH:59, set up by INIT
    /jffs/log/rxtx/  -- 1 file / IF, 1 line / day, 1 word / hour<RXMB>TXMB
    /tmp/rxtx/       -- 1 file / IF, RXkB TXkB RXpkt TXpkt -- prev. hour
    /jffs/log/day    -- 1 line / day: users. conn's, load, processes, memory
    /jffs/log/events -- one line per boot, one word per WAN-Up or FW-Up
      Add the following to the Admin Scripts:
      mkdir -p /jffs/log; echo "" >>/jffs/log/events; echo -n Init >>/jffs/log/events # Init
      echo -n " Shutdown-$(date '+%Y-%m-%d_%H:%M')" >> /jffs/log/events # Shutdown
      echo -n " FW-Up-$(date '+%Y-%m-%d_%H:%M')" >> /jffs/log/events # FW Up
      echo -n " WAN-Up-$(date '+%Y-%m-%d_%H:%M')" >> /jffs/log/events # WAN Up
    # Eventualities:
      # (Re-)Start -- first time or first post-reboot run (same day?)
      # New Day -- check LOGDIR change time -- add EOL to log files; do daily stats
      # 2^32 Counter Reset -- need add 4.3GB if fewer bytes but more packets
      # Ordinary
      # Syslog Output
    PPP outage, crash reboot, crash stuck; 2^31 sh arith overflow; 2^32 ifconfig overflow
    Frequencies: DSL:1-6w PPP:6-36h ifconfig-reset:B>2h,P>40d boot/crash:12-72h (also:re-FW)
    Installation?

    Put this someplace, e.g. /jffs/rxtx
    make it executable, e.g. chmod +x /jffs/rxtx
    schedule it: cru a RXTX "59 * * * * /jffs/rxtx"
    Put that into Init script (too)

    If you feel like it, put a line into each Admin script, as suggested at the end:

    Add the following to the Admin Scripts:
    mkdir -p /jffs/log; echo "" >>/jffs/log/events; echo -n Init >>/jffs/log/events # Init
    echo -n " Shutdown-$(date '+%Y-%m-%d_%H:%M')" >> /jffs/log/events # Shutdown
    echo -n " FW-Up-$(date '+%Y-%m-%d_%H:%M')" >> /jffs/log/events # FW Up
    echo -n " WAN-Up-$(date '+%Y-%m-%d_%H:%M')" >> /jffs/log/events # WAN Up

    Probably won't hurt terribly much to read the comments. :)
     
  5. belbo

    belbo Addicted to LI Member

    The logs look fantastic. I would be interested to try it out but my scripting knowledge is pretty much zero and I haven't even gotten around to accessing the command line on the router (presumably telnet?). So I will eagerly await you completing testing and productionising it - hopefully with install instructions for dummies like me. (Presumably it is a script (python ?) that just has to be put anywhere on the router, given 775 permissions and called by cron?). Looking forward to updates. Good luck with the testing.
     
  6. Planiwa

    Planiwa LI Guru Member

    So, I corrected a small flaw and added some "installation" instruction in the original post (which just repeats what's inside the program already). Would appreciate feedback.

    The only system I have access to rebooted 9h ago, and in about 12h should hit the first 32 bit reset. Let's see how that is handled. :)

    Code:
    # vit -M -H
    VIT: br0:3  59 F:744 C:4612 B:1324 A:4660 I:2900 .1 2/23 +23s +16s 9h ppp0<1759>1019 eth1<1807>1051
    
     
  7. Planiwa

    Planiwa LI Guru Member

    I'm not a "developer". I'm not partial to this division of people into producers and consumers, actors and spectators, etc. :(

    Tomato's Unix-style environment enables users to be their own "developers", using tools to enhance their experience, limited only by their own interests and imagination. There is an opportunity for collaboration and learning in all this.

    So, I appreciate the feedback. :)

    (I only have one Tomato network left, it's remote, only about 6 users, and for the last 5 hours, there's been no LAN activity at all. (Is everyone away or asleep, or is the WL broken?) Makes it a little hard to test things. :-|)
     
  8. Planiwa

    Planiwa LI Guru Member

    We now have a good deal experience with this, and some refinements as a result.
    (If there is interest I'll post an updated version.)

    Syslog now contains a nice hourly summary:

    Code:
    Dec 18 19:59:03 ROUTER user.info RXTX: (IF MB B/P CMB)  ppp0 <388>7 {1469}48 [699]1743  eth1 <9>394 {62}1491 [1797]788
    Dec 18 20:59:03 ROUTER user.info RXTX: (IF MB B/P CMB)  ppp0 <220>4 {1465}52 [918]1748  eth1 <5>223 {63}1486 [1802]1011
    Dec 18 21:59:03 ROUTER user.info RXTX: (IF MB B/P CMB)  ppp0 <427>9 {1453}54 [1346]1756  eth1 <11>434 {68}1480 [1813]1445
    Dec 18 22:59:03 ROUTER user.info RXTX: (IF MB B/P CMB)  ppp0 <387>8 {1453}56 [1732]1764  eth1 <10>392 {69}1480 [1823]1837
    Dec 19 00:00:03 ROUTER user.info RXTX: (IF MB B/P CMB)  ppp0 <738>14 {1463}53 [2470]1779  eth1 <18>749 {66}1489 [1841]2586
    Another useful side effect is a recording of byte and packet count, every hour, until reboot.

    Most of these things, such as packet size, are only of interest to technical users.

    Here are the logs, so far:

    Code:
    # head day rxtx/*;echo
    ==> day <==
    20091216_21:59 br0:3 77 .3 2/27 F:724 C:4392 B:1692 A:4960 I:3076
    20091217_00:59 br0:5 277 .4 2/25 F:644 C:4508 B:1312 A:4680 I:3028
    20091218_00:59 br0:3 17 .1 2/25 F:688 C:5164 B:1036 A:5168 I:2736
    20091219_00:00 br0:2 33 .3 2/25 F:1128 C:5256 B:900 A:5140 I:2724
    
    ==> rxtx/eth1 <==
    20091216 21<97>246 22<163>266 23<132>272
    20091217 00<129>325 01<154>308 02<180>223 03<174>147 04<172>172 05<213>117 06<248>72 07<25>108 08<0>0 09<0>0 10<0>0 11<0>0 12<0>0 13<0>0 14<0>0 15<0>0 16<0>2 17<0>0 18<0>1 19<2>9 20<3>20 21<9>93 22<11>109 23<21>783
    20091218 00<9>213 01<12>299 02<7>75 03<1>4 04<0>0 05<0>0 06<0>0 07<0>2 08<0>0 09<0>0 10<2>20 11<8>391 12<0>0 13<0>0 14<0>1 15<0>0 16<4>16 18<1>3 18<9>390 19<9>394 20<5>223 21<11>434 22<10>392 23<18>749
    ==> rxtx/ppp0 <==
    20091216 21<240>94 22<259>159 23<265>127
    20091217 00<318>125 01<300>149 02<216>175 03<142>170 04<167>169 05<113>209 06<69>245 07<107>24 08<0>0 09<0>0 10<0>1 11<0>1 12<0>1 13<0>1 14<0>2 15<0>1 16<3>1 17<0>0 18<2>0 19<13>2 20<19>2 21<99>8 22<106>9 23<777>17
    20091218 00<208>8 01<293>11 02<72>6 03<4>1 04<0>0 05<0>0 06<0>0 07<2>0 08<0>0 09<0>0 10<20>2 11<385>6 12<0>0 13<0>0 14<1>0 15<0>0 16<16>4 18<3>1 18<385>7 19<388>7 20<220>4 21<427>9 22<387>8 23<738>14
    
     

Share This Page