1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Shibby - Web Usage History

Discussion in 'Tomato Firmware' started by Cyberian75, Aug 10, 2012.

  1. Cyberian75

    Cyberian75 Network Guru Member

    Hey --

    Where does it save its history? It'd be nice if users can change its logging path like Syslog so that it survive reboots..

    Thanks
     
  2. mraneri

    mraneri LI Guru Member

    /proc/webmon_recent_searches
    and
    /proc/webmon_recent_domains

    (edit, fixed a path mistake. Thanks to subsequent posters for catching it.)
     
  3. Cyberian75

    Cyberian75 Network Guru Member

    Thanks for replying.

    I found them under "/proc/" but they are empty. Also, the "proc" directory doesn't exist under "/var/log/".
     
  4. leandroong

    leandroong Addicted to LI Member

    Tomato Firmware 1.28.0000 MIPSR1-099 K26 USB AIO
    mine, under "/proc"
     
  5. mraneri

    mraneri LI Guru Member

    Sorry, I misinterpreted a script I had written and assumed /proc was preceeded by var/log.
    You are correct, they are in /proc.
    However, if you "ls -l" them, they appear to have 0 size. but in fact there is data there. (I have NO idea how this is possible, but it is true!!! If anyone can explain how zero length files actually have data, please do!!! I wrote a script to email the contents of these files to me daily (search my threads if you are interested), and it works reliably. So you should have no problem using them as if they're normal files. But in some way, they're not normal.)

    try:
    tail \proc\webmon*

    and you'll see the stuff is in those files... (tail shows last 10 lines in each file...)
     
  6. Cyberian75

    Cyberian75 Network Guru Member

    Thanks, mraneri!

    I will try to append it to a file on my SD card using cron.
     
  7. Cyberian75

    Cyberian75 Network Guru Member

    There's no separate "webmon" process running. Wonder what controls it.
     
  8. leandroong

    leandroong Addicted to LI Member

    using winscp,
    1.) "webmon_recent_searches", size 0, empty contents upon double click. Looking at web usage, this is indeed empty. Correct in short.

    2.) "webmon_recent_domains", size is 0. Shows ff data:

    http://repo.or.cz/w/tomato.git/shortlog/refs/heads/tomato-RT1344701758 10.0.1.2 tags.bluekai.com
    1344701758 10.0.1.2 www.bkrtx.com
    1344701756 10.0.1.2 dcad.watersoul.com
    1344701755 10.0.1.2 kona5.kontera.com
    1344701754 10.0.1.2 www.luminate.com
    1344701754 10.0.1.2 b.scorecardresearch.com
    1344701752 10.0.1.2 konac.kontera.com
    1344701751 10.0.1.2 counter.yadro.ru
    1344701750 10.0.1.2 www.google-analytics.com
    1344701750 10.0.1.2 static.driverscollection.com
    1344701749 10.0.1.2 kona.kontera.com
    1344701749 10.0.1.2 apis.google.com
    1344701748 10.0.1.2 driverscollection.com
    1344701650 10.0.1.2 tomatousb.org
    1344701649 10.0.1.2 tomatousb.wdfiles.com
    1344701648 10.0.1.2 pixel.quantserve.com
    1344701566 10.0.1.2 dynupdate.no-ip.com
    1344701565 10.0.1.2 crl.microsoft.com
    1344701152 10.0.1.2 mail.yimg.com
    1344701150 10.0.1.2 us.bc.yahoo.com
    1344701150 10.0.1.2 l.yimg.com
    1344701150 10.0.1.2 ucs.query.yahoo.com
    1344701145 10.0.1.2 36ohk6dgmcd1n-c.c.yom.mail.yahoo.net
    1344701145 10.0.1.2 d.yimg.com
    1344701145 10.0.1.2 3ae1mc4rm4e9l-c.c.yom.mail.yahoo.com
    1344701145 10.0.1.2 3ie33cpgj6dhi-c.c.yom.mail.yahoo.com
    1344701144 10.0.1.2 dps.msg.yahoo.com
    1344701142 10.0.1.2 ads.yimg.com
    1344701142 10.0.1.2 3cp9lcoq32dpn-c.c.yom.mail.yahoo.com
    1344701137 10.0.1.2 image-c.c.yom.mail.yahoo.net
    1344701132 10.0.1.2 l2.yimg.com
    1344701126 10.0.1.2 ocsp.digicert.com
    1344701057 10.0.1.2 zoink.it
    1344701037 10.0.1.2 i49.tinypic.com
    1344701037 10.0.1.2 img259.imageshack.us
    1344701036 10.0.1.2 www.d-addicts.com
    1344701036 10.0.1.2 static.d-addicts.com
    1344701036 10.0.1.2 w.dramafever.com
    1344700997 10.0.1.2 imgdf-a.akamaihd.net
    1344700987 10.0.1.2 media.dramafever.com
    1344700940 10.0.1.2 safebrowsing-cache.google.com
    1344700915 10.0.1.2 d3g0gp89917ko0.cloudfront.net
    1344700901 10.0.1.2 ocsp.godaddy.com
    1344700898 10.0.1.2 www.google.com
    1344700896 10.0.1.2 edge.quantserve.com
    1344700873 10.0.1.2 safebrowsing.clients.google.com
    1344700851 10.0.1.2 www.google.com.au
    1344700845 10.0.1.2 google.com.au
    1344700809 10.0.1.2 c.microsoft.com
    1344700798 10.0.1.2 c.live.com
    1344700797 10.0.1.2 c.atdmt.com
    1344700791 10.0.1.2 res2.windows.microsoft.com
    1344700791 10.0.1.2 res1.windows.microsoft.com
    1344700790 10.0.1.2 m.webtrends.com
    1344700789 10.0.1.2 windows.microsoft.com
    1344700789 10.0.1.2 js.microsoft.com
    1344700789 10.0.1.2 evsecure-ocsp.verisign.com
    1344700785 10.0.1.2 download.microsoft.com
    1344700785 10.0.1.2 go.microsoft.com
    1344700784 10.0.1.2 ajax.aspnetcdn.com
    1344700784 10.0.1.2 ajax.microsoft.com
    1344700775 10.0.1.2 explore.live.com
    1344700772 10.0.1.2 download.live.com
    1344700767 10.0.1.2 get.live.com
    1344700682 10.0.1.9 ocsp.startssl.com
    1344700659 10.0.1.9 weather.service.msn.com
    1344700565 10.0.1.9 ssl.gstatic.com
    1344700565 10.0.1.9 www.google.com.au
    1344700563 10.0.1.9 www.google.com
    1344700559 10.0.1.9 dynupdate.no-ip.com
    1344700559 10.0.1.9 google.com.au
     
  9. Cyberian75

    Cyberian75 Network Guru Member

    Is there a way to change its path?
     
  10. koitsu

    koitsu Network Guru Member

    Because /proc is a kernel-level filesystem on Linux known as procfs, not an actual disk like what you'd think. This is 100% normal. You can read about details here:

    http://www.thegeekstuff.com/2010/11/linux-proc-file-system/
    http://linux.die.net/man/5/proc
    http://en.wikipedia.org/wiki/Procfs#Linux
    http://www.linuxjournal.com/article/8381
    http://www.ibm.com/developerworks/linux/library/l-proc/index.html

    This means that the kernel itself is what's tracking/storing all of those IPs/etc., not a daemon. If you ask me, this is the wrong place for tracking such data (that's a matter of opinion though, because on FreeBSD we try very hard to offload things like this to userland using ioctl() or similar models; it keeps the crap out of the kernel, while Linux tends to shove a lot of things into the kernel which should probably be in userspace).

    Anyway, hope that answers your question.
     
  11. Cyberian75

    Cyberian75 Network Guru Member

    Thanks for the links.
     
  12. Cyberian75

    Cyberian75 Network Guru Member

    Again, it'd be nice if the path can be changed.
     
  13. Cyberian75

    Cyberian75 Network Guru Member

    Is there a way to redirect the output to a file?
     
  14. leandroong

    leandroong Addicted to LI Member

    Redirection is possible. File is accessible by putty, telnet or winscp. Above example, I was able to display contents.
     
  15. Cyberian75

    Cyberian75 Network Guru Member

    To save it on another drive, I mean.
     
  16. dc361

    dc361 LI Guru Member

    Why not create a script or cron job to periodically copy the contents of the file to somewhere else like a connected usb drive?
     
  17. Cyberian75

    Cyberian75 Network Guru Member

    I thought of doing that, but there will be duplicates unless I reboot the router after each save.
     
  18. Cyberian75

    Cyberian75 Network Guru Member

    How do you empty those files via CLI?
     
  19. Dark_Shadow

    Dark_Shadow Addicted to LI Member

  20. Cyberian75

    Cyberian75 Network Guru Member

    Uhh, those are system files that can't be deleted.

    I've tried "rm" and "cat /dev/null" methods, but they simply do not work.
     
  21. leandroong

    leandroong Addicted to LI Member

    Hopefully, Shibby would modify FW and provide saving path textbox.
     
  22. Cyberian75

    Cyberian75 Network Guru Member

  23. Monk E. Boy

    Monk E. Boy Network Guru Member

    I imagine that if you restart whatever's creating the webmon entries that it would clear the files. Basically the script, when run periodically, would touch a file (or files) with a timestamp of today in an external directory, append data from webmon to the timestamped file (/files), then change settings for whatever process is handling webmon to reset the source files (either reload the process, or set them to 0 then set them back to 1000 or whatever value you choose).

    A dirtier way would be to just append the output of webmon to a file with today's date on it. Since each entry is timestamped, someone with better scripting/perl/etc. skills than I could extract non-duplicate data from the daily files and export it to another file.
     
  24. Cyberian75

    Cyberian75 Network Guru Member

    I believe it's the iptables that's creating the webmon entries.
     
  25. koitsu

    koitsu Network Guru Member

  26. Monk E. Boy

    Monk E. Boy Network Guru Member

    Yeah, I knew when I wrote that that process was the wrong term to use. I still don't understand linux's monolithic kernel design except in a kind of esoteric way, I started out in the NeXTStep camp which I think is why so many things seem odd to me (even OS X is getting odder as it goes along).

    There's a way to control those webmon entries, to turn them off completely, make them larger, etc. and if someone understood the mechanism for changing the values used to determine the number of entries then it would be a simple method of basically copying all the information out of the file periodically, setting it to 0 to completely erase the contents, then set it back to 1000 or whatever value you feel is appropriate for your usage, and off it goes until the next time the cron job runs.

    The "dirty" way would work without turning anything off, just periodically append the contents of a webmon file to a daily file, then after the day is over run some kind of script to massage the contents to remove duplicate entries.

    I did stumble across this post:
    http://www.linksysinfo.org/index.php?threads/script-email-webmon-log-daily.38362/

    Which is using grep to pull individual entries out and email them.

    The real glaring problem with all these methods is if you don't have static leases then you also have the problem of resolving the IP to a specific host when looking through the logs. I sometimes have well over 300 devices pass through a router in a single day, figuring out which device had which IP at which time would take some advanced wizardry.
     
  27. koitsu

    koitsu Network Guru Member

    I'm hearing you on FM. What I don't understand is why this kind of crap isn't in userland. In fact, I just read this today -- more specifically, the quoted "boxed" content here -- and my heart sank even more. Just shove it all in the kernel. *shakes head*
     
  28. Monk E. Boy

    Monk E. Boy Network Guru Member

    I remember when having your project run across all Unix platforms was the most important thing. But I digress...
     
    koitsu likes this.

Share This Page