1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[SCRIPT] Email webmon log daily...

Discussion in 'Tomato Firmware' started by mraneri, Jul 18, 2012.

  1. mraneri

    mraneri LI Guru Member

    You can choose whether this poses a security risk or not. I use this to monitor what my kids are doing. I put the below script in the scheduler, and have it run daily at 4AM. It returns all the web searches by machine (sorted by machine IP address) and all the web hosts as well. Since almost all of my machines are static DHCP machines, I pull in the machine name from the hosts file so I don't have to look it up.

    You need a tomato installation which supports webmon (duh!) and also you need sendmail. I'm using Toastman 7500 USB with VPN, K2.6.

    Here's the code:
    Code:
    cat > /tmp/skiphosts <<EOF
    .apple.
    .avast.
    EOF
     
    cd /var/log
    (sed "s/^/Web Searches\t/" /proc/webmon_recent_searches|sort -t. -k4n -k1n,2;sed "s/^/Targets\t/" /proc/webmon_recent_domains|grep -Fvf /tmp/skiphosts|sort -t. -k4n -k1n,2)|
    awk 'BEGIN{OFS="\t";FS="\t";last="";group="";max=0;cnt=0
    if ((getline dt<"wm.ts")!=1) dt=0;close("wm.ts")
    print"To:EMAIL@EMAIL.COM\nSubject: Internet Usage for "strftime("%Y-%m-%d",systime()-86400)}
    {
    if($4!="")if($2>dt){
    if(group!=$1){if(group!="")print"";print"";print "##### "$1" #####";last="";group=$1}
    if(last!=$3){if(last!="")print"";cmd="grep -F \""$3" \" /tmp/etc/dnsmasq/hosts/hosts";cmd|getline ip;close(cmd);last=$3;if(ip!="")$3=ip;print"------------------------------ "$3;ip=""}
    print strftime("%m-%d %H:%M:%S",$2),$4};cnt=cnt+1
    if($2>max)max=$2;
    }END{print max > "wm.ts";system("logger Emailed "cnt" webmon items.")}'|
    sendmail -f EMAIL@EMAIL.COM -S smtp.EMAIL.COM "-auUSERNAME" "-apPASSWORD"
    There's a here document for hosts to skip (does not filter searches, only domains.) Feel free to put in there what you want. We get lots of useless entries of apple devices calling home, so I filter those out, for instance.

    Don't forget to change your e-mail stuff.. Replace two instances of your e-mail address (EMAIL@EMAIL.COM) and also your USERNAME and PASSWORD on the last line. Note username goes after -au, and password goes after -ap. (Don't remove the extra u or the extra p.)
    I manually "packed" the code to make it as small as possible so as not to take up too much nvram. I'm sure others can do better, but it's pretty compact for what it does, I think.

    Email looks like this:
    Code:
    ##### Web Searches #####
    ------------------------------ 192.168.1.20 Device_2
    07-13 14:10:01  searchterm
    07-13 14:10:33  searchterm
    07-13 14:46:36  searchterm
    07-13 14:46:49  searchterm
     
    ------------------------------ 192.168.1.40 Device_3
    07-13 16:59:10  searchterm
    07-13 22:33:26  searchterm
    07-13 22:35:59  searchterm
    07-13 22:36:48  searchterm
    
    
    ##### Targets #####
    ------------------------------ 192.168.1.10 Device_1
    07-13 06:25:01  www.domain.com
    ------------------------------ 192.168.1.20 Device_2
    07-13 21:05:59  www.domain.com
    07-14 03:48:24  www.domain.com
     
    ------------------------------ 192.168.1.50 Device_4
    07-14 00:05:28  www.domain.com
    07-14 03:36:29  www.domain.com
    07-14 03:57:29  www.domain.com
    
    I hope someone finds it useful.
     
  2. mraneri

    mraneri LI Guru Member

    P.S. I email this to myself (of course), and I have my mail program automatically file these in a folder so they don't clutter my inbox. This way, I have a log of what my kids are doing on the web which survives reboots, and can go back in time to see when they started looking into something.

    (Of course, the logs that are e-mailed survive reboots! But a reboot during the day means any monitoring since 4AM that morning is lost.)

    Also, I do realize, if my kids start using https://www.google.com to do their searches, the search terms don't show up. No way for the router to snoop a secure connection (That IS the point, right?) Anyway, hope they don't figure that out too soon.

    - Mike
     
  3. jbktomato

    jbktomato Connected Client Member

    Mike, this is exactly what I'm looking for... I have several tomato routers on various locations and rather then have to login to each one remotely to check web usage, I would like each one to email me there web usage but I'm having trouble using the above code. In you inital paragraph you mention having sendmail. Could someone please help me setting up sendmail on a tomato router or any working alternative for either a smtp or gmail emailing account.

    Thank you...
     
    Last edited: Jan 8, 2015
  4. mraneri

    mraneri LI Guru Member

    spammers have forced mail servers to become smarter. My ISP suddenly started rejecting the emails. I looked into it a while back, but didn't come up with a solution. If anyone else has some suggestions to make the e-mail sending part more "robust", please share...
     
  5. jbktomato

    jbktomato Connected Client Member

    mraneri, thanks for the reply. I truly hoping there is solution. thanks again!
     
  6. mstombs

    mstombs Network Guru Member

    I do recall someone getting gmail to work - needed certain openssl components for the secure send (s_client), but has the advantage you don't need to use ISP mailservers, should be able to use busybox sendmail, see help comments in the code

    http://git.busybox.net/busybox/tree/mailutils/sendmail.c

    secure send might help?
     
  7. jbktomato

    jbktomato Connected Client Member

    mstombs, I appreciate the reply but I got to be honest, that code is way above this rookies's skill levels. I wouldn't know where to start... I've thought I had a solution by searching this forum with mraneri's original post from 2012 but seems he is having issues now. They end result from mraneri's post was exactly what I want. I have roughtly 50+ remote locations that I would like to have send me daily various info (web usage, bw, usage by ip). Any and all help is always appreciated.
     
  8. mstombs

    mstombs Network Guru Member

    Well 50+ locations I think you should buy some proper qualified support!

    First you need a version of tomato with the s_client patch on openssl, despite what is says in this thread

    http://www.linksysinfo.org/index.ph...shibby-toastman-openssl-s_client-patch.68359/

    Then it would just be a patch on the original script.

    I don't think any of my routers toastman or shibby have the command.

    I'm sure there are plenty of other ways to do what you want with existing tools though.
     
  9. jbktomato

    jbktomato Connected Client Member

    If I could find someone to pay for qualified help on this matter, I wouldn't even hesitate. I'm currently using ASUS RT-16 routers with shibby newest firmware with VPN. How would one test to see if there version of tomato has the s_client patch? Thanks again for your help...
     
  10. jerrm

    jerrm Network Guru Member

    Are we sure ssl is the issue?

    Shibby was the only dev to include s_client and he has since removed it due to space concerns (all to support 4MB units, the 4MB builds should be forked off onto a critical security updates only tree but it's not my call). He subsequently added it back to ARM builds, but not MIPS.

    If ssl support is the issue, short of making your own build with s_client, you would need Optware or Entware installed, then either use their version of openssl, or just install msmtp and use it. Alternatively I think there are some static openssl and/or msmtp binaries around from either @lancethepants or @rhester72.
     
  11. lancethepants

    lancethepants Network Guru Member

    I have msmtp binary, can easy fit in jffs of rt- n16.
     
  12. jbktomato

    jbktomato Connected Client Member

    Yeah I see that shibby included s_client in the MIPS builds around v109 but removed it a short time later. Sure wish there was a easy solution similiar to the one mraneri did in the orginal post in this thread.

    I see that mraneri talks about "sendmail". Is s_client needed for sendmail?
     
  13. jbktomato

    jbktomato Connected Client Member

    lancethepants, whats the chance of me getting that binary file from you?
     
    Last edited: Jan 9, 2015
  14. jerrm

    jerrm Network Guru Member

    If you need ssl support, yes.
     
  15. jerrm

    jerrm Network Guru Member

    lancethepants likes this.
  16. lancethepants

    lancethepants Network Guru Member

    Jeerm got the link.
     
  17. jbktomato

    jbktomato Connected Client Member

    Thanks guys... Now that I got the file, how do I install it on router?
     
  18. jbktomato

    jbktomato Connected Client Member

    Installed Optware on usb, now have sendmail and tried using Gmail and well as my ISP provided email and still no go with above script. Does anyone have a simple script to send an email using router with sendmail. Thank you...
     
  19. mstombs

    mstombs Network Guru Member

    Most tomato mods have a simple BusyBox "sendmail", I don't know what variants are available using optware/Entware.

    I used to use a RouterTech distributed msmtp binary before Tomato had one built in

    http://www.linksysinfo.org/index.php?threads/send-email-upon-reconnect.27093/#post-127731

    I have just tested Lance's msmtp above, using the example gmail configs in

    http://www.linksysinfo.org/index.ph...man-openssl-s_client-patch.68359/#post-225301

    This runs but it is seen as an attempt at account hack by Google. It is possible to tell Google to use less secure mode - but I don't recommend doing that on your main mobile google account!

    https://support.google.com/accounts/answer/6010255
    https://wiki.archlinux.org/index.php/msmtp

    I wonder if ISP specific mail servers now have similar issues, probably want a strong certificate exchange etc as well as originate from an IP in their customer range.
     
  20. jerrm

    jerrm Network Guru Member

    The gmail security issues are not really anything to be scared about. It arises out of google's desire to be the center of your online world without the need to share passwords with third parties. It protects users from themselves more than anything else.

    That said, you should be aware of it, and I would never use my "regular" email account for something like this where the password would be stored as plain text on multiple devices.

    The bigger issue is the tls-certcheck=off. Realize, however small you think the risk is, without authenticating the certificate, you are subject to man in the middle attacks, and can't be sure you are talking to the desired server. Another reason to not only use a throw away service account, but also to be mindful of what data you are sending using this method.
     
  21. jerrm

    jerrm Network Guru Member

    I agree with @mstombs, use msmtp. Should be in Optware (but I use and prefer Entware). The link he posted shows usable config files, but everything can also be specified on the command line.
     
  22. jbktomato

    jbktomato Connected Client Member

    Thanks guys, I really appreciate your help... In the first link that mstombs provided, how and where do I input and/or modify that code. For the life of me, I can't figure out how this is done. Also, it appears the code is referencing the file "msmtp" being in the jffs. Mine is located on the attached USB. Lance the pants in his earlier post talked about having a msmtp binary file that could be placed in the jffs. How do I place my current msmtp file on the jffs and second question, to use Lance's file, is it necessary to use Tomatoware or will it run on Optware?

    Thank you...
     
  23. jerrm

    jerrm Network Guru Member

    Lance's file is built as a static executable, meaning it is completely self contained and does not need any external libraries. You don't need optware or anything else to use it, just copy the single file to jffs and run it.

    The Optware msmtp requires the Optware libraries and environment to be available. Moving to jffs is useless unless you load all of Optware on jffs. Chances are you don't have jffs large enough for that to be practical.
     
  24. jbktomato

    jbktomato Connected Client Member

    How? Is there a special program I need to download to do this? Do I use Telnet?
     
    Last edited: Jan 11, 2015
  25. jerrm

    jerrm Network Guru Member

    Assuming jffs is enabled:
    Code:
    cd /jffs
    wget http://files.lancethepants.com/Binaries/msmtp/msmtp%201.4.32/msmtp
    chmod +x msmtp
     
  26. mstombs

    mstombs Network Guru Member

    I used router usb drive, downloaded using a windows PC and coped to the network available shared drive (could have also coped to jffs, but I don't have it enabled at the moment).

    Lances binary supports config files

    Code:
    root@rtn66u:/tmp/mnt/usb4gb# msmtp --version
    msmtp version 1.4.32
    Platform: mipsel-unknown-linux-gnu
    TLS/SSL library: OpenSSL
    Authentication library: built-in
    Supported authentication methods:
    plain cram-md5 external login
    IDN support: enabled
    NLS: disabled
    Keyring support: none
    System configuration file name: /jffs/etc/msmtprc
    User configuration file name: /root/.msmtprc
    I used the second to test - location on the ram disk the dot name makes it a hidden file

    Code:
    root@rtn66u:/tmp/mnt/usb4gb# ls /root -laF
    lrwxrwxrwx    1 root     root            13 Jun  7  2014 /root -> tmp/home/root/
    Then used putty shh session to test

    Code:
    rtn66u:/tmp/mnt/usb4gb# msmtp -t <testmail.txt
    msmtp: authentication failed (method PLAIN)
    msmtp: server message: 534-5.7.14 <https://accounts.google.com/ContinueSignIn?sarp=1&scc=1&plt=AKgnsbsUU
    ...
    msmtp: server message: 534-5.7.14 l0f3_N1ZoD563X-Wag93DQQdBuAA> Please log in via your web browser and
    msmtp: server message: 534-5.7.14 then try again.
    msmtp: server message: 534-5.7.14 Learn more at
    msmtp: server message: 534 5.7.14 https://support.google.com/mail/bin/answer.py?answer=78754 fo12sm3668781wic.19 - gsmtp
    msmtp: could not send mail (account default from /root/.msmtprc)
     
  27. jbktomato

    jbktomato Connected Client Member

    Thanks guys for both your replies... Using Jerrm's example and a little of mstombs ("msmtp --version"), here is what I got:

    root@unknown:/tmp/home/root# cd /jffs
    root@unknown:/jffs# wget http://files.lancethepants.com/Binaries/msmtp/msmtp 1.4.32/msmtp
    Connecting to files.lancethepants.com (69.167.49.34:80)
    msmtp 100% |***********************************************************************************************| 599k 0:00:00 ETA
    root@unknown:/jffs# chmod +x msmtp
    root@unknown:/jffs# mstmp --version
    -sh: mstmp: not found
    root@unknown:/jffs# msmtp --version
    msmtp version 1.4.28
    Platform: mipsel-unknown-linux-gnu
    TLS/SSL library: GnuTLS
    Authentication library: GNU SASL
    Supported authentication methods:
    plain scram-sha-1 cram-md5 external digest-md5 login
    IDN support: enabled
    NLS: disabled
    Keyring support: none
    System configuration file name: /opt/etc/msmtprc
    User configuration file name: /root/.msmtprc

    When I used the command "msmtp --version" how come I don't see Lance's version of mstmp? I'm using telnet to login into router. Should I be using something else?
     
    Last edited: Jan 13, 2015
  28. jerrm

    jerrm Network Guru Member

    The optware version is in your path andfound by the system. Execute lances by specifying the full path - /jffs/msmtp.
     
  29. jbktomato

    jbktomato Connected Client Member

    Got it and thank you jerrm... Using telnet, is there a way to view all install files on a attached usb and/or jffs?

    Also, I noticed that mstombs router name was in his prompt "root@rtn66u:" and my simply says "root@unknown:" is there a way for me to change that to read "root@rtn16:"??
     
  30. mstombs

    mstombs Network Guru Member

    My router name comes from basic tomato web gui config "Basic->Identification->Hostname" it certainly helps when you have multiple command windows to multiple devices! I also define the domain as "lan" and I can access the shared folder fom windows machine using "\\rtn66u.lan" (I also have nslu2.lan and E3000.lan available!).
     
  31. jbktomato

    jbktomato Connected Client Member

    Thank you mstombs for the reply... just updated my hostname in router gui.

    In you above 3rd example from this morning you have on the first line "msmtp -t <testmail.txt" I'm a little confused what "<testmail.txt" means.
     
  32. Grimson

    Grimson Networkin' Nut Member

  33. jbktomato

    jbktomato Connected Client Member

    LOL!!! I've been managing 50+ locations for approx 6 years just fine without this script or tomato routers. Getting ready to add even more locations and rather than having to remotely login into each and every location daily, I was looking to make the router do a little work by having each one email me various reports (bw usage, webmon etc...) nightly and came across this thread started by mstombs 2 1/2 years ago. Looked like something that would work for me but I'm not familar with programming and needed a little guidance. I guess we can't all be doctors, lawyers or computer programmers. Right???? Based on varioius threads I have read on this forum and several others, sounds like having the routers send emails has been "iffy" (word ???). I was just looking for a little guidance... Once I got 1 router to work, all other routers would be configured the same. I'll still hold out for hope. Thanks to all for the help...
     
    Last edited: Jan 13, 2015
  34. Grimson

    Grimson Networkin' Nut Member

    That's what the first two links are about, the basics of shell scripts on Linux, exactly what you need to learn to get the above script running. If you are unwilling to learn new things it's your problem, and yours alone.
     
  35. jbktomato

    jbktomato Connected Client Member

    OKay... I do appreciate your links and your comments and as we speak, am looking at them. When I responded to this 2+ year old thread a couple days ago, I was simply wondering why it wasn't working for me and have since learned it not even working for the person who made the script "mstombs".
     
  36. jbktomato

    jbktomato Connected Client Member

    Grimson, do have any suggestions on a code editor programming I can use?
     
  37. Grimson

    Grimson Networkin' Nut Member

    Personally I'm using the build-in notepad on windows or vi on linux, but as I'm a purist when it comes to most things I don't need stuff like syntax highlighting.

    For people that like/need more options in an editor Notepad++ (http://notepad-plus-plus.org) is probably a good tool.
     
  38. jbktomato

    jbktomato Connected Client Member

    Cool... thanks for all your help I'll give it a try.

    Just wondering, have you tried the original script from mstombs and had any luck on getting it to work?
     
  39. Grimson

    Grimson Networkin' Nut Member

    Yes, the original script works, except that you need to relay the mail through an existing account instead of letting the router act as a mail server.
     
  40. jbktomato

    jbktomato Connected Client Member

    Okay... Do you mind helping me do this? I hoping this means that I can still have the router send mail without any additional hardware such as a computer...
     
  41. Grimson

    Grimson Networkin' Nut Member

  42. jerrm

    jerrm Network Guru Member

    We've made assumptions the issue is ssl support, but @jbktomato has not given any details as to what is really happening. Without more details, there isn't much else to be said.
     
  43. jbktomato

    jbktomato Connected Client Member

    Thanks Grimson and Jerrm for your replies. Grimson, I have tried and tried to make msmtp work with examples simliar to your above link and its a no go... Jerrm, I don't what details I can give you other that I can't get the script to send a email of the webmon files. I tried even to make a script in the scheduler to send just a test email and nothing. I've work on this off and on all weekend with no luck. Thank you to all for your help...
     
  44. jerrm

    jerrm Network Guru Member

    Post the output of a failed session with debug output.

    If using busybox sendmail, add the -v parameter.

    If using msmtp add the -d parameter.

    msmtp will generally give better info.
     
  45. jbktomato

    jbktomato Connected Client Member

    Thank you for your reply jerrm. As soon I can "learn" how to add stated parameters to these programs, I will try your suggestion. Thanks for all the input/help...
     
  46. jbktomato

    jbktomato Connected Client Member

    jerrm, been trying to set up msmtp thru the linux terminal and configured the msmtprc serveral times using various configs including the link that Grimson sent above and still not working.

    Here's the log:

    Code:
    Jan 13 12:54:27 host=smtp.gmail.com tls=on auth=on user=myemail@gmail.com
    from=myemail@gmail.com recipients=somebody@gmail.com smtpstatus=534 
    smtpmsg='534-5.7.14 <https://accounts.google.com/ContinueSignIn?
    sarp=1&scc=1&plt=AKgnsbuK5\n534-5.7.14 
    Please log in via your web browser and then try again.\n534-5.7.14
    Learn more at\n534 5.7.14
    https://support.google.com/mail/bin/answer.py?answer=78754 rv3sm10727202oeb.16
    - gsmtp' errormsg='authentication failed (method PLAIN)' exitcode=EX_NOPERM
     
    Last edited: Jan 13, 2015
  47. jerrm

    jerrm Network Guru Member

    Did you follow the links @mstombs already posted here regarding ssl smtp connections and gmail security settings: http://www.linksysinfo.org/index.php?threads/script-email-webmon-log-daily.38362/#post-255498
     
  48. Grimson

    Grimson Networkin' Nut Member

    If it doesn't work with gmail, or if you don't want to lower the security settings on your gmail account, there are enough other (free) mail providers out there you can use to send the mails.
     
  49. jbktomato

    jbktomato Connected Client Member

    jerrm, I see that... you asked for me to post the log so I did. My log looks exactly like mstombs in this thread (post#26)

    Grimson, I was determined to make gmail work as per your reply / link in post #41 but am realizing that gmail is not the way to go due to security settings. In your post #39 you stated that the "original script" posted by mstombs 2+ years ago "works". Whats the chance you posting a working script and what changes I need to do to make it work for me. I'm truly can't get it to work and have applied your advise and been trying to make this work on my own to no avail.

    I'm sure I'm not the only one that can / will benefit from having a working script.
     
  50. jbktomato

    jbktomato Connected Client Member

    Grimson, cancel the above request... I got it working!!! Its now sending emails using the orginal above script with many hours and hair loss... Just trying to tweak the script to print how I need it...
     
  51. jbktomato

    jbktomato Connected Client Member

    Now that I have figured out how to make the orginal script in this thread work (with much appreciate help), I'm looking for some additional help... I would like to have the info that is emailed converted to a .txt file and emailed as an attachment. Any and all help is always appreciated...
     
  52. jerrm

    jerrm Network Guru Member

    You'll need to read up on MIME encoding and build the message manually, or use something like mutt to do it for you.
     
  53. jbktomato

    jbktomato Connected Client Member

    Ok... thanks for the help...
     

Share This Page