1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Upload logs to external ftp server

Discussion in 'Tomato Firmware' started by likuidkewl, Nov 23, 2007.

  1. likuidkewl

    likuidkewl Network Guru Member

    This is mainly due to Comcast, yeah its Comcastic alright..

    This is what was filling my /var/log/messages file, making it create a new one every 30-40 minutes or so:
    I decided to venture down this path and have came up with the following, I have put these in the INIT section of my scripts and also have used them from the command line and they work fine.

    You will notice that I like to keep my scripts separate, I think it is a control issue :)

    These scripts work on my WRT54GS v1.1 others are untried.

    I did this so if I felt the need to check my log file from work or somewhere else I didn't have to log into the SSH session and try to do it that way. I also wanted a larger file size so I could capture all the events not just some, hence the syslogd script. Comcast is a pain with the way they have the dhcp server sending out udp packets all the time so I "grep -v" for the comcast dhcp server and delete the messages log, restarting syslogd with custom settings every 4 hours.

    Any comments/suggestions are welcome

    Code:
    sleep 15
    logger ***SYSLOGD SCRIPTS RUNNING***
    echo 'pid=`pidof syslogd` && kill $pid' > /tmp/syslogkill.sh && chmod 777 /tmp/syslogkill.sh
    echo 'syslogd -m 60 -s 500 -b 2 -L' > /tmp/syslogup.sh && chmod 777 /tmp/syslogup.sh
    echo '/tmp/syslogkill.sh && /tmp/syslogup.sh' > /tmp/restartsyslog.sh && chmod 777 /tmp/restartsyslog.sh' > /tmp/upload.sh && chmod 777 /tmp/upload.sh
    echo 'grep -v SRC=73.141.118.1 /var/log/messages >> /var/log/messages.txt && rm /var/log/messages && logger '*LOG FILE COMPACTED*' && sleep 10 && /tmp/restartsyslog.sh' > /tmp/cleanlog.sh && chmod 777 /tmp/cleanlog.sh
    echo 'busybox ftpput -v -u UseR -p P@$$W0Rd YOUr_URL.ORG /Messages_log_$(date -I) /var/log/messages.txt && rm /var/log/messages.txt && logger '*LOG FILE UPLOADED SUCCESSFULLY
    logger **LOG CREATION OF SCRIPTS**
    ls -l /tmp | grep .sh | grep -v script_wanup.sh | logger
    logger **ATTEMPTING TO RESTART SYSLOGD**
    /tmp/restartsyslog.sh
    sleep 2
    logger **NEW SYSLOGD SETTINGS BELOW**
    ps | grep syslogd | grep -v grep | logger
    logger ***END OF SYSLOGD CHANGES***
    cru a cleanlog "58 22,2,6,10,14,18 * * * /tmp/cleanlog.sh"
    cru a ftpupload "00 7 * * * /tmp/upload.sh"
    
    I know I could do an if exist type statement to check the existence of the scripts but I will save that for another time.
     
  2. mstombs

    mstombs Network Guru Member

    Interesting, thanks

    I note you don't put the shebang at the start of your little bash files - but guess it works so why bother!

    One comment, when "messages" is full it gets copied to "messages.0", if messages.0 exists shouldn't you copy that first?
     
  3. likuidkewl

    likuidkewl Network Guru Member

    Maybe I should for "proper" structure! although I think I left it out for a reason.....

    Right and that was the main problem the default settings for the linksys I think were really low and I was only getting a few hours max in the two log files. That is actually how this whole thing started, comcast was loading up the log files with dhcp probes. So I tried to just add the 500kb to the syslogd service and that was fine but then I realized if I left it that way I would have to keep and eye on the logs in regards to size and amount(99 is the max). I also shy away from using a box inside the network to catch the logs as I have a tendency to turn off all the boxes before bed(Think Green! :) ) And if I have webspace why not use it?

    So as of now the log file gets stripped of all(only dhcp) the comcast crap every 4 hours and written to the messages.txt file and then uploaded once a day to the ftp and deleted from the router. This allows me to look at it via webpage at work or dl from ftp. Next is to look at adding a datestamp to the file name.... Maybe!
     
  4. mstombs

    mstombs Network Guru Member

  5. likuidkewl

    likuidkewl Network Guru Member

    Thanks for the link, as for the TAR-ing I thought about that, but I decided against it as I really wanted the option to view th logs in a web browser occasionally.

    //EDIT//
    Here is an example of the amount of traffic that Comcast is generating:
    The first is before grep, the second is after I "grep -v" with the comcast dhcp server.
     
  6. likuidkewl

    likuidkewl Network Guru Member

    Ok I updated the first post as to what I had to do to get this moving.

    I have to put it in the WANUP section Init hates me. :)

    Also I think I had an issue at 0258 with the syslogd service not restarting and I don't know why so I put in "sleep" as I think it got hosed trying to cp and grep the file(s) around....
     
  7. mstombs

    mstombs Network Guru Member

    A couple more comments,

    1. Now you have longer scripts, would be easier to read/maintain if you used "here" documents to create them. ie

    cat >/tmp/cleanlog.sh <<-EOF
    #!/bin/sh
    more /var/log/messages | grep -v SRC=73.141.118.1 >> /var/log/messages.txt
    rm /var/log/messages
    logger '*LOG FILE COMPACTED*'
    sleep 10
    /tmp/restartsyslog.sh
    EOF
    chmod +x /tmp/cleanlog.sh

    2. You do realize that the WANUP script runs every time the WAN restarts? It should work once only if run from init - but you may need a 'sleep 10' at the top - it may run too early.
     
  8. likuidkewl

    likuidkewl Network Guru Member

    Thanks for the info, as stated this is simply a personal endeavor with not much time for review :). I also read somewhere yesterday about the use of cat in scripts, I don't remember why I didn't do it that way.. HA!
    Yes, I do actually. This was only done to check and make sure that it worked :)

    I will try sleeping it longer.
     
  9. likuidkewl

    likuidkewl Network Guru Member

    Added the /message_log_$(date -I) to rename the logs by date.
    Still no joy on cat-ing the scripts but they still work. HTH
     
  10. Searcher61

    Searcher61 Network Guru Member

    I have 2 questions....

    first this script should be placed in the Init tab correct?


    second... in the line below I have replaced User with my comcast user ID and P@$$W0Rd with my comcast password and YOUr_URL.ORG with upload.comcast.net but cannot seem to get it to work. Any Ideas as to why?

    ftpput -v -u UseR -p P@$$W0Rd YOUr_URL.ORG
     
  11. likuidkewl

    likuidkewl Network Guru Member

    Yes the init tab works with the delay.

    Using SSH to the router, can you pass that command directly and actually log in?
    ftpput -v -u UseR -p P@$$W0Rd YOUr_URL.ORG /Messages_log_$(date -I) /var/log/messages.txt

    There is a little issue that I found out by trial and error and forgot to mention.

    ftpput -v -u UseR -p P@$$W0Rd YOUr_URL.ORG /Messages_log_$(date -I)

    Note the bold type, the "/" denotes that this is the "home" of that user.
    ie. my "router-upload" account is set to have a sub directory of mydomain.net/router as its root(home directory).
    I found that when I tried to be a 'normal' user(with domain level access) and then specified a sub directory the ftpput command would not work for some reason. So I created another user with mydomain.net/router as a home directory and that worked fine, I think it has something to do with passing "/" in the script. As this works for me I have not really looked into again, well atleast until I get a break at the end of this semester.
     
  12. Searcher61

    Searcher61 Network Guru Member

    Well I have finally had a few min. to test the ftpput command in a SSH. I used putty to connect and turned on SSH and login on the router for the testing. From the shell I can use the same line to ftp and it works great. When I run it from the script, I never get the file uploaded message. I am going to continue to look at it....the last time I did any shell scripting was years ago....

    I also noticed the line 'syslogd -m 60 -s 500 -b 2 -L' but when I look the syntax I don't see and -m (see below).

    syslogd

    syslogd [OPTION]...
    System logging utility. Note that this version of syslogd ignores /etc/syslog.conf.

    Options:

    -n Run in foreground
    -O FILE Log to given file (default=/var/log/messages)
    -l n Set local log level
    -S Smaller logging output
    -s SIZE Max size (KB) before rotate (default=200KB, 0=off)
    -b NUM Number of rotated logs to keep (default=1, max=99, 0=purge)
    -R HOST[:pORT] Log to IP or hostname on PORT (default PORT=514/UDP)
    -L Log locally and via network (default is network only if -R)
    -C[size(KiB)] Log to shared mem buffer (read it using logread)

    I think my ftp problem might be a setting in the router that might be causing it. Any ideas?
     
  13. Searcher61

    Searcher61 Network Guru Member

    Well I found my problem...in likuidkewl's script the line:

    <code>
    echo 'busybox ftpput -v -u UseR -p P@$$W0Rd YOUr_URL.ORG /Messages_log_$(date -I) /var/log/messages.txt && rm /var/log/messages.txt && logger '*LOG FILE UPLOADED SUCCESSFULLY
    </code>

    Should have "> /tmp/upload.sh" on the end. without it the upload.sh file is not created.

    I have been rewriting the script with a few of the suggestions from others...but am having a problem rewriting one section the way mstombs suggested. I have to find the syslogd process and kill it....how do I find the right syslogd process if there are more than one?
     
  14. mstombs

    mstombs Network Guru Member

    I don't think there should be more than one syslogd!

    "killall syslogd"

    should kill all of them!
     
  15. likuidkewl

    likuidkewl Network Guru Member

    Sorry about that the script must have got truncated.

    The following find the running instance of syslogd and kill it, but killall should work too.
    Code:
    echo 'pid=`pidof syslogd` && kill $pid' > /tmp/syslogkill.sh && chmod 777 /tmp/syslogkill.sh
    This was cobbled together in pieces over a few days so there is bound to be a better setup, but for now it works. :)
    Post up what you have done in the end, I would be interested to see it.
     
  16. likuidkewl

    likuidkewl Network Guru Member

    Also syslogd --help spits out this:
    Code:
    Options:
            -m MIN          Minutes between MARK lines (default=20, 0=off)
            -n              Run as a foreground process
            -O FILE         Use an alternate log file (default=/var/log/messages)
            -S              Make logging output smaller.
            -s SIZE         Max size (KB) before rotate (default=200KB, 0=off)
            -b NUM          Number of rotated logs to keep (default=1, max=99, 0=purge)
            -R HOST[:PORT]  Log to IP or hostname on PORT (default PORT=514/UDP)
            -L              Log locally and via network logging (default is network only)
    What version of Tomato are you using?
     
  17. Searcher61

    Searcher61 Network Guru Member

    Well, I have not changed much...I just cleaned likuidkewl's script up and made it a little bit easier for me to read. I am running the cleanlog and upload script the frist time at the end of the script as an easy check to make sure it works. I have not had a lot of time either but I am going to get the time and date stamp working. Might not be until after Christmas but I will.

    Code:
    sleep 15
    logger ***SYSLOGD SCRIPTS RUNNING***
    
    cat >/tmp/syslogkill.sh <<-EOF
    #!/bin/sh
    killall syslogd
    EOF
    chmod 777 /tmp/syslogkill.sh
    logger *SYSLOGKILL SCRIPT CREATED*
    
    cat >/tmp/syslogup.sh <<-EOF
    #!/bin/sh
    syslogd -m 60 -s 500 -b 2 -L
    EOF
    chmod 777 /tmp/syslogup.sh
    logger *SYSLOGUP SCRIPT CREATED*
    
    cat >/tmp/restartsyslog.sh <<-EOF
    #!/bin/sh
    /tmp/syslogkill.sh 
    /tmp/syslogup.sh
    EOF
    chmod 777 /tmp/restartsyslog.sh
    logger *RESTARTSYSLOG SCRIPT CREATED*
    
    cat >/tmp/cleanlog.sh <<-EOF
    #!/bin/sh
    more /var/log/messages | grep -v SRC=73.141.118.1 /var/log/messages >> /var/log/messages.txt
    rm /var/log/messages 
    logger *LOG FILE COMPACTED*
    sleep 10
    /tmp/restartsyslog.sh
    chmod 777 /var/log/messages.txt
    EOF
    chmod 777 /tmp/cleanlog.sh
    logger *CLEANLOG SCRIPT CREATED*
    
    cat >/tmp/upload.sh <<-EOF
    #!/bin/sh
    ftpput -v -u username -p p@ssword upload_address /Messages_log.txt /var/log/messages.txt
    rm /var/log/messages.txt
    logger *LOG FILE UPLOADED SUCCESSFULLY*
    EOF
    chmod 777 /tmp/upload.sh
    logger *UPLOAD SCRIPT CREATED*
    
    logger ******ALL SCRIPTS CREATED******
    
    ls -l /tmp | grep .sh | grep -v script_wanup.sh | logger
    logger **ATTEMPTING TO RESTART SYSLOGD**
    /tmp/restartsyslog.sh
    sleep 2
    
    logger **NEW SYSLOGD SETTINGS BELOW**
    ps | grep syslogd | grep -v grep | logger
    
    /tmp/cleanlog.sh
    /tmp/upload.sh
    
    logger ***END OF SYSLOGD CHANGES***
    cru a cleanlog "58 22,2,6,10,14,18 * * * /tmp/cleanlog.sh"
    cru a ftpupload "00 7 * * * /tmp/upload.sh"
    
    I like this Idea because I can check logs from where ever I am and don't have to leave access to the router turned on.

    BTW, I am using Tomato Firmware v1.11.1217 on a WRT54G v2.
     
  18. mstombs

    mstombs Network Guru Member

    Great - I find that easier to read - does it work? I thought the EOF markers had to be unique so use EOF1 EOF2 etc - but you learn something every day!
     
  19. Searcher61

    Searcher61 Network Guru Member

    So far so good...creates each script correctly. I have been working on adding the date to the end of the file...got that to work also, however I cannot get ftpput to upload the file if you don't know the exact name. I even have set a variable equal to the name but ftpput does not like it. It only works if you know the name and you enter it in both remote and local paths.

    After thinking about this, since I use comcast as my storage, and I am not sure comcast allows browsing of dir. I might have a hard time viewing the files if I did not know the exact name. Hmmm have to think about this.
     
  20. likuidkewl

    likuidkewl Network Guru Member

    I fixed this issue like this:
    Code:
    ftpput -v -u uXXX-p Password MY.url.org [B]/log_$(date -I)[/B] /var/log/messages.txt && rm /var/log/messages.txt && logger '*LOG FILE UPLOADED SUCCESSFULLY*' && sleep 10 && /tmp/restartsyslog.sh' > /tmp/upload.sh && chmod 777 /tmp/upload.sh
    Then the files are named http://www.yoursite.org/log_2007-12-22 (log_YEAR-MO-DA)

    HTH
     
  21. Searcher61

    Searcher61 Network Guru Member

    Thanks likuidkewl, I have incorporated your last change into the script.

    I only have one problem now... I have so much traffic on my router that I seem to have multiple message files. EX: messages.0, messages.1, etc. I could as one person suggested tar them and then send it. But like likuidkewl, I wanted to be able to read them from the web without untar' ing. The way the script is now...it does nothing with any of the other files, so you lose the data in messages.0 and messages.1. Looks like some type of a loop will be needed to cycle through any message files, search and remove the info you want to remove... and either rename each of the files and upload them, tar them together and upload them (then you cant view from web) , append each of the files together upload the one file or upload the file more often and change the date stamp to include time.

    If I do the last one...I don't know how I would know what the name of the file is since comcast does not allow you to list files on the website. Thinking the loop and then appending the output of each of the messages.0, messages.1 etc to the messages.txt file might be the easiest.

    Anyone have any suggestions? The script works Great except that one problem.

    Code:
    sleep 15
    logger ***SYSLOGD SCRIPTS RUNNING***
    
    cat >/tmp/syslogkill.sh <<-EOF
    #!/bin/sh
    killall syslogd
    EOF
    chmod 777 /tmp/syslogkill.sh
    logger *SYSLOGKILL SCRIPT CREATED*
    
    cat >/tmp/syslogup.sh <<-EOF
    #!/bin/sh
    syslogd -m 60 -s 500 -b 2 -L
    EOF
    chmod 777 /tmp/syslogup.sh
    logger *SYSLOGUP SCRIPT CREATED*
    
    cat >/tmp/restartsyslog.sh <<-EOF
    #!/bin/sh
    /tmp/syslogkill.sh 
    /tmp/syslogup.sh
    EOF
    chmod 777 /tmp/restartsyslog.sh
    logger *RESTARTSYSLOG SCRIPT CREATED*
    
    cat >/tmp/cleanlog.sh <<-EOF
    #!/bin/sh
    more /var/log/messages | grep -v SRC=73.141.118.1 /var/log/messages >> /var/log/messages.txt
    rm /var/log/messages 
    logger *LOG FILE COMPACTED*
    sleep 10
    /tmp/restartsyslog.sh
    EOF
    chmod 777 /tmp/cleanlog.sh
    logger *CLEANLOG SCRIPT CREATED*
    
    cat >/tmp/upload.sh <<-EOF
    #!/bin/sh
    ftpput -v -u uXXX-p Password MY.url.org /log_$(date -I) /var/log/messages.txt
    rm /var/log/messages.txt
    logger *LOG FILE UPLOADED SUCCESSFULLY*
    sleep 10
    EOF
    chmod 777 /tmp/upload.sh
    logger *UPLOAD SCRIPT CREATED*
    
    logger ******ALL SCRIPTS CREATED******
    
    ls -l /tmp | grep .sh | grep -v script_wanup.sh | logger
    logger **ATTEMPTING TO RESTART SYSLOGD**
    /tmp/restartsyslog.sh
    sleep 2
    
    logger **NEW SYSLOGD SETTINGS BELOW**
    ps | grep syslogd | grep -v grep | logger
    
    /tmp/cleanlog.sh
    /tmp/upload.sh
    
    logger ***END OF SYSLOGD CHANGES***
    cru a cleanlog "58 22,2,6,10,14,18 * * * /tmp/cleanlog.sh"
    cru a ftpupload "00 7 * * * /tmp/upload.sh"
    
     
  22. mstombs

    mstombs Network Guru Member

    I think you could change

    Code:
    more /var/log/messages | grep -v SRC=73.141.118.1 /var/log/messages >> /var/log/messages.txt
    to

    Code:
    [ -r /var/log/messages.0 ] && cat /var/log/messages.0 | grep -v SRC=73.141.118.1 > /var/log/messages.txt
    cat  /var/log/messages | grep -v SRC=73.141.118.1 >> /var/log/messages.txt
    
    to pick up both message files in correct order, if the old one exists. It would also be easier if you used messages.txt in say /var, so you could delete both message files with "rm messages*" !
     
  23. likuidkewl

    likuidkewl Network Guru Member

    The syslog -s option is the size option you may increase it to fit your needs.
    I had set it to 500 in case of "room" issues. Looking at my router at this instant "free" says I have more than 17 megs available. So I don't think increasing the size to say 1024 will hurt, since you are still cleaning it out every four hours. This would alleviate the need to step through all the logs.

    But stepping through should just entail parsing messages && messages.* with an IF.

    //Edit///
    Which is what was posted by mstombs as I was posting [-rfile]

    //#2//
    Although I can't recall if "if"s used -r correctly in tomato or did I have to use -s??? Sorry but it is one of the two if you feel like parsing them all instead of increasing the size.
    -r is "exists and is readable"
    -s is "exists and is greater then 0"

    hth
     
  24. Searcher61

    Searcher61 Network Guru Member

    Well, with all the things my kids, wife and myself are doing on the internet these days...increasing the size to 1024 did not help. Seems I am still getting multiple message files.... ex: messages, messages.0, messages.1. Looks like I am going to have to either loop through them or upload more often.
     
  25. likuidkewl

    likuidkewl Network Guru Member

    That is odd, because I have 3 pcs online all the time and 2 on half the time and my logs are not that large after 4 hours.... What else are you doing on the router? ie. site blocking etc.
     
  26. likuidkewl

    likuidkewl Network Guru Member

    FYI -- I also found that if you hava a Wii connected to the Tomato AP that it polls for an IP address every 3-10 minutes so the the logs are full of requests for that too. you may want to grep -v those.
     
  27. Searcher61

    Searcher61 Network Guru Member

    Yeah I do have a Wii...but connected to a different router. I think it may come from the amount of GAME play my kids do.
     

Share This Page