Script Contribution: e-mail dShield Firewall logs.

Discussion in 'Tomato Firmware' started by mraneri, Aug 16, 2011.

  1. mraneri

    mraneri Network Guru Member

    I wrote this script and have it running for about 2 or 3 months now without issue. This script processes your router logs and emails them to dshield on a daily basis. There are a number of prerequisites. First here's the code:
    cd /var/log
    if [ ! -s fw.ts ] ; then date +%s > fw.ts ; fi
    time=$(( ($(date +%s) - $(cat fw.ts) ) / 60 ))
    entries="$(cat messages.8 |wc -l)"
    if [ -s messages ] ; then
        if [ ! -s messages.0 ] ; then mv messages messages.0 ; fi
        if [ $entries -le 4000 ] ; then grep " user.warn kernel: DROP " messages.0 |sed -e "s/ DF / /g" -e "s/ \(IN\|OUT\|MAC\|TOS\|PREC\|ACK\|WINDOW\|RES\|SEQ\|TTL\|ID\|LEN\|URGP\)=[^ ]*//g" -e "s/user.warn kernel/u.w k/" >> messages.8 ; fi
        grep -v " user.warn kernel: DROP \|DHCPREQUEST\|DHCPDISCOVER\|DHCPOFFER\|DHCPINFORM" messages.0 |cat messages.3 - |awk '{a[i++]=$0}END{for(j=0;j<i;j++)if(j>(NR-100)||a[j]!~/DHCPACK|-- MARK --|no change was needed/)print a[j]}'|tail -1000 > ltmp
        rm messages.0
        mv ltmp messages.3
        entries="$(cat messages.8 |wc -l)"
        echo "$(date "+%b %d %T ------") $entries FIREWALL ENTRIES LOGGED IN THE LAST $(($time/60))h$(($time%60))m." > messages.2
    if [ $entries -gt 2000 -a $time -gt 57 -o $time -gt 1437 -a $entries -ne 0 ] ; then
        logger "EMAILING $entries FIREWALL ENTRIES TO DSHIELD... (Last e-mail was $(($time/60))h$(($time%60))m ago)"
        date +%s > fw.ts
        echo -e "To:\nBcc: ###YOUREMAILADDRESS###\nSubject: FORMAT IPTABLES USERID ###YOURDSHIELDID### TZ $(date +%z | sed "s/[+-]\([0-9][0-9]\)/&:/")\n\n" \
              |cat - messages.8 |sendmail -f ###YOUREMAILADDRESS### -S ###YOURSMTPSERVER### "-au###YOURUSERNAME###" "-ap###YOURPASSWORD###"
        rm messages.8
    Prerequisistes and Usage Notes:
    1. You shouldn't be bothering if you don't understand what's in here: Read that first.
    2. It is dependent on your tomato firmware having been compiled with "sendmail". There are a number of distributions which do, and many which don't. I'm using toastman's VPN build with good success, which includes sendmail. My recommendation is you log into your router and type sendmail. If you have it, you'll know. If you don't, you'll know. If your firmware doesn't include sendmail, please don't complain about it here. If you can't get sendmail to work with your e-mail/ISP provider, please don't complain or ask for help here. Please search the net and/or ask for help in another thread. Otherwise, this will turn into a sendmail support thread, which isn't what it's really about.
    3. You need a dshield account unless you want to submit your logs anonymously. Again, you should know this since you already read the link in the first bullet.
    4. You must edit anything which is wrapped in ### with your own information. Sorry I didn't make variables at the top. Just search for ### and make the edits. You can do it.
    5. This code was written to be inserted into one of the three Administration->Scheduler->Custom commands. My recommendation is to copy and paste the code block into a text editor. Make the required edits (search for ###), and then cut and paste it into an available Custom scheduled command. My recommendation is to run every 5 minutes every day. (Why will become clear once you understand what the script is doing.)
    6. My recommendation also is to set this up for a few days and change the "To:" to your own e-mail address. You should make sure the e-mails are being sent properly and you are ok with the information in the log before you start sending these logs to a third party. Once you are comfortable, you can change it back.
    7. There is a Bcc: in the E-Mail which gets sent. This way, you can continue to receive a copy of the e-mails for your own monitoring. My recommendation is to setup a filter in your e-mail program to put those messages somewhere useful, or possibly in the trash (where you can find them for 30 days or whatever.) If you don't want to get the e-mails, just remove the Bcc: through the \n.
    8. Don't forget to enable Dropped connection logging!!! Go to "Administration->Logging->Connection Logging->Inbound" and enable packets "If blocked by firewall". Also, my recommendation is to keep the 60 messages per limit maximum. 60 messages per minute equals 3600 messages per hour. I don't think logging more than that will add much value.
    Other Notes:
    1. Each time the script runs (every 5 minutes, per the schedule you set up above), the script splits the messages log into two separate logs. (Note: the complete "log" is the concatenation of all messages.* files in reverse order, so the way the script is written, the complete log is always available.)
    2. The new firewall entries are appended to messages.8. This is the file which eventually gets e-mailed. Note, "sed" is applied heavily on the files so as to remove unnecessary information which dSheild does not use. This cut out about 70% of the file size without affecting the useful information in the log (from a dshield perspective.) Useful for saving precious RAM. (I'm running with 16MB of RAM.)
    3. If more than 4000 firewall entries already exist in the log (if you're getting blasted), then new entries are discarded so as not to crash your router.
    4. Also, in the interest of more useful logs, and overly aggressive DHCP renews from my apple devices, the script filters out all non-error DHCP messages. Also, the script filters out DHCPACK's, cron MARK's, and "no change" ntp messages from all but the last 100 log lines.
    5. Since the kernel is no longer really handling truncating of the log (once the file size exceeds 50kB), the script truncates the main log at about 1000 lines. For me, this is approximately 100k of logs.
    6. Per dShield guidelines, logs are e-mailed every 24 hours, or when the firewall log hits 2000 entries, but never more often than once per hour. An entry is added to the log whenever the e-mail is sent. Also, a message is updated at the bottom of the log file each time the script runs to show how many firewall entries are in the firewall log, waiting to be sent to dshield.
    What you may observe:
    1. If you view your entire log ("View All" in the router web interface) you will see all processed firewall entries at the TOP of the log. Then, you will find the rest of the router log. Remember, unless you modify the grep and awk elements on line 11, this log is filtered based on #4 above.
    2. At the very bottom of the log (most recent information) you will see any firewall entries which have not been processed yet. Theoretically, those should be processed within the next 5 minutes. Also, at the bottom you will see the summary of how many firewall entries exist in the current log. Note, this summary is not actually "Logged" and probably won't show up as an entry if you are "sending logs to a remote host." It's just a line written to messages.2 in order that it always shows up at the bottom of the log.
    3. If you look in your "var/log" directory, you will see a number of messages.n files.
      • messages - this file contains any log entries since the previous script run. It shouldn't ever really contain more than 5 minutes of log entries.
      • messages.0 - If you're getting hammered and your messages file grows to > 50kB in 5 minutes, this will contain the continuation of that log.
      • messages.2 - This contains the one liner which states how many entries are in the firewall log.
      • messages.3 - This is actually the main router log now. Every 5 minutes, messages (and messages.0) is filtered and appended to messages.3. Firewall entries are stripped out.
      • messages.8 - This is the cleaned-up firewall log which will eventually get sent to dShield.
    I'm curious how you all find this. Hope it's useful/interesting to some. It certainly was interesting to me. Good luck... And remember, please no sendmail support in this thread!!! If you can't get sendmail working, search for another thread, and/or start your own!!

    - Mike
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice