1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Tomato - shouldn't LOGs be standardized?

Discussion in 'Tomato Firmware' started by MatteoV, Jan 13, 2014.

  1. MatteoV

    MatteoV Serious Server Member

    Hi all.
    I am experiencing a little problem with Tomato RAF's logs. Quite sure it's a common code so I'm posting it here.
    I wrote some scripts that look for something in the log file (/var/log/messages) using awk, the space as a separator, and doing some kind of print $n. This procedure seemed reliable and universally accepted to me, if I'm not proven to be wrong...

    The problem with Tomato is sometimes logs have an additional space after the month: example of normal behaviour
    Code:
    Jan 13 11:09:39 TOMATO daemon.info dnsmasq[1787]: read /etc/hosts - 17 addresses
    
    example of weird behaviour:
    Code:
    Jan  6 15:47:37 TOMATO daemon.info dnsmasq[31398]: read /etc/hosts - 5 addresses
    
    Please see the double space after Jan. That is a big matter, all the scripts will just not work because there's an additional variable there and all the positions hard-coded should now be shifted (+1).
    Is this normal/expected? Or is my method plain wrong? If the method is correct I should do another really silly check if the second print $2 is a space (or empty string) and then change all the positions accordingly, but that highly seems a non-standard method to me...

    Thank you.
     
  2. jerrm

    jerrm Network Guru Member

    There's nothing weird about this, it is pretty standard. Its is how single digit vs two digit dates are formatted by syslog. My ubuntu boxes defaults are the same.

    I don't think there is an option to change this with busybox's syslogd.
     
    koitsu likes this.
  3. MatteoV

    MatteoV Serious Server Member

    Mmh, ok, I understand. I didn't get that was the underlying fact making it act this way.

    Well, so, how would you parse the logs without getting an erroneous value?
    I was doing something like this, for example, to read the IP of last device connected:
    Code:
    LOGFILE="/var/log/messages"
    LOGFILE_br0="$(grep br0 < "$LOGFILE" | grep -v grep)"
    OLD9_br0="$(echo "$LOGFILE_br0" | awk -F'[=| ]' '/DHCPACK/ {print $8}' | tail -1)" #IP
    
    but it would not work then it's a day with a single number...
    Are there better practices?

    Thanks!
     
  4. jerrm

    jerrm Network Guru Member

    If you can get by with multiple consecutive delimiters being treated as one in general, the just use -F'[=| ]+'
     
    MatteoV likes this.
  5. PeterT

    PeterT Network Guru Member

    Try this

    Code:
    LOGFILE="/var/log/messages"
    OLD9_br0="$(awk '/DHCPACK/ && /br0/ { ip=$8 } END { print ip}' $LOGFILE)"
    
     
    MatteoV likes this.
  6. MatteoV

    MatteoV Serious Server Member

    It tuned out both of the ideas are perfectly fitting! I still should directly test the "+"..when I find time, I'm out of home and remotely everything is tooo slow :)...but for now I'm just trusting it. I like improving my code and learning. Thank you

    Inviato dal mio Nexus 4 utilizzando Tapatalk
     
  7. MatteoV

    MatteoV Serious Server Member

    Found a problem with the simplification about LOGFILE reading. I even thought to it before but then I just forgot why I preferred to read logs in a single operation: the opposite is not an atomic operation! It just happened the script found a different hostname but was unable to associate it with the proper (logged) ip and mac due to non-atomic way of reading, as far as I understand. In fact the $8 is just a part, I'm checking also $9 and $10 (ip,mac,hostname).
    It's happening rarely but it's a problem.

    I would like to make everything simple like using awk for finding lines with dhcpack and br0 in the logfile, all in an atomic operation, something like:
    LOGFILE_br0= .. last line with '/DHCPACK/ && /br0/ ...
    then I would like to give awk this input and make it extract $8 into:
    OLD9_br0= ... ...
    I would use grep for LOGFILE_br0:
    Code:
    LOGFILE_br0="$(grep br0 < "$LOGFILE" | grep -v grep | tail -1)"
    then do the old way:
    Code:
    OLD9_br0="$(echo "$LOGFILE_br0" | awk {print $8}')"
    is there a better way with awk also on the first part? I don't know how to make it print the whole line with that filter...

    Thanks!
     
    Last edited: Jan 16, 2014
  8. PeterT

    PeterT Network Guru Member

    The awk variable $0 refers to the entire input record, so changing IP=$8 to lastline=$0 and then the print IP to print lastline will just output the last line that contained br0 and DHCPACK
     
    MatteoV likes this.
  9. koitsu

    koitsu Network Guru Member

    Two semi-off-topic shell scripting comments in passing:

    Code:
    LOGFILE_br0="$(grep br0 < "$LOGFILE" | grep -v grep | tail -1)"
    1. You don't need to make use of redirection here; just grep br0 "$LOGFILE" would be sufficient.
    2. "Double greps" should be avoided if at all possible -- not a necessity in the least (lots of legit reasons to do so), but if using egrep with better regex can help avoid grep -v grep then it should be used.

    Basically "excessive piping" is a really bad habit to get into when writing shell scripts; it's fine for command-line use or "I just need something dirty/quick that does what I want right now and not for permanent use".

    However @PeterT's solution with awk and && is definitely the better choice.

    Code:
    OLD9_br0="$(echo "$LOGFILE_br0" | awk {print $8}')"
    This code, when used with the above code, creates a seriously confusing situation for anyone who looks at it: "does the shell expand $LOGFILE and then append the string _br0 to it, or does it expand $LOGFILE_br0 as a whole variable?"

    The proper solution for this in shell is to use ${} when using variables expansion, e.g. ${LOGFILE} for the former, or ${LOGFILE_br0} for the latter. It's a good habit to get into, especially due to situations like the above.

    There's always points in shell scripting where it's time to give it up and simply use an actual programming language (ex. perl) for what you need. It's remarkable how much crappy shell there is out there in Linux land (speaking generally here, not WRT this thread), stuff that should just become perl given what it's doing.
     
  10. PeterT

    PeterT Network Guru Member

    Not being a scripting expert, I would also think that for the sake of efficiency and maintainability one would want to avoid too many distinct executables being used, along with avoiding the storing of intermediary results in shell variables.
     
  11. MatteoV

    MatteoV Serious Server Member

    Thank you all guys. I started integrating your valuable suggestions!
    In many cases I did use grep -v grep and even add | tail -1 or a lot other stuff. I was able to eliminate any of them and many piping. Same revolution and rethinking of most of the code. Everything looks quite better now. There are still things that seem bad however :p so I have some questions:
    1. is there a way to count lines with awk? For example I'm taking a log in a web accessible file, and I want it to be auto-refreshed. I would not like many "auto-refresh lines" in that file (useless) so I wrote this:
      Code:
      if [ "$(grep "<meta http-equiv=\"refresh\" content=\"${PAGE_REFRESH_TIME}\">" "${LOGFILE0}" | wc -l)" -lt 1 ]; then echo "<meta http-equiv=\"refresh\" content=\"${PAGE_REFRESH_TIME}\">" >> "${LOGFILE0}" ; fi
      ...can this be better (without grep and the piping?)?
    2. How do I pass a command line result to the echo $(#variablename) system I am using to count the characters in a string? Let me show what I am doing and explain a bit:
      Code:
      #Settings
      LOGFILE="/var/log/messages"
      DNSMASQCONF="/etc/dnsmasq.conf"
      
      LOGLINE_br1="$(awk '/DHCPACK/ && /br1/ { lastline=$0 } END { print lastline}' ${LOGFILE})" (remember I want this atomic because shit is happening instead)
      
      NEW_IP_br1="$(echo "${LOGLINE_br1}" | awk -F'[=| ]+' '{print $8}')"
      NEW_MAC_br1="$(echo "${LOGLINE_br1}" | awk -F'[=| ]+' '{print toupper($9)}')"
      
      CONF_DNSM_IPMAC_RES="$(awk '/dhcp-host=/ && /'"${NEW_MAC_br1}"','"${NEW_IP_br1}"'/' "${DNSMASQCONF}")"
      CONF_DNSM_IPMAC_RES_COUNT="$(echo "${#CONF_DNSM_IPMAC_RES}")"
      if [ "${CONF_DNSM_IPMAC_RES_COUNT}" = 0 -o "${CONF_DNSM_HOST_RES_COUNT}" = 0 ]; then
      send e-mail, bla bla bla...
      fi
      even here I wanted to toggle the avoidable (?) variable CONF_DNSM_IPMAC_RES but I was unable to understand the right sintax, shell moans about bad substitution...how to do it right?
    3. Giving awk a string instead of a file where to look for does not work as I hoped. I was willing to eliminate the piping into NEW_IP_br1 and NEW_MAC_br1 up here, I mean, something like NEW_IP_br1="$(awk -F'[=| ]+' '{print $8}' "${LOGLINE_br1}")" ... but it just locks the shell...can this be done in some way?
    Thanks!!!

    [EDIT1]
    Fixed the non-tested awk syntax I wrote, it was wrong for variables' regex ;)
     
    Last edited: Jan 18, 2014
  12. koitsu

    koitsu Network Guru Member

    1. Cease use of wc -l and replace with grep -c.

    2. I have no idea what $#VAR (that includes ${#VAR} although maybe it's $#{VAR} ?) does outside of a subroutine or main (ex. argv) context. I think what you want is described in my last paragraph.

    3. You've now run into "quoting hell" in shell scripts. There are very few ways out of this (and you will not like the solutions given, trust me):

    Code:
    NEW_IP_br1="$(echo "${LOGLINE_br1}" | awk -F'[=| ]+' '{print $8}')"
    NEW_MAC_br1="$(echo "${LOGLINE_br1}" | awk -F'[=| ]+' '{print toupper($9)}')"
    
    Look very close at the double quoting there -- I can see this causing all sorts of problems with some parsers. I recommend you remove the outside double-quotes entirely, ex. NEW_IP_br1=$(echo ...). You should then add appropriate test (ex. if [ ... ]) conditions to ensure that these variables contain something, specifically using the -n or -z test operators.

    Your if [ "${CONF_DNSM_IPMAC_RES_COUNT}" = 0 -o "${CONF_DNSM_HOST_RES_COUNT}" = 0 ]; could really just be re-worked in preceding logic to therefore become if [ -z ${CONF_DNSM_IPMAC_RES} ]; and so on.

    For "null variable content" testing, you can use what I said, although the "old classic way" for older shells (ex. Solaris's /bin/sh which is in no way/shape/form like bash) is to do something like this (you may see it in some older scripts written in older days);

    Code:
    foo=`some commands here which may or may not result in anything`
    
    if [ x$foo = x ]; then
       ...handle situation where $foo is actually empty...
    fi
    
    Though in later shells this has become as simple as (the quoting is necessary):

    Code:
    if [ -z "$foo" ]; then .....
    At this point I am choosing to bow out of this thread. This forum is not for shell scripting. There are better resources online for this sort of thing (try stackoverflow) and over 3 decades worth of books.
     
    Last edited: Jan 19, 2014
    MatteoV likes this.

Share This Page