1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking ads using dnsmasq with an additional hosts file

Discussion in 'Tomato Firmware' started by Macskeeball, Jul 20, 2007.

  1. Macskeeball

    Macskeeball LI Guru Member

    For my home network, I'm trying to make it so that my WRT54Gv2.0 (running Tomato 1.07) will automatically download (once a week) a list of known ad servers in hosts file format, and then effectively block all of the servers listed in that file. It would thus block ads for any device connected to the LAN, without any additional setup, without a lot of manual tinkering, and without using a proxy.

    The automatic downloading part works flawlessly, but the problem I'm running into is that the hosts file doesn't seem to actually be taking effect.

    Under Administration > Scripts > WAN Up, I have the following:
    Code:
    ## Downloads an extensive list of known ad servers
    ## saves it at /etc/host.ads, and schedules weekly updates
    
    # URL of ad server list
    adserverlist="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"
    
    # Ensure that there is a downloaded list available, in case the router recently booted
    wget -q -O /etc/hosts.ads "$adserverlist"; service dnsmasq restart
    
    # Schedule automatic weekly updates
    echo "@weekly wget -q -O /etc/hosts.ads '$adserverlist'; service dnsmasq restart #WeeklyAdServerListUpdate#" >> /var/spool/cron/crontabs/root
    Under Advanced > DHCP/DNS, I have the following for my custom dnsmasq config:
    Code:
    address=/doubleclick.net/127.0.0.1
    address=/intellitxt.com/127.0.0.1
    address=/vibrantmedia.com/127.0.0.1
    address=/kontera.com/127.0.0.1
    address=/tribalfusion.com/127.0.0.1
    address=/adbrite.com/127.0.0.1
    address=/advertising.com/127.0.0.1
    address=/contextweb.com/127.0.0.1
    address=/kontera.com/127.0.0.1
    address=/sageanalyst.net/127.0.0.1
    address=/tacoda.net/127.0.0.1
    addn-hosts=/etc/hosts.ads
    strict-order
    The servers listed in the dnsmasq custom config are blocked, but not those in /etc/hosts.ads

    Any help would be very much appreciated.
     
  2. Macskeeball

    Macskeeball LI Guru Member

    Never mind. I found my answer with a simple search of this forum. Silly me. In case anyone else missed it, go to www.linksysinfo.org/forums/showthread.php?t=53133

    I simply dumped my script and used his, but with the URL I wanted and with @weekly for the time in cron. It works perfectly and is amazing.
     
  3. pgl@yoyo.org

    pgl@yoyo.org LI Guru Member

  4. Macskeeball

    Macskeeball LI Guru Member

    First of all, I'd like to thank you very much for the service you provide via this list, and for somehow finding this thread, creating a new format, and replying.

    Unfortunately, now that I've modified it to use this, I seem to be having the same sort of problem I had at the beginning. The list downloads fine, but does not actually take effect.

    Here's what I have in Administration > Scripts > WAN Up
    Code:
    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
        echo -e "#!/bin/sh\nwget -O - 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext' | grep 'address=' >> /etc/dnsmasq.conf\nlogger DOWNLOADED 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'\nkillall -1 dnsmasq" > /tmp/dlhosts
        chmod 777 /tmp/dlhosts
        /tmp/dlhosts
    fi
    cru a Gethosts "@weekly /tmp/dlhosts"
    As you can see, I'm trying to stay pretty close to mraneri's script, but this time it still doesn't work. I'm not sure what I'm doing wrong, and help from someone would be appreciated.
     
  5. mraneri

    mraneri LI Guru Member

    A few things... You're using >> to append whatever you download to your config file. This is fine the first time. But every time your cron runs, you will be appending the previous list with the downloaded list. Of course, you can't just overwrite, because you would overwrite your config file.

    What I would do is something like: (I haven't tried these statement, you may need to adjust syntax if I screw them up.)

    echo -s "--conf-file=\etc\adblock.conf" >> /etc/dnsmasq.conf

    Do this within the IF statement, so it only executes once per reboot. Then take one of the >'s away and download the file into adblock.conf (overwriting each time). DNSMasq will read its regular config file, and read the extra file as well.

    The killall -1 dnsmasq sends "SIGHUP" to dnsmasq, which causes it to reload the HOSTS file, not re-read it's configuration. I don't think there's an option for that. You will actually have to kill dnsmasq completely and restart it. To do that, inside that really long echo line, change killall -1 dnsmasq to

    killall dnsmasq\ndnsmasq

    That SHOULD kill and restart it... It should re-read the config file, and everything SHOULD work.

    Note, I haven't tried ANY of the stuff I discussed in this e-mail. So there may be typos, but you seem like you know enough about scripting and what I'm talking about to make it work. Good luck.

    Also, maybe someone can send a link to why blocking this way may be better than blocking with the straight HOSTS file? Just curious. (not questioning it, just curious to learn more...)

    Good luck, and post back with results.

    - Mike
     
  6. yaqui

    yaqui LI Guru Member

    He is saying in your other thread : "As I understand it, blocking by domain name is more efficient than blocking by hosts, because it also covers subdomains."

    In blocking in hosts it includes the subdomains. At least thats how I always understood and seems to work that way. (ie. try pinging a site's subdomain from a top domain name in the list)

    I don't see any reason why dnsmasq would be better ??

    Plus mvps's list still looks alot more complete !
     
  7. Macskeeball

    Macskeeball LI Guru Member

    Here's where I read it: http://pgl.yoyo.org/adservers/index.php#othernameservers

    Perhaps ours' is a special case, because dnsmasq uses the host files (including /etc/hosts), and maybe in the way it does that it makes them apply to subdomains as well, while normally a /etc/hosts wouldn't do that? I'm just making a guess here, as I haven't yet tried any of that.
     
  8. yaqui

    yaqui LI Guru Member

    Your right... I see a bunch of these in mvps's list:
    127.0.0.1 doubleclick.net #[McAfee.Cookie-Doubleclick]
    127.0.0.1 ad.doubleclick.net #[MVPS.Criteria]
    127.0.0.1 ad2.doubleclick.net #[Panda.Spyware:Cookie/Doubleclick]
    127.0.0.1 ad.3ad.doubleclick.net
    127.0.0.1 ad.3au.doubleclick.net
     
  9. Macskeeball

    Macskeeball LI Guru Member

    You posted while I was in the middle of posting. Good ol' Google Notifier. At any rate, I was basically going to post that I tested by comparing the effects of modifying the /etc/hosts file on my Mac (connected via dial-up so that I'd only be using the Mac's hosts file and not the router's) with the same changes to the router. Both hosts files were basically empty for the purposes of testing. The result in both cases was that http://yahoo.com was blocked, but not mail.yahoo.com.

    Now for the next step. I verified that dnsmasq does have a --conf-file flag available, but I'm unsure whether or not it's able to use multiple config files or if that's just meant to be used as flag when starting dnsmasq from the command line. My hand is starting to hurt a bit right now from too much computer use, so if someone else would test that I would appreciate it. Also, in mraneri's post, he accidentally used Windows-style path separators (\) instead of *nix-style separators (/).
     
  10. mraneri

    mraneri LI Guru Member

    the man page says it's for either. put it on the command line, OR, put it in a config file to include another config file.

    There's a man page linked on this dnsmasq homepage...
    http://www.thekelleys.org.uk/dnsmasq/doc.html

    Yup, I did use the wrong style path separators... (I said I didn't check it out!!!) Sorry.
     
  11. yaqui

    yaqui LI Guru Member

    So in theory this should work?
    Code:
    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
        echo -s "--conf-file=/etc/adblock.conf" >> /etc/dnsmasq.conf
        echo -e "#!/bin/sh\nwget -O - 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext' | grep 'address=' > /etc/adblock.conf\nlogger DOWNLOADED 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'\nkillall dnsmasq\ndnsmasq" > /tmp/dlhosts
        chmod 777 /tmp/dlhosts
        /tmp/dlhosts
    fi
    cru a Gethosts "@weekly /tmp/dlhosts"
     
  12. Macskeeball

    Macskeeball LI Guru Member

    Actually, the -- prefix before dnsmasq configuration options is just for when using as a flag when launching dnsmasq from the command line or a script, as in "dnsmasq --option[=value]" For the configuration file, there is no -- prefix.

    Also, if that script works (which it should) there's no need to use a script to add a line to the custom dnsmasq configuration. Just go to the web based config and add it to the Dnsmasq custom configuration field in Advanced > DHCP / DNS.

    Now to give that script a try.
     
  13. yaqui

    yaqui LI Guru Member

    Edit: nevermind... curious if this is working though.
     
  14. Macskeeball

    Macskeeball LI Guru Member

    No, I'm having some problems with it. I am working on it, however.
     
  15. Macskeeball

    Macskeeball LI Guru Member

    This works.

    Code:
    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
        echo -e "conf-file=/etc/adblock.conf" >> /etc/dnsmasq.conf
        echo -e "#!/bin/sh\nwget -O - 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext' | grep 'address=' > /etc/adblock.conf\nlogger DOWNLOADED 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'\nkillall dnsmasq\ndnsmasq" > /tmp/dlhosts
        chmod 777 /tmp/dlhosts
        /tmp/dlhosts
    fi
    cru a Gethosts "@weekly /tmp/dlhosts"
    I tried putting "conf-file=/etc/adblock.conf" in the dnsmasq custom configuration field, but that resulted in the file being empty on the first run (from a reboot). Running the script manually after that populated the file however.

    The one problem I have with the script in this code is that the "conf-file=/etc/adblock.conf" added by the script goes away in the event that the user modifies the custom dnsmasq configuration, or a "service dnsmasq restart" happens.
     
  16. Macskeeball

    Macskeeball LI Guru Member

    I thought I should simplify the code a little bit, so here is the result (which I successfully tested).

    Code:
    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
        blocklisturl="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
        echo -e "conf-file=/etc/adblock.conf" >> /etc/dnsmasq.conf
        echo -e "#!/bin/sh\nwget -O /etc/adblock.conf '$blocklisturl'\nlogger DOWNLOADED '$blocklisturl'\nkillall dnsmasq\ndnsmasq" > /tmp/dlhosts
        chmod 777 /tmp/dlhosts
        /tmp/dlhosts
    fi
    cru a Gethosts "@weekly /tmp/dlhosts"
    
    What's the -e flag on the echo commands for, anyway?
     
  17. mraneri

    mraneri LI Guru Member

    I don't question your result, but I don't understand why that would be. Maybe put an additional delay in your script. (put a sleep 60 before the wget)
     
  18. mraneri

    mraneri LI Guru Member

    --help is your friend...

    type echo --help, and you'll see the command line options for echo...

    according to the help, that one converts the \n's the new lines. Didn't try without it, but figured it was the right way to code it.

    And yes, it isn't necessary to pip the whole thing through GREP. I am doing that to filter the MVPS list, which has a fair bit of comments in it.


    While we're all learning from each other, other than the obvious, what does @weekly do? execute every 168 hours? or at midnight every sunday or something? I'm always used to seeing the * * * * *...
     
  19. Macskeeball

    Macskeeball LI Guru Member

    Heh, echo --help on my Mac simply echoes "--help" but on my router it does something different. I guess that's just a difference between BSD echo and Linux echo.

    This is where I found out about @weekly, among other special strings: www.ss64.com/osx/crontab.html I'm not sure exactly what it really does behind the scenes, and now that I've done a little more searching I'm not as sure that it's available in the cron used by Tomato (not saying it isn't). I suppose temporarily setting an @daily job that outputs a text file or updates a Twitter status (via wget), etc. would be a good way to test that. Perhaps such a job could output the current time as well.
     
  20. mraneri

    mraneri LI Guru Member

    Or, of course, set a job that calls Logger with something like "YUP, IT WORKED" would do the trick too. By now, you've certainly figured out what logger does!!

    He he he... Let me know if @weekly or @daily work!
     
  21. Macskeeball

    Macskeeball LI Guru Member

    Better yet, I'll try @hourly.

    Edit: I just looked at the page where I found out about @weekly, and another thing it says is that @midnight is the same as @daily. That makes it sound to me like @weekly just runs every Sunday at midnight. Crud, I guess we can't all just use @weekly then.
     
  22. Macskeeball

    Macskeeball LI Guru Member

    I tried putting "sleep 60" before the wget line, but it made no difference.
     
  23. Macskeeball

    Macskeeball LI Guru Member

    From the looks of it, @eekly and such are not supported by the cron variant used by Tomato.
     
  24. yaqui

    yaqui LI Guru Member

    Hey Mac.. is this similar to what you are getting in your log file?
    Code:
    Jul 23 17:00:01 lynxpark cron.notice crond[135]: USER root pid 261 cmd /tmp/dlhosts #Gethosts# 
    Jul 23 17:00:01 lynxpark cron.notice crond[135]: USER root pid 263 cmd rcheck --cron #rcheck# 
    Jul 23 17:00:02 lynxpark user.notice root: DOWNLOADED http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0& mimetype=plaintext
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[169]: exiting on receipt of SIGTERM
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: started, version 2.39 cachesize 150
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: compile time options: no-IPv6 GNU-getopt no-RTC no-ISC-leasefile no-DBus no-I18N no-TFTP
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: DHCP, IP range 192.168.10.100 -- 192.168.10.149, lease time 2d12h
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: reading /etc/resolv.dnsmasq
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: using nameserver #.#.#.##53
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: using nameserver #.#.#.##53
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: read /etc/hosts - 0 addresses
    Jul 23 17:00:02 lynxpark daemon.info dnsmasq[270]: read /etc/hosts.dnsmasq - 1 addresses
    I wasn't sure about the hosts.dnsmasq saying just 1 address?? normal?
     
  25. yaqui

    yaqui LI Guru Member

    I don't think mine is working... something is wrong, ad sites show up !!
     
  26. mraneri

    mraneri LI Guru Member

    You're not using HOSTS blocking here... You're blocking with dnsmasq options, so this is what I would expect.

    As to why it may or may not be working, you must make sure that "Use Internal Caching DNS Forwarder" is CHECKED, or when your PC gets it's IP address from the router, the router will supply your ISP's DNS server, and dnsmasq will not be used for name resolution.

    I personally still use the MVPS HOSTS file blocking as in the referenced thread, and it is still working like a champ. I cannot comment on the dnsmasq style blocking which is the subject of this particular thread. (Other than helping getting people's scripts functional.)

    There may be an option somewhere which needs to be set right. Also, telnet into the router, and make sure what you expect to be in the dnsmasq config files is actually there..

    - Mike
     
  27. Macskeeball

    Macskeeball LI Guru Member

    I figured out how to make this new method we've been working on together work without any problems. Remember, the benefit of this method over the hosts file method is that it allows for a shorter (and thus less computationally intensive) and yet theoretically more effective block list to be used.

    IMPORTANT: It is recommended that you restart your router after doing the following, especially if you have been using previous solutions (either the hosts file method or previous versions of the adblock.conf method). You may need to delete scripts involved in the previous solutions if you used them.

    1. In Advanced > DHCP / DNS, make sure that the "Dnsmasq Custom Configuration" insert the following and save.
      Code:
      conf-file=/etc/adblock.conf
    2. In Administration > Scripts > Init, insert the following and save.
      Code:
      sleep 5
      logger Executing init script
      touch /etc/adblock.conf
      chmod 666 /etc/adblock.conf
    3. In Administration > Scripts > WAN Up, insert the following and save. Please change "<your interval here>" in the "cru a UpdateAdblock..." line near the bottom so that we do not all update at the same time. The syntax is: minute hour day month week. An asterick for any of these means "every."
      Code:
      logger WAN UP Script Executing after 60sec
      sleep 15
      ADLIST="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
      CONN=`ping -c 1 $ADLIST | grep "208.69"`
      sleep 5
      if [ ! -x /tmp/update-adblock.sh ] ; then
      cat > /tmp/update-adblock.sh <<EOF
      #!/bin/sh
      if [ -n "$CONN" ] ; then 
         wget -O /etc/adblock.conf 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
         else 
         logger Adblock list download link UNREACHABLE
      fi
      if grep -q address /etc/adblock.conf ; then
         logger Adblock file population SUCCESS
         service dnsmasq restart
         else
         logger Adblock file population FAILURE
      fi
      EOF
        chmod +x /tmp/update-adblock.sh
        /tmp/update-adblock.sh
        cru a UpdateAdblock "<your interval here> /tmp/update-adblock.sh"
        else
        logger update-adblock.sh already exists and is executable will not re-run
      fi
      
      ## END AUTO-UPDATING AD BLOCK
    4. Reboot your router using the Reboot link in the side bar.
     
  28. yaqui

    yaqui LI Guru Member

    Couple things... for some reason I can't get the list to download when using the blocklistURL variable, I have to just use the url by itself in the echo line... have no idea why !! my busybox doesn't like that for some odd reason, even when i try doing the commands in telnet I got some error, i can't remember

    And the other thing... why are you doing the test for file size? whatever the result it seems as if the 'if then' runs anyways, i'm not seeing the purpose of that
     
  29. Macskeeball

    Macskeeball LI Guru Member

    What version of Tomato are you using? I'm using 1.07 and it works for me.

    The reason we can just do normal cron jobs and have to use the script fields in the web config is because the shell script and cron jobs are stored in RAM. RAM is of course erased on startup, so we have to check to see if the shell script exists. See the first thing mraneri mentioned in his first post in this thread.
     
  30. yaqui

    yaqui LI Guru Member

    But $? just means if the last command's exit status successful or not (ie.was 1 or 0). The if..then test is not testing for the existance of the file ! The command "test -s /tmp/update-adblock.sh" is always successful whether the file is empty or not !!

    Correct me if I am wrong but that is how I understand it.

    Edit: for example I make sure the file /tmp/update-adblock.sh does not exist in telnet, and I run the command "test -s /tmp/update-adblock.sh" there is no error message.... doesn't that mean that it ran and exited successfully ????
     
  31. Macskeeball

    Macskeeball LI Guru Member

    You may be right on that. I was just going with it because I was following the original script made by mraneri in the other thread.
     
  32. yaqui

    yaqui LI Guru Member

    Not sure if you saw my edit to the post but: for example I make sure the file /tmp/update-adblock.sh does not exist (using telnet), and I run the command "test -s /tmp/update-adblock.sh" there is no error message.... doesn't that mean that it ran and exited successfully ????
     
  33. Macskeeball

    Macskeeball LI Guru Member

    Yeah, I think you're right. I'll edit the post with the supposedly flawless script (uurgh).
     
  34. yaqui

    yaqui LI Guru Member

    I also saw in another script somewhere where it checked to see if downloading was for sure successful... it compared the file, i'll have to search for it again, i need that in this script because sometimes it seems like it doesn't grab the data the first time
     
  35. Macskeeball

    Macskeeball LI Guru Member

    Did you do the Init script part of this? There are three places where you need to put stuff. http://www.linksysinfo.org/forums/showpost.php?p=305837&postcount=27
     
  36. yaqui

    yaqui LI Guru Member

    Yes but regardless... it seems as if the 'test' line exit status is always successful, I really don't think that is checking for anything, the "if.. then" always executes after it :/
     
  37. Macskeeball

    Macskeeball LI Guru Member

    I get that, but I don't know how to properly check that the script exists and is executable. I need help with that.
     
  38. yaqui

    yaqui LI Guru Member

    Go in telnet and type "test -s /tmp/anyrandomname.sh" You never get any kind of error for that... I have no idea why, in theory it should say "anyrandomname.sh" does not exist.
     
  39. yaqui

    yaqui LI Guru Member

    Ah, I know I am looking for that other script, I saw one written up with a really good way to check for the file and that it was downloaded properly.
     
  40. yaqui

    yaqui LI Guru Member

    Aggghh this doesn't make any sense... look at this site: http://www.ss64.com/bash/test.html

    to test if file exists you use -e but in telnet it still doesn't send an error or report that it doesn't exist !!

    test -e anyname.sh - can you see it's output somehow ??
     
  41. Macskeeball

    Macskeeball LI Guru Member

  42. yaqui

    yaqui LI Guru Member

    It should have worked the other way... if you do echo $? after you can see it returns 1 when a file doesn't exist, but does that mean that command 'test' itself did not exit successful?? maybe not then, i'm confused
     
  43. Macskeeball

    Macskeeball LI Guru Member

    Don't worry about it, I think we've found something that works. This is really working well for me now. Give it a try, and make sure you've filled out the three parts.

    Ideally, we'd only have two parts. The simpler it is to get going the less likely people will make a mistake setting it up.
     
  44. yaqui

    yaqui LI Guru Member

    Filled in everything and rebooted... went into telnet and see that adblock.conf did not populate :( So I put this in init:
    Code:
    sleep 5
    touch /etc/adblock.conf
    touch /tmp/update-adblock.sh
    chmod 666 /etc/adblock.conf
    chmod 777 /tmp/update-adblock.sh
    adblock.conf still empty
     
  45. yaqui

    yaqui LI Guru Member

    Using version 1.07, For some reason when I reboot it won't download but I go in manually with telnet and run update-adblock.sh and it works.

    I guess I just will have to do it manually that way till I figure out what is going on
     
  46. Macskeeball

    Macskeeball LI Guru Member

    I think what we need is a while loop in the WAN Up script that repeats until /etc/adblock.conf exists and has downloaded properly.

    http://www.tech-recipes.com/bourne_shell_scripting_tips316.html

    Be careful with this. I threw my router into an infinite loop on my first (and currently only) try. I ended up having to pull the WAN cable out of the router and force a reboot by unplugging and replugging.

    Edit: Yikes! Everything I try seems to cause big trouble for my router, and I even had to do a hard reset once. Fortunately I had a backup config file. I'mgoing to bed. I've had enough breaking things for now.
     
  47. mstombs

    mstombs Network Guru Member

    Bash script tips - no change in functionality - hope this is useful

    I assume the single long echo line is to avoid multiple echo statements? - but you can also use 'Here documents' to just include the file formatted as you like. Here is the same script using such a method.

    Code:
    #!/bin/sh -x
    logger WAN UP Script Executing
    sleep 5
    
    ## BEGIN AUTO-UPDATING AD BLOCK
    SHF=/tmp/update-adblock.sh
    
    if [ ! -x $SHF ] ; then
      ADS="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
      cat > $SHF <<EOF
    #!/bin/sh -x
    wget -O /etc/adblock.conf "$ADS"
    logger DOWNLOADED "$ADS"
    service dnsmasq restart
    EOF
      chmod +x $SHF
      $SHF
    fi
    #cru a UpdateAdblock "28 6 2 * 3 $SHF" CHANGE THIS
    
    ## END AUTO-UPDATING AD BLOCK
    Note if multiple documents created in same file use EOF2 etc, these end of file markers must be unique.

    Note the first line only needed because I saved the contents as an executable test.sh to test with echoing of the commands, outside of the web interface. Also the cru statement is commented out. Referring to the script file by variable is useful if you want to move the file - to jffs for example rather than create it every time.
     
  48. mraneri

    mraneri LI Guru Member

    Can someone who implemented this method do a ps and let me know the number in the 3rd column on the dnsmasq line? Curious what the memory usage is using this method.

    Also, mstombs, thanks for the tip on Here Documents. Didn't know about them. They look quite useful. Here's a page I found on them for those that want to learn more. (OOPS, Bad reference, see next post.)

    Also, instead of a while loop, try a longer sleep time. I needed at least 20 seconds before I had good reliable downloading. Try 60 to start, and see if it helps.

    - Mike
     
  49. mstombs

    mstombs Network Guru Member

  50. mraneri

    mraneri LI Guru Member

    OOPS! Yeah, what he said!!!

    (Sorry)
     
  51. yaqui

    yaqui LI Guru Member

    I want some type of connection and file population error-checking, can you guys look at this and tell me if you think it will work, or please correct it.. i'm still new to this scripting (if you can't tell lol)

    Code:
    #!/bin/sh -x
    sleep 30
    logger WAN UP Script Executing after SLEEP 30sec
    
    ## BEGIN AUTO-UPDATING AD BLOCK
    
    SHF=/tmp/update-adblock.sh
    
    if [ ! -x $SHF ] ; then
      ADS="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
      cat > $SHF <<EOF
    #!/bin/sh
    conn=`ping -c 1 $ADS | grep "208.69.32.130"`
    if [ -n "$conn" ]; then 
       logger ping to server list OK proceding to download
       wget -O /etc/adblock.conf $ADS 
        else
       logger connection to server list DOWN cannot download
    fi
    if grep -q address /etc/adblock.conf ; then
       logger adblock file population SUCCESS from '$ADS'
       service dnsmasq restart
        else
       logger adblock file population FAILURE
    fi
    EOF
      chmod +x $SHF
      $SHF
      else
      logger update-adblock.sh already exists and is executable
    fi
    
    #cru a UpdateAdblock "28 6 2 * 3 $SHF"
    
    ## END AUTO-UPDATING AD BLOCK
     
  52. Macskeeball

    Macskeeball LI Guru Member

    It says 652. This is with Tomato 1.07 running on a WRT54Gv2.0.

    I've made some changes to mstombs' script, because he had commented out the line that scheduled the automatic updates and because using variables that way was actually causing problems for us earlier, for some reason (especially for yaqui). Note that this was made before I saw yaqui's post above, so this is not in response to yaqu's script.

    Code:
    logger WAN Up Script Executing
    sleep 5
    
    ## BEGIN AUTO-UPDATING AD BLOCK
    
    if [ ! -x /tmp/update_adblock.sh ] ; then
        cat > /tmp/update_adblock.sh <<EOF
    #!/bin/sh
    wget -O /etc/adblock.conf "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
    logger DOWNLOADED "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
    service dnsmasq restart
    EOF
        chmod +x /tmp/update_adblock.sh
        /tmp/update_adblock.sh
    fi
    
    cru a UpdateAdblock "28 6 2 * 3 /tmp/update_adblock.sh"
    
    ## END AUTO-UPDATING AD BLOCK
     
  53. mraneri

    mraneri LI Guru Member

    Ok, Thanks.
     
  54. yaqui

    yaqui LI Guru Member

    Well with this I get a syntax error unexpected "<" when i try to run it in telnet. If I let it run on it's own at reboot it just hangs at the wan up script !!
    Code:
    #!/bin/sh -x
    sleep 30
    logger WAN UP Script Executing after SLEEP 30sec
    
    ## BEGIN AUTO-UPDATING ADBLOCK
    
    if [ ! -x /tmp/update-adblock.sh ] ; then
      cat > /tmp/update-adblock.sh <<EOF
    #!/bin/sh
    conn=`ping -c 1 http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext | grep "208.69.32.130"`
    if [ -n "$conn" ]; then 
       logger connection to server list OK proceding to download
       wget -O /etc/adblock.conf http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext
        else
       logger connection to server list DOWN cannot download
    fi
    if grep -q address /etc/adblock.conf ; then
       logger adblock file population SUCCESS 
       service dnsmasq restart
        else
       logger adblock file population FAILURE
    fi
    EOF
      chmod +x /tmp/update-adblock.sh
      /tmp/update-adblock.sh
      else
      logger update-adblock.sh already exists and is executable
    fi
    
    cru a UpdateAdblock "<your interval here> /tmp/update-adblock.sh"
    
    ## END AUTO-UPDATING ADBLOCK
    
     
  55. yaqui

    yaqui LI Guru Member

    Well this is my last attempt:
    Code:
    #!/bin/sh -x
    sleep 60
    logger WAN UP Script Executing after 60sec
    
    ## BEGIN AUTO-UPDATING ADBLOCK
    if [ ! -f /etc/adblock.conf ] ; then
    touch /etc/adblock.conf
    else
    logger adblock.conf existence confirmed
    fi 
    if [ ! -x /tmp/update-adblock.sh ] ; then 
    cat > /tmp/update-adblock.sh <<EOF
    #!/bin/sh
    ADLIST="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
    CONN=`ping -c 1 $ADLIST | grep "208.69.32."`
    sleep 15
    if [ -n $CONN ]; then 
       wget -O /etc/adblock.conf $ADLIST
       else
       logger connection to server list DOWN cannot download 
    fi
    sleep 15
    if grep -q address /etc/adblock.conf ; then
       logger adblock file population and download SUCCESS 
       else
       logger adblock file population FAILURE
    fi
    sleep 15
    service dnsmasq restart
    EOF
      chmod +x /tmp/update-adblock.sh
      /tmp/update-adblock.sh
      else
      logger update-adblock.sh already exists and is executable
    fi
    cru a UpdateAdblock "<your interval here> /tmp/update-adblock.sh"
    
    ## END AUTO-UPDATING ADBLOCK
    This is my result everytime on reboot :
    Code:
    Jul 24 14:36:11 lynxpark user.notice root: WAN UP Script Executing after 60sec
    Jul 24 14:36:42 lynxpark user.notice root: adblock file population FAILURE
    Jul 24 14:36:57 lynxpark daemon.info dnsmasq[107]: exiting on receipt of SIGTERM
    Jul 24 14:36:57 lynxpark daemon.info dnsmasq[271]: started, version 2.39 cachesize 150
    Jul 24 14:36:57 lynxpark daemon.info dnsmasq[271]: compile time options: no-IPv6 GNU-getopt no-RTC no-ISC-leasefile no-DBus no-I18N no-TFTP
    Jul 24 14:36:57 lynxpark daemon.info dnsmasq[271]: DHCP, IP range 192.168.10.100 -- 192.168.10.149, lease time 2d12h
    
     
  56. mstombs

    mstombs Network Guru Member

    If you put your code into a file and run it you get this output, the ping and grep commands are wrong!

    Code:
    Tomato  v1.07.1039
    
    
    BusyBox v1.2.2 (2007.05.06-15:48+0000) Built-in shell (ash)
    Enter 'help' for a list of built-in commands.
    
    # cd /var/tmp
    # ./test.sh
    + sleep 60
    + logger WAN UP Script Executing after 60sec
    + [ ! -x /etc/adblock.conf ]
    + touch /etc/adblock.conf
    + [ ! -x /tmp/update-adblock.sh ]
    + cat
    + ping -c 1
    + grep 208.69.32.
    BusyBox v1.2.2 (2007.05.06-15:48+0000) multi-call binary
    
    Usage: ping [OPTION]... host
    
    Send ICMP ECHO_REQUEST packets to network hosts.
    
    Options:
            -c COUNT        Send only COUNT pings
            -s SIZE         Send SIZE data bytes in packets (default=56)
            -q              Quiet mode, only displays output at start
                            and when finished
    
    + chmod +x /tmp/update-adblock.sh
    + /tmp/update-adblock.sh
    BusyBox v1.2.2 (2007.05.06-15:48+0000) multi-call binary
    
    Usage: wget [-c|--continue] [-q|--quiet] [-O|--output-document file]
                    [--header 'header: value'] [-Y|--proxy on/off] [-P DIR] url
    
    wget retrieves files via HTTP or FTP
    
    Options:
            -c      continue retrieval of aborted transfers
            -q      quiet mode - do not print
            -P      Set directory prefix to DIR
            -O      save to filename ('-' for stdout)
            -Y      use proxy ('on' or 'off')
    
    
    Done.
    
    I'll now see if I can fix it...
     
  57. Macskeeball

    Macskeeball LI Guru Member

    This is just something to try. I'm not sure if it will fix your problem or not. Try temporarily changing the WAN Up scriipt so that it is no different than the one in my earlier post. However, do change the Init script to the following.

    Code:
    touch /etc/adblock.conf
    sleep 5
    logger Init script executed
    
    Then reboot and see if it works. If not, go back to what you had before trying this.
     
  58. yaqui

    yaqui LI Guru Member

    Look what happens BEFORE init runs !!

    Code:
    Jan  1 00:00:08 lynxpark daemon.crit dnsmasq[83]: cannot read /etc/adblock.conf: No such file or directory
    Jan  1 00:00:08 lynxpark daemon.crit dnsmasq[83]: FAILED to start up
    Jan  1 00:00:09 lynxpark user.notice root: Executing Init Script after 5sec
    Jan  1 00:00:09 lynxpark user.notice root: creation of adblock.conf SUCCESS
    I have my init set up as you suggest Mac
     
  59. yaqui

    yaqui LI Guru Member

    But if you try this at command line :
    Code:
    ping -c 1 $ADLIST | grep "208.69.32."
    It works... of course after setting the ADLIST=
     
  60. Macskeeball

    Macskeeball LI Guru Member

    Ok, then try changing the init script to this
    Code:
    logger Init script executing
    touch /etc/adblock.conf
    sleep 5
    service dnsmasq start
    
    How's that?
     
  61. yaqui

    yaqui LI Guru Member

    What is to stop dnsmasq from starting before init ? Can we put a 'sleep 20' in the dnsmasq configuration screen ??
     
  62. Macskeeball

    Macskeeball LI Guru Member

    No, because that's a config file and not a script. There's nothing we can do about dnsmasq starting before init. Just try what I put in the last post and see if it works.
     
  63. yaqui

    yaqui LI Guru Member

    I'm not even seeing 'Init script executing' now. I think you need the sleep 5 first.
     
  64. mstombs

    mstombs Network Guru Member

    @yaqui
    There was a bug in my script the adlist variable not being properly asigned because the "&" needed to be protected with "\&", explains why Macskeeball script works without variable and why your wget doesn't work. I don't think the ping works either but...

    Edit neater just to protect string with " " when used then \& not needed
     
  65. Macskeeball

    Macskeeball LI Guru Member

    Code:
    sleep 5
    logger Init script executing
    touch /etc/adblock.conf
    sleep 5
    service dnsmasq start
     
  66. Macskeeball

    Macskeeball LI Guru Member

    Actually, mine works without variables because I put the URLs in quotes. Even if you're usinga varible and put the URL in quotes when setting the variable, you still need quotes when you use the variable.
     
  67. yaqui

    yaqui LI Guru Member

    So it should be like this? what about the "?" mark also?

    Code:
    ADLIST="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq\&showintro=0\&mimetype=plaintext"
     
  68. Macskeeball

    Macskeeball LI Guru Member

    No need to have both the quotes and the \ before the &. Also, you need to make sure the URL is escaped when you actually call the variable.
     
  69. yaqui

    yaqui LI Guru Member

    Well this seems to work
    Code:
    lynxpark.net login: root
    Password:
    
    
    Tomato  v1.07.1039
    
    
    BusyBox v1.2.2 (2007.05.06-15:48+0000) Built-in shell (ash)
    Enter 'help' for a list of built-in commands.
    
    # ping -c 1 "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&sho
    wintro=0&mimetype=plaintext" | grep "208.69."
    PING http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0
    &mimetype=plaintext (208.69.32.130): 56 data bytes
    64 bytes from 208.69.32.130: icmp_seq=0 ttl=48 time=108.4 ms
    I will try it in the script now
     
  70. mstombs

    mstombs Network Guru Member

    OK - up wrong tree - yaqui's script doesn't work with the ADLIST variable because it assigned inside the "here document" and the second $ADLIST is null in the .sh file.

    Need to move the ADLIST= above the "cat" command and "\&" is not needed
     
  71. yaqui

    yaqui LI Guru Member

    I tried both ways with variable and without and the way mstombs suggests. Now it always just hangs at 'WAN UP Script Executing after 60sec'
     
  72. mstombs

    mstombs Network Guru Member

    Are you testing by pasting the script into the wanup window - if so you don't need the first #!/bin/sh line.

    If you are editing the file on the router make sure you are using a 'nix' compatible editor such as notepad2 which understands unix line ends
     
  73. yaqui

    yaqui LI Guru Member

    How about this?
    Code:
    sleep 5
    logger Executing Init Script
    killall -hup dnsmasq
    touch /etc/adblock.conf
    service dnsmasq start
     
  74. Macskeeball

    Macskeeball LI Guru Member

    I think -hup just makes dnsmasq reload the hosts files, not the config files which is what we're dealing with. Try that without the killall line and change "service dnsmasq start" to "service dnsmasq restart"

    I'm feel like I'm kind of losing track with what's going on. Could you post your dnsmasq custom config, init script, and wan up script?
     
  75. mstombs

    mstombs Network Guru Member

    The ping command is not reliable for me, sometimes works, sometimes doesn't very different dns for the subdirs

    Code:
    #  ping -c 1 pgl.yoyo.org
    PING pgl.yoyo.org (193.110.91.2): 56 data bytes
    64 bytes from 193.110.91.2: icmp_seq=0 ttl=50 time=27.9 ms
    
    --- pgl.yoyo.org ping statistics ---
    1 packets transmitted, 1 packets received, 0% packet loss
    round-trip min/avg/max = 27.9/27.9/27.9 ms
    #  ping -c 1 pgl.yoyo.org/adservers
    PING pgl.yoyo.org/adservers (212.68.205.66): 56 data bytes
    
    --- pgl.yoyo.org/adservers ping statistics ---
    1 packets transmitted, 0 packets received, 100% packet loss
     
  76. yaqui

    yaqui LI Guru Member

    Code:
    Jan  1 00:00:08 lynxpark daemon.crit dnsmasq[83]: cannot read /etc/adblock.conf: No such file or directory
    Jan  1 00:00:08 lynxpark daemon.crit dnsmasq[83]: FAILED to start up
    Jan  1 00:00:09 lynxpark user.notice root: Executing Init Script
    Jan  1 00:00:12 lynxpark cron.notice crond[104]: crond 2.3.2 dillon, started, log level 8 
    Jan  1 00:00:14 lynxpark daemon.info dnsmasq[118]: started, version 2.39 cachesize 150
    Jan  1 00:00:14 lynxpark daemon.info dnsmasq[118]: compile time options: no-IPv6 GNU-getopt no-RTC no-ISC-leasefile no-DBus no-I18N no-TFTP
    Jan  1 00:00:14 lynxpark daemon.info dnsmasq[118]: DHCP, IP range 192.168.10.100 -- 192.168.10.149, lease time 2d12h
    Jan  1 00:00:14 lynxpark daemon.info dnsmasq[118]: reading /etc/resolv.dnsmasq
    
     
  77. yaqui

    yaqui LI Guru Member

    The only way I can get this to work partially like I want is this:

    In WAN UP:
    Code:
    logger WAN UP Script Executing and waiting 60sec
    sleep 60
    test -x /tmp/update-adblock.sh
    if [ $? == 1 ] ; then
        echo -e "#!/bin/sh\nwget -O /etc/adblock.conf 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'" > /tmp/update-adblock.sh
        echo -e "logger Ad Server list download SUCCESS" >> /tmp/update-adblock.sh
        echo -e "sleep 15" >> /tmp/update-adblock.sh
        echo -e "if grep -q address /etc/adblock.conf" >> /tmp/update-adblock.sh
        echo -e "then" >> /tmp/update-adblock.sh
        echo -e "logger Adblock file population SUCCESS" >> /tmp/update-adblock.sh
        echo -e "else" >> /tmp/update-adblock.sh
        echo -e "logger Adblock file population FAILURE" >> /tmp/update-adblock.sh
        echo -e "fi" >> /tmp/update-adblock.sh
        echo -e "service dnsmasq restart" >> /tmp/update-adblock.sh
        chmod +x /tmp/update-adblock.sh
        /tmp/update-adblock.sh
    fi
    cru a UpdateAdblock "<your time interval here> /tmp/update-adblock.sh"
    
    In init:
    Code:
    sleep 5
    logger Executing Init Script
    touch /etc/adblock.conf
    chmod 666 /etc/adblock.conf
    service dnsmasq restart
    
    Here is the log ! yey
    Code:
    Dec 31 18:01:19 lynxpark user.notice root: WAN UP Script Executing and waiting 60sec
    Dec 31 18:01:20 lynxpark daemon.info dnsmasq[149]: DHCPREQUEST(br0) 192.168.10.118 00:18:71:54:37:96 
    Dec 31 18:01:20 lynxpark daemon.info dnsmasq[149]: DHCPACK(br0) 192.168.10.118 00:18:71:54:37:96 HP543795
    Jul 24 17:47:57 lynxpark user.info ntpc[183]: Time Updated: Tue, 24 Jul 2007 17:47:57 -0500 [+1185317194s]
    Jul 24 17:48:35 lynxpark cron.warn crond[105]: time disparity of 19755287 minutes detected 
    Jul 24 17:49:01 lynxpark user.notice root: Ad Server list download SUCCESS
    Jul 24 17:49:16 lynxpark user.notice root: Adblock file population SUCCESS
    
    Now if I could somehow just incorporate the ping test... :(
     
  78. Macskeeball

    Macskeeball LI Guru Member

    Okay, I basically just reformatted your WAN Up script to be more like what we've been working on. Not in terms of code, but in terms of appearance. Tell me if this works.

    Code:
    logger WAN UP Script Executing and waiting 60sec
    sleep 60
    if [ -x /tmp/update_adblock.sh ]; then
        cat > /tmp/update_adblock.sh <<EOF
    #!/bin/sh
    wget -O /etc/adblock.conf "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
    logger Ad Server list download SUCCESS
    sleep 15
    if grep -q address /etc/adblock.conf
    then
    logger Adblock file population SUCCESS
    else
    logger Adblock file population FAILURE
    fi
    service dnsmasq restart
    EOF
        chmod +x /tmp/update-adblock.sh
        /tmp/update-adblock.sh
    fi
    cru a UpdateAdblock "<your time interval here> /tmp/update-adblock.sh"
     
  79. mstombs

    mstombs Network Guru Member

    I've had a go should wait up to 5 mins for a ping to get to the primary host before w getting:-

    Code:
    SHF=/tmp/update-adblock.sh
    if [ ! -x $SHF ] ; then
      logger WAN UP Script Executing
      WT=10
      CNF=/etc/adblock.conf
      TGT=pgl.yoyo.org
      ADS="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
      x=$WT
      while ! ping -c 1 "$TGT" && [ $x -le 300 ]; do
        logger WAN UP Script waiting $x secs
        sleep $WT
        x=$(expr $x + $WT)
      done
      cat > $SHF <<EOF
    #!/bin/sh
    wget -O $CNF "$ADS"
    sleep 15
    if grep -q address $CNF ; then
      logger Adblock file population SUCCESS
    else
      logger Adblock file population FAILURE
    fi
    service dnsmasq restart
    EOF
      chmod +x $SHF
      $SHF
    fi
    cru a UpdateAdblock "28 7 2 * 3 $SHF"
    #cru a UpdateAdblock "<your time interval here> $SHF"
    ## END AUTO-UPDATING AD BLOCK
    but of course sod's law applies and on reboot it got through first time didn't wait at all

    Code:
    Jul 25 01:09:05 wrt54gs user.notice root: WAN UP Script Executing
    Jul 25 01:09:06 wrt54gs user.info rcheck[255]: Activating rule 1
    Jul 25 01:09:06 wrt54gs user.info rcheck[255]: Activating rule 2
    Jul 25 01:09:21 wrt54gs user.notice root: Adblock file population SUCCESS
    Jul 25 01:09:21 wrt54gs daemon.info dnsmasq[123]: exiting on receipt of SIGTERM
     
  80. yaqui

    yaqui LI Guru Member

    I'm not sure if that works but looks like it does... I've been going back to my previous script now :)

    This works !! But something odd I will explain below:
    Code:
    logger WAN UP Script Executing after 60sec
    sleep 60
    ADLIST="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
    CONN=`ping -c 1 $ADLIST | grep "208.69"`
    test -x /tmp/update-adblock.sh
    if [ $? == 1 ] ; then 
    cat > /tmp/update-adblock.sh <<EOF
    #!/bin/sh
    if [ -n "$CONN" ] ; then 
       wget -O /etc/adblock.conf $ADLIST
       else 
       logger Adblock list site UNREACHABLE
    fi
    sleep 10
    if grep -q address /etc/adblock.conf ; then
       logger Adblock file population SUCCESS
       else
       logger Adblock file population FAILURE
    fi
    sleep 10
    service dnsmasq restart
    EOF
      chmod +x /tmp/update-adblock.sh
      /tmp/update-adblock.sh
      else
      logger update-adblock.sh already exists and is executable
    fi
    cru a UpdateAdblock "<your interval here> /tmp/update-adblock.sh"
    Now the problem is in update-adblock.sh... I see this in telnet, I don't think the grep is working ??
    Code:
    #!/bin/sh
    if [ -n "PING http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&sh
    64 bytes from 208.69.32.130: icmp_seq=0 ttl=48 time=100.4 ms" ] ; then
       wget -O /etc/adblock.conf http://pgl.yoyo.org/adservers/serverlist.php?hostfo
       else
       logger Adblock list site UNREACHABLE
    fi
    sleep 10
    if grep -q address /etc/adblock.conf ; then
       logger Adblock file population SUCCESS
       else
       logger Adblock file population FAILURE
    fi
    sleep 10
    service dnsmasq restart
    Instead of that whole line being in the null test, i thought it should just be part of the ip numbers (grepped).... not sure how to fix that, but the script is working and populating adblock.conf !!
     
  81. Macskeeball

    Macskeeball LI Guru Member

    I don't think you really need to worry about pinging. The grep should be all you need.
     
  82. yaqui

    yaqui LI Guru Member

    I didn't see your post when I was posting... that looks great but one thing if you ping "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext" you get the same first 3 ip numbers every time, I just kinda wanted to do the ping test on that because that is exactly where we are downloading from.

    If they change the link I want my script to bark at me in the log, not just have nothing happen or just saying "file population FAILED" I wish I could get it to say "download link UNREACHABLE"

    I guess my question is then can TGT="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext" somehow and work ????
     
  83. yaqui

    yaqui LI Guru Member

    Maybe my script is working how it should? It seems like it returns 0 so that means grep is finding what it should?
    Code:
    # ADLIST="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showin
    tro=0&mimetype=plaintext"
    # CONN=`ping -c 1 $ADLIST | grep "208.69"`
    # echo $?
    0
    #
     
  84. yaqui

    yaqui LI Guru Member

    Well this is my final answer / version / whatever you want to call it, lol. This is working just fine for me and seems to do everything I want.
    Code:
    logger WAN UP Script Executing after 60sec
    sleep 60
    ADLIST="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
    CONN=`ping -c 1 $ADLIST | grep "208.69"`
    sleep 5
    if [ ! -x /tmp/update-adblock.sh ] ; then
    cat > /tmp/update-adblock.sh <<EOF
    #!/bin/sh
    if [ -n "$CONN" ] ; then 
       wget -O /etc/adblock.conf 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
       else 
       logger Adblock list download link UNREACHABLE
    fi
    if grep -q address /etc/adblock.conf ; then
       logger Adblock file population SUCCESS
       service dnsmasq restart
       else
       logger Adblock file population FAILURE
    fi
    EOF
      chmod +x /tmp/update-adblock.sh
      /tmp/update-adblock.sh
      cru a UpdateAdblock " <your interval here> /tmp/update-adblock.sh"
      else
      logger update-adblock.sh already exists and is executable will not re-run
    fi
    If anyone wants to use it, remember to fill in the cru "*"s part. Thanks for everyone's help guys !!
     
  85. Macskeeball

    Macskeeball LI Guru Member

    And, of course, that doesn't work for me. Figures. I don't understand how if we have the same version of the same firmware, it behaves so differently. I guess we'll just each have to use our own version. If it ain't broke, don't fix it.
     
  86. yaqui

    yaqui LI Guru Member

    This is in my init:
    Code:
    sleep 5
    logger Executing init script
    touch /etc/adblock.conf
    chmod 666 /etc/adblock.conf
    I was having problems with the other things we were trying for init. Try that and it has to work !!
     
  87. Macskeeball

    Macskeeball LI Guru Member

    Yep, it works now with the same Init and WAN Up Scripts. It seems to take forever because of the sleep 60 though. I'll have to try experimenting a little with that to see how low I can go. I'll try 30 first, then 15 if it does and 45 if it doesn't.
     
  88. Macskeeball

    Macskeeball LI Guru Member

    15 works for me. How about you?

    Out of curiosity, what router and Internet connection type are you using? I'm using a WRT54Gv2.0 with ADSL.
     
  89. pgl@yoyo.org

    pgl@yoyo.org LI Guru Member

    Just to check - it *is* possible to use comments in dnsmasq format files, right? If so, I'll add a header to the top of the file (I normally do anyway for other formats).

    yaqui: I'm sorry, but there's an error in your script. It's in the part that -- as I understand it -- is checking to see if the server is available to download the list of adservers from.

    You have this command:

    Unfortunately this will not ping the right server. I have to admit that I don't have a Linksys with Tomato installed available (although I do have a Linksys and the Tomato firmware downloaded... :)), but I'm fairly certain that the ping command is expecting to be given a hostname, and not a URL. The only way it could work is if the ping command extracted the hostname from the URL you've given it - but ping really has nothing to do with HTTP so it's unlikely this is the case.

    I think this is confirmed a few posts back when you include this in your script:

    The server pgl.yoyo.org has a different IP address:
    The IP address you listed is:
    This is just a guess, but:

    I think what is probably happening is that you're using some kind of DNS service that has wildcard DNS set up - so that you can ping <anything>.example.com. When you say "ping http://pgl.yoyo.org/...", ping is passing "http://pgl.yoyo.org" to the dns resolver, which is returning something like "http://pgl.yoyo.org.example.com" (which is obviuosly an invalid hostname, but if the resolver assumes that error checking is done beforehand, it's possible), which then resolves to 208.69.32.130. If Macskeeball isn't using the same DNS service as you, this might explain why it's not working for him.

    Personally, I'd add the script to be executed by the router's cron daemon regardless of whether the connection is available at the time you add the script itself. After all, the connection to the server might go down anyway right after you install the cronjob. If you move the error checking into the download script itself, then you can (try to) make sure that you get a valid file every time - and you can use the error checking already built into wget (which is probably going to be better than anything you or I could come up with).

    eg, something like:

    [20 minutes later...]

    OK I have to admit something here, I think I got a little carried away. Not that carried away, but I meant this to be smaller - I just figured that I might be actually using this soon (hopefully! if I don't **** up my Linksys in the process of trying to flash it). Anyway, here's what I came up with. Please bear in mind I don't have any way of testing it:


    Code:
    ## bits that you might want to change
    ##
    
    # location of the ad list
    adlisturl='http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
    
    # target for the download
    adblockconfig='/etc/adblock.conf'
    
    # local path of the script that will be updating the ad list
    updatescript=/tmp/update-adblock.sh
    
    
    ## say hello
    ##
    
    logger WAN UP Script Executing
    
    
    ## check the update script exists
    ##
    
    if [ ! -r "$updatescript" ]
    then
    	# using "<<-" in your heredoc definition strips leading tabs
    	cat > $updatescript <<- EOSCRIPT
    	#!/bin/sh
    
    	# try to download the list of ad servers
    	if wget -O "$adblockconfig" "$adlisturl"
    	then
    		# hooray
    		logger "ad list update success; restarting dnsmasq service"
    		service dnsmasq restart
    	else
    		# egads
    		logger "ad list update failed! not restarting dnsmasq"
    		#add some more output here to help solve errors
    		#logger "adblock config file: $adblockconfig ... etc..."
    	fi
    
    	# maybe add some sanity checking here? wget might be able to download
    	# the URL, but who's to say the URL is what we thought? paranoia cha
    	# cha cha
    
    	EOSCRIPT
    fi
    
    
    ## check it's executable
    ##
    
    if [ ! -x "$updatescript" ]
    then
    	chmod +x "$updatescript"
    fi
    
    
    ## install the cron job
    ##
    
    # eg: cru a UpdateAdblock "28 7 2 * 3 $SHF"
    #cru a UpdateAdblock UpdateAdblock "<your time interval here> $SHF"
    
    ## say goodbye
    ##
    
    #logger WAN UP Script Completed
    
    Hope this was in some way useful. It's entirely possible (probably) that I've just got the wrong end of the stick again and should mind my own business.

    cheers,

    - Peter
     
  90. mstombs

    mstombs Network Guru Member

    Fully agree with your ping comments, my ISP DNS behave differently - my unresolvables go to an ISP holding page.

    Must also always use "$ADLIST" as the "&" symbols can break up the string when a variable used.

    I think I understand why yaqui wants to hold back the first connection attempt - might be a month before next update attempt.

    With my setup with dhcp WAN to half-bridge DSL modem it seems the WAN is UP when the WANUP script is run (rebooting the routing alone), maybe other connections such as PPPOE to bridge modem connect earlier before Internet connection established and could take between 20 and 120 seconds to get fully connected?
     
  91. pgl@yoyo.org

    pgl@yoyo.org LI Guru Member

    Re: slow startup times - it's a fairly large set of zones you're loading into the poor little router, so it's quite fair for it to take a minute to think about it. I use a P-133 running FreeBSD as my router and local DNS server, and it can take a few seconds to load all the zones. Actually, I'm curious how long now...

    (It's actually a good example of why hosts files aren't a great way to do this - the time taken here is for the server to turn all the parse all the zone files and load them into memory - takes a little while to start with, but once it's finished you get fast lookups and properly cached name resolution; with a hosts file, it (should) look up a hostname every time, for every DNS request. Whinge whinge, etc. :))

    About holding back the first connection attempt: the easiest way to do this would be to just call the update-adblock.sh script after making sure it exists, right? And instead of having to set up a while loop that implements your own timeouts, just use the --timeout argument for wget.

    Code:
    	# try to download the list of ad servers - 60 seconds timeout
    	if wget --timeout=60 -O "$adblockconfig" "$adlisturl"
    
    and

    Code:
    ## install the cron job - and run the update script once to start with a fresh copy
    ##
    
    # eg: cru a UpdateAdblock "28 7 2 * 3 $SHF"
    $updatescript
     
  92. pgl@yoyo.org

    pgl@yoyo.org LI Guru Member

    While you're on the subject...

    Also: always use single quotes unless you specifically need a variable inside a string expanded. :)
     
  93. mstombs

    mstombs Network Guru Member

    Any particular reason? I have never been taught or read a book on bash, just picked up examples via Google...

    [Answer: following up later replies there is - single quote is strong, double quote is weak and allows parameter substitution, so just luck things have been working for me to date...]

    I don't personally like single quote "'" coz I can't easily tell the difference with back tick "`", so I only use single quote when I need nested quotes. I also don't like back tick so I use "$( )" form. The trouble of having such a flexible script language is that you can do many things in different ways ie "test" or "[ ]". No doubt there are optimal ways in terms of speed/memory/temp files/ readability but shouldn't use a script if efficiency of operation important!

    PS I don't think the Tomato/Busybox "wget" supports timeout parameters...
     
  94. pgl@yoyo.org

    pgl@yoyo.org LI Guru Member

    Just good practice. I only noted it because you noted that it's good to make sure to quote variables because they might expand to something the shell interprets as multiple variables - using single quotes is not necessary by any means, but it's good practise because:

    • you know that whatever you're quoting won't change (there's no possibility of a variable expansion)
    • it makes you more aware of situations when you are using variables that need to be expanded
    • it's ever so slightly more efficient than double quotes - the shell (or whatever is interpreting your code) doesn't have to parse the string for variables

    The man page is all you need. :) And lots of patience...

    Like I said, it's not exactly required.

    Me too! I hate backticks. Ever tried to do multiple layers of nested command substitutions? Ugh.

    Yep. Actually, I prefer to write if then statements like this, if I can get away with it:

    Code:
    grep -q arse /etc/bodyparts && {
    	# do stuff here
    	}
    
    True, true, but that doesn't mean that you shouldn't always keep efficiency in the back of your mind. :)

    Are you sure? I'd be surprised if it was called wget and didn't have this functionality... Why call it wget it it's not pretty much the same as the normal one? Not saying you're wrong, but that would be an odd decision to me. :)
     
  95. yaqui

    yaqui LI Guru Member

    Here is your answer on wget in busybox:
    Code:
    # wget --help
    BusyBox v1.2.2 (2007.05.06-15:48+0000) multi-call binary
    
    Usage: wget [-c|--continue] [-q|--quiet] [-O|--output-document file]
                    [--header 'header: value'] [-Y|--proxy on/off] [-P DIR] url
    
    wget retrieves files via HTTP or FTP
    
    Options:
            -c      continue retrieval of aborted transfers
            -q      quiet mode - do not print
            -P      Set directory prefix to DIR
            -O      save to filename ('-' for stdout)
            -Y      use proxy ('on' or 'off')
    
    # wget --timeout=60
    wget: unrecognized option `--timeout=60'
    BusyBox v1.2.2 (2007.05.06-15:48+0000) multi-call binary
    
    Usage: wget [-c|--continue] [-q|--quiet] [-O|--output-document file]
                    [--header 'header: value'] [-Y|--proxy on/off] [-P DIR] url
    
    wget retrieves files via HTTP or FTP
    
    Options:
            -c      continue retrieval of aborted transfers
            -q      quiet mode - do not print
            -P      Set directory prefix to DIR
            -O      save to filename ('-' for stdout)
            -Y      use proxy ('on' or 'off')
    
    #
     
  96. yaqui

    yaqui LI Guru Member

    I suppose this is the only way I can do this:
    Code:
    # ADLISTHOST="pgl.yoyo.org"
    # ping -c 1 $ADLISTHOST | grep "193.110.91"
    PING pgl.yoyo.org (193.110.91.2): 56 data bytes
    64 bytes from 193.110.91.2: icmp_seq=0 ttl=46 time=165.7 ms
    # echo $?
    0
    #
    So I guess just as a check to make sure the server is up... I could do this:
    Code:
    logger WAN UP Script Executing after 60sec
    sleep 60
    ADLISTHOST="pgl.yoyo.org"
    CONN=`ping -c 1 $ADLISTHOST | grep "193.110.91"`
    if [ ! -x /tmp/update-adblock.sh ] ; then
    cat > /tmp/update-adblock.sh <<EOF
    #!/bin/sh
    if [ -n "$CONN" ] ; then 
       wget -O /etc/adblock.conf 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
       else 
       logger Adblock list download link UNREACHABLE
    fi
    if grep -q address /etc/adblock.conf ; then
       logger Adblock file population SUCCESS
       service dnsmasq restart
       else
       logger Adblock file population FAILURE
    fi
    EOF
      chmod +x /tmp/update-adblock.sh
      /tmp/update-adblock.sh
      cru a UpdateAdblock " <your interval here> /tmp/update-adblock.sh"
      else
      logger update-adblock.sh already exists and is executable will not re-run
    fi
    
    That should work for everyone... correct ?
     
  97. Macskeeball

    Macskeeball LI Guru Member

    Hey, cool. Just check to make sure that Tomato can run on your router. Also, do NOT upload the firmware over WiFi; you must use a wired connection for that.

    Before you do it, save the revival guide on your computer just in case.
     
  98. yaqui

    yaqui LI Guru Member

    Here I cleaned up my script a bit and now added 2 attempts to download, seems to work fine
    Code:
    logger WAN UP Script will execute after 30sec please wait.....
    sleep 30
    
    ## DEFINE VARIABLES
    
    adlistfile='/etc/adblock.conf'
    updatescript='/tmp/update-adlist.sh'
    adlisthost='pgl.yoyo.org'
    adlisturl='http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
    pingtest=$(ping -c 1 $adlisthost | grep "from 193.110.91")
    
    ## BEGIN update-adlist.sh CREATION
    
    if [ ! -x $updatescript ] ; then
    cat > $updatescript <<EOF
    #!/bin/sh
    ## Why try to download if host unreachable
    ## Better to test first and if it is not down make two attempts
    if [ -n "$pingtest" ] ; then
    logger Adlist host ping test SUCCESS proceeding download attempts....
           if wget -O $adlistfile "$adlisturl" ; then
           logger Adlist data first download attempt SUCCESS
              elif wget -O $adlistfile "$adlisturl" ; then
              logger Adlist data second download attempt SUCCESS
           fi
    else 
    logger Adlist data host UNREACHABLE
    fi
    if grep -q address $adlistfile ; then
       logger adblock.conf file population SUCCESS
       service dnsmasq restart
       else
       logger adblock.conf file population FAILURE
    fi
    EOF
      chmod +x $updatescript
      /tmp/update-adlist.sh
      cru a UpdateAdlist " <your interval here> /tmp/update-adlist.sh"
      else
      logger update-adlist.sh already exists and is executable will not re-run
    fi
    
    ## END update-adlist.sh CREATION
    And here is my init... I still had instances where dnsmasq was starting before adblock.conf was created, so I had to shave the time a bit.
    Code:
    sleep 3
    touch /etc/adblock.conf
    logger adlbock.conf created
    sleep 2
    logger Remainder of init script starting
    Results:
    Code:
    Jan  1 00:00:38 lynxpark user.info rcheck[139]: Activating rule 1
    Dec 31 18:00:40 lynxpark user.notice root: WAN UP Script will execute after 30sec please wait.....
    Jul 25 12:46:50 lynxpark user.info ntpc[169]: Time Updated: Wed, 25 Jul 2007 12:46:50 -0500 [+1185385566s]
    Jul 25 12:47:07 lynxpark cron.warn crond[117]: time disparity of 19756426 minutes detected 
    Jul 25 12:47:17 lynxpark user.notice root: Adlist host ping test SUCCESS proceeding download attempts....
    Jul 25 12:47:19 lynxpark user.notice root: Adlist data download SUCCESS
    Jul 25 12:47:19 lynxpark user.notice root: adblock.conf file population SUCCESS
    Jul 25 12:47:19 lynxpark daemon.info dnsmasq[136]: exiting on receipt of SIGTERM
    Jul 25 12:47:20 lynxpark daemon.info dnsmasq[251]: started, version 2.39 cachesize 150

    Have fun
     
  99. Macskeeball

    Macskeeball LI Guru Member

    That works for me too.
     
  100. mstombs

    mstombs Network Guru Member

    iif ?

    Still don't know what the ping does - it just checks whether you can convert the name to an IP address via DNS lookup doesn't it? And if you can't - you do the same anyway with different message?
     

Share This Page