1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Blocking ads using dnsmasq with an additional hosts file

Discussion in 'Tomato Firmware' started by Macskeeball, Jul 20, 2007.

  1. yaqui

    yaqui LI Guru Member

    It is pinging to see whether 'pgl.yoyo.org' is reachable or not. If ping fails (with grep not finding the numbers) it will fail to populate the variable $pingtest and hence not try to wget.

    Play with it at the command line and you can see what happens when grep won't find.... (echo $pingtest won't show anything then)

    Sorry for the "iif" it happened when I was copying and pasting.

    Here examples:
    Code:
    # ADLISTHOST="pgl.yoyo.org"
    # pingtest=$(ping -c 1 $ADLISTHOST | grep "193.110.91")
    # echo $pingtest
    PING pgl.yoyo.org (193.110.91.2): 56 data bytes 64 bytes from 193.110.91.2: icmp
    _seq=0 ttl=46 time=169.7 ms
    # #see it is not null now we try to get it null
    # #there is no number 8 so lets try it
    # pingtest=$(ping -c 1 $ADLISTHOST | grep "8")
    # echo $pingtest
    
    # #ahh see null !! no reason for wget !
     
  2. mstombs

    mstombs Network Guru Member

    @yacqui - but what does the ping return if it can convert the name into numbers but not get a reply and timeout?

    Code:
    PING 193.110.91.2 (193.110.91.2): 56 data bytes
    
    --- 193.110.91.2 ping statistics ---
    1 packets transmitted, 0 packets received, 100% packet loss
    Not null when grepped also I believe!
     
  3. yaqui

    yaqui LI Guru Member

    Ok I see what you mean, here what if it is this:
    Code:
    pingtest=$(ping -c 1 $ADLISTHOST | grep "from 193.110.91")
    A failed ping does not contain "from 193.110.91" correct?
     
  4. Macskeeball

    Macskeeball LI Guru Member

    A failed ping contains "100% packet loss"
     
  5. mstombs

    mstombs Network Guru Member

    Correct - on my system that remembers the mapping from before I pulled the WAN cable out

    Code:
    # ping -c 1 $ADLISTHOST
    PING pgl.yoyo.org (193.110.91.2): 56 data bytes
    
    --- pgl.yoyo.org ping statistics ---
    1 packets transmitted, 0 packets received, 100% packet loss
    But I don;t think you need to grep because you can just "if" the "ping" i.e.

    Code:
    # if ping -c 1 $ADLISTHOST ;then echo "yes";fi
    PING pgl.yoyo.org (193.110.91.2): 56 data bytes
    64 bytes from 193.110.91.2: icmp_seq=0 ttl=49 time=19.6 ms
    
    --- pgl.yoyo.org ping statistics ---
    1 packets transmitted, 1 packets received, 0% packet loss
    round-trip min/avg/max = 19.6/19.6/19.6 ms
    yes
    # if ping -c 1 ADLISTHOST ;then echo "yes";fi
    ping: ADLISTHOST: Unknown host
    # if ping -c 1 $ADLISTHOST ;then echo "yes";fi
    PING pgl.yoyo.org (193.110.91.2): 56 data bytes
    
    --- pgl.yoyo.org ping statistics ---
    1 packets transmitted, 0 packets received, 100% packet loss
    #
     
  6. yaqui

    yaqui LI Guru Member

    That is a better/simple way. I guess the other way you could use to ping multiple sites easier - setting variables for each.

    Thanks for simplifying, I am using it now and it works great!
     
  7. yaqui

    yaqui LI Guru Member

    Well this is hopefully my last and FINAL version! :) THANK YOU everyone for all your help !!!

    Place under WAN UP and change the cru to a custom interval in this format:

    cru a <unique id> <"min hour day month week command">

    * will mean every !

    Example:
    cru a UpdateAdlist "00 9 * * * $updtadlist"
    That will do it at 9 am everyday but change to your own time.


    Code:
    ##################  BEGIN SCRIPT ####################
    
    logger WAN UP Script will execute after 30sec please wait.....
    sleep 30
    
    ## DEFINE VARIABLES
    
    adlistfile='/etc/adblock.conf'
    updtadlist='/tmp/update-adlist.sh'
    adlisthost='pgl.yoyo.org'
    adlisturl='http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
    numlines=0
    numbytes=0
    listhosttest=$(ping -c 1 $adlisthost | grep "from 193.110.91")
    
    ## BEGIN update-adlist.sh CREATION
    if [ ! -x $updtadlist ] ; then
    cat > $updtadlist <<EOF
    #!/bin/sh
      if [ -n "$listhosttest" ] ; then
         logger Ad list host ping test SUCCESS proceeding download and entries test....
            
              if wget -O $adlistfile "$adlisturl" ; then
                 numlines=$(awk 'END { print NR }' $adlistfile)
                 lowamt=1700
                 highamt=300000
                 sleep 5
                 
                    if [ "$numlines" -lt "$lowamt" ] ; then
                       logger Adblock.conf contains less than 1700 entries check file !  
                       else
                             if [ "$numlines" -gt "$highamt" ] ; then
                                logger Adblock.conf has excessive number of lines ! check the file !!
                                else
                                logger Adblock.conf entry size within normal range download SUCCESS
                             fi
                     fi       
              
              else    
                    logger Adlist data host UNREACHABLE
                    logger exiting script....
                    exit
              fi
      
      else
        logger Adlist host ping test FAILURE
        logger exiting script....
        exit
      fi
      
      if ps | grep -q dnsmasq ; then
         service dnsmasq stop
         sleep 5
         service dnsmasq start
         else
            logger Restarting dnsmasq FAILURE
      fi
    EOF
      chmod +x $updtadlist
      /tmp/update-adlist.sh
      cru a UpdateAdlist "<your interval here> $updtadlist"
      
      ## See how many lines and bytes adblock.conf contains and output values to logger
      if grep -q address $adlistfile ; then
         numlines=$(awk 'END { print NR }' $adlistfile)
         numbytes=$(ls -lg $adlistfile | awk '{ sum += $5} END {print sum }')
         sleep 5
         logger adblock.conf populated with $numlines lines and $numbytes bytes of data
        else
          logger adblock.conf is not populated !!
      fi
    
    else
      logger update-adlist.sh already exists and is executable will not re-run
    fi
    ## END update-adlist.sh CREATION
    
    
    ## BEGIN Testing the first host listed in adblock.conf on each WAN UP
      sleep 5
      if grep -q address $adlistfile ; then
         read var1 <$adlistfile
         echo -e $var1 > /tmp/testhost
         awkdhost=$(awk 'BEGIN { FS="/" } { print $2 }' /tmp/testhost)
         adblocktest=$(ping -c 1 $awkdhost | grep "from 127")
                 
            if [ -n "$adblocktest" ] ; then 
               logger First host "$awkdhost" unreachable - Ad blocking is working !!  SUCCESS
            else 
               logger Ad blocking is NOT working FAILURE
            fi
      
      else
        logger Adblock.conf is not populated with address !!
      fi
    
    ## END Testing the first host listed in adblock.conf on each WAN UP
    
    ##################  END SCRIPT ###########################
    

    Result:
    Code:
    Dec 31 18:05:51 lynxpark user.notice root: WAN UP Script will execute after 30sec please wait.....
    Jul 30 12:04:31 lynxpark user.info ntpc[258]: Time Updated: Mon, 30 Jul 2007 12:04:31 -0500 [+1185814717s]
    Jul 30 12:04:38 lynxpark cron.warn crond[121]: time disparity of 19763579 minutes detected 
    Jul 30 12:04:59 lynxpark user.notice root: Ad list host ping test SUCCESS proceeding download and entries test....
    Jul 30 12:05:06 lynxpark user.notice root: Adblock.conf entry size within normal range download SUCCESS
    Jul 30 12:05:06 lynxpark daemon.info dnsmasq[224]: exiting on receipt of SIGTERM
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: started, version 2.39 cachesize 150
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: compile time options: no-IPv6 GNU-getopt no-RTC no-ISC-leasefile no-DBus no-I18N no-TFTP
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: DHCP, IP range 192.168.10.100 -- 192.168.10.149, lease time 2d12h
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: reading /etc/resolv.dnsmasq
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: using nameserver blah
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: using nameserver blah
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: read /etc/hosts - 0 addresses
    Jul 30 12:05:12 lynxpark daemon.info dnsmasq[338]: read /etc/hosts.dnsmasq - 1 addresses
    Jul 30 12:05:17 lynxpark user.notice root: adblock.conf populated with 1978 lines and 70511 bytes of data
    Jul 30 12:05:22 lynxpark user.notice root: First host 101com.com unreachable - Ad blocking is working !! SUCCESS 
     
  8. asparagui

    asparagui Network Guru Member

    how funny, i downloaded tomato just last week and this was the exact thing i was trying, albiet with iptables -j DROP. the problem with the localhost approach for me is that i have to turn off my local webserver else i get apache not found barf all over my pages.

    but all in all, very cool stuff. i was gonna work up the courage to write noxxic to request this as a feature now it looks like all the hard work is done for him...you should write him. nice work.
     
  9. madsul

    madsul LI Guru Member

    So I just paste this into the Wan Up Script window and Save it? Or is there more to do?
     
  10. mstombs

    mstombs Network Guru Member

    I have to question the use of WAN UP script, my testing of this and the reconnect script suggests that WANUP is called everytime the WAN comes up, a couple of seconds after the firewall script - in fact I don't see why there are 2. Will have to re-check the docs and have a look at the source, I expected init once every boot and WAN-UP only the first time the WAN comes up, firewall on every reconnection?
     
  11. Meffy

    Meffy LI Guru Member

    cru a UpdateAdlist " <put yours here> $updtadlist"
    What am i suposed to put in there?
     
  12. yaqui

    yaqui LI Guru Member

    Meffy:
    From FAQ.

    From mraneri's post:
     
  13. Macskeeball

    Macskeeball LI Guru Member

    To think that just last week, keeping up with the back and forth going on in this thread had a lot of my focus. Now it's become almost meaningless to me. When I went to the hospital to say goodbye to my granddad while he was on his deathbed, I found out that my stepmom's niece had been killed in a car accident with her fiancee at the wheel. Now, my dad is in the same boat as my granddad. He lost his pulse, and six or seven doctors in the ER had to work for 60 minutes to revive him. All of this is happening at once.

    I know this is completely off-topic and that this is just a router forum and that no one here really knows me, but I have to get this out. This is a living nightmare.
     
  14. yaqui

    yaqui LI Guru Member

    Damn and they say bad things come in three's. I hope things get better for you and your loved ones... sorry to hear about all your going through.
     
  15. mikester

    mikester Network Guru Member

    Best wishes to you Macskeeball. Sorry about your bad news.

    -----------------------------------------------------------

    Can anybody been explain what I'm seeing with my Tomato?

    I tried modifying the /etc/hosts file. Instead of routing to 127.0.0.1 I route to my tomato ip where I have a web page displaying "web page blocked" to users on the network.

    The problem I'm seeing is that typing a URL as 101com.com gets blocked, whereas if someone types www.101com.com the web page opens. Any hints as to why?
     

    Attached Files:

  16. damwill

    damwill Network Guru Member

    If using a hosts file you will need to block both www.101com.com and 101com.com. If you want to block anything from 101com.com try adding address=/101com.com/127.0.0.1 to your Dnsmasq Custom Configuration under Advanced --> DHCP/DNS.
     
  17. yaqui

    yaqui LI Guru Member

    Huh? They both ping the same for me.. to 127.0.0.1
    Code:
    # ping -c 1 www.101com.com
    PING www.101com.com (127.0.0.1): 56 data bytes
    64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=1.1 ms
    
    --- www.101com.com ping statistics ---
    1 packets transmitted, 1 packets received, 0% packet loss
    round-trip min/avg/max = 1.1/1.1/1.1 ms
    # ping -c 1 101com.com
    PING 101com.com (127.0.0.1): 56 data bytes
    64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.8 ms
    
    --- 101com.com ping statistics ---
    1 packets transmitted, 1 packets received, 0% packet loss
    round-trip min/avg/max = 0.8/0.8/0.8 ms
     
  18. Macskeeball

    Macskeeball LI Guru Member

    Basically, I think he read the beginning of the thread and is still trying the hosts file method as opposed to the adblock.conf method. There are a few things I need to do in this thread, but that's all going by the wayside right now as I deal with these family crisises. My dad passed at around noon or 1pm today. At least it was peaceful. I really appreciate people's concern; it means a lot to us, so thanks.
     
  19. mikester

    mikester Network Guru Member

    My condolences.

    When I used the adblock.conf method I would get dnsmasq errors reading the file and websites like google and linksysinfo were getting blocked/dropped.

    I don't like the loop back method as we have some web servers running in our intranet.
     
  20. mikester

    mikester Network Guru Member

    My condolences.

    When I used the adblock.conf method I would get dnsmasq errors reading the file and websites like google and linksysinfo were getting blocked/dropped.

    I don't like the loop back method as we have some web servers running in our intranet.
     
  21. yaqui

    yaqui LI Guru Member

    Mac: sorry for your losses, condolences to you.
    ============================================

    Mikester: set up a method in the script to change the loopback address in the adblock.conf file... or did you try that already?
     
  22. PeterT

    PeterT Network Guru Member

     
  23. yaqui

    yaqui LI Guru Member

    Yes that is a better method.

    When i put listhosttest=$(ping -c 1 $adlisthost | grep "from 193.110.91") in the script it wouldn't work with my old way of doing things, but since I found out a new way with how to create literal 'here documents' it will work. But your method is fine.
     
  24. Macskeeball

    Macskeeball LI Guru Member

    Are YouTube videos broken for anyone else? They were working for me earlier today, but now they don't seem to work in any browser I try, on both computers. Safari's Activity Monitor makes it look like Google has now made YouTube videos go through Doubleclick for tracking purposes. Am I just experiencing a fluke, or is YouTube broken for other people blocking DoubleClick with a method like this?
     
  25. mraneri

    mraneri LI Guru Member

    They work fine for me. Of course, I am adblocking as well. Maybe something else is going on.
     
  26. Macskeeball

    Macskeeball LI Guru Member

    Whatever it was, they're working now. Weird, but I'm glad it's back.
     
  27. guest

    guest Network Guru Member

    Is there an easy way to edit the script to load a second file to RAM that contains a list of custom domain information not included in pgl@yoyo's list? This way when the script autoupdates, the custom list is not overwritten.

    Also is it possible to block a specific directory of a domain through the conf method without blocking the SLD outright?


    Example:
    ----------------------------------------------------------

    Block this specific directory
    http://www.miscwebsite.com/1475/ty/Advertisements/

    Meanwhile the SLD, and any subdomains are not blocked (etc)
    http://www.miscwebsite.com/
     
  28. najevi

    najevi LI Guru Member

    1) set up CIFS1

    2)
    wget the pgl@yoyo's list to a file /cifs1/adblock.base
    separately maintain your own add-on to that list as /cifs/1adblock.addon
    then concatenate the two files to produce /etc/adblock.conf
    TIP:
    cat foo bar | sort -u > foobar
    will eliminate duplicate lines

    All of the step 2 work can be done within the WAN UP script.

    I use a similar process for concatenating a few different sources for my HOSTS file. I am in the process of learning about how to use dnsmasq to block whole domains so perhaps when I have that down pat I'll post the script I use.

    enjoy!
     
  29. Macskeeball

    Macskeeball LI Guru Member

    In the web GUI, choose Advanced -> DHCP/DNS, and then edit the field with your custom dnsmasq config. The adblock stuff goes into a separate file already, and both take effect together.
     
  30. najevi

    najevi LI Guru Member

    So at http://192.168.1.1/advanced-dhcpdns.asp
    in the Dnsmasq Custom Configuration field I enter:

    conf-file=/cifs1/adblock.base
    conf-file=/cifs1/adblock.addon


    and the effect is that both lists are applied as though they were concatenated?

    ... too easy!

    Thanks.
     
  31. Macskeeball

    Macskeeball LI Guru Member

    Actually, you're still making it overly complicated.

    Assuming that your manual selections for adblock.addon was small (as a manual list probably is), there would be no reason to not simply put the lines in adblock.addon directly in that field and just edit it with the Web GUI. No CIFS required, for either adblock.base (completely automated and stored on the router itself) or adblock.addon (a few lines directly in the field in the Web GUI), and that means no dependence on anything beyond your router.
     
  32. najevi

    najevi LI Guru Member

    Fair enough. My addons was growing too much and it looked like it would be a chore to keep up with it so about 24 hours ago I undid the hosts file and dnsmasq configuration work and enabled openDNS with a number of the dashboard features enabled. VERY pleased so far.

    Thanks for the tips.
     
  33. guest

    guest Network Guru Member

    Thanks for your reply Macskeeball.


    Unfortunately, if I follow your suggestion the main domain is blocked entirely



    Advanced --> DHCP / DNS Server --> Dnsmasq

    Text added:
    address=/www.miscrandomwebsite.com/banners/127.0.0.1

    Entering this way does not block the "banners" subdirectory, but it blocks the entire "www.miscrandomwebsite.com" website.

    Subdomains however are accessible "forums.miscrandomwebsite.com" , etc
     
  34. Macskeeball

    Macskeeball LI Guru Member

    Right, this thread is about domain based blocking, not more specific blocking. The idea is to identify common third party ad servers and ad-specific subdomains on "real" sites, and cut them off completely at the domain or subdomain level.

    The / you included in there is probably causing a problem, considering it's the separator that dnsmasq is using.

    The benefit of the domain and subdomain level blocking is that it can be applied on the router and therefore provide pretty good adblocking for your entire LAN (including devices that you otherwise couldn't get any adblocking on, such as a gaming system). You'll need to use an off-router solution in addition to this to get what you want.

    For the more specific than domain or subdomain level blocking, you'll need to use an additional ad-blocking solution, such as CSS files or, with Firefox, Adblock Plus and Filterset G (an autoupdater for AdBlock).
     
  35. rommels

    rommels LI Guru Member

    So is somebody going to sum up this huge thread with a coherent guide on the setup of this?

    Something like THIS or should I just use that one, because I sure as hell don't have the time to read though 14 pages...
     
  36. mraneri

    mraneri LI Guru Member

    I still use the one in your link (I wrote it, with stuff I learned from others). Currently filtering ~16400 host names... No problems.

    (In the last week or two, a bad entry popped into the MVPS file, It throws a warning, but everything still works normally.) I suspect the maintainer of the list will clean it up and the warning will go away within the next few weeks.
     
  37. guest

    guest Network Guru Member


    I edited the script as per your suggestion , but the log is returning FAILURE during wanup.

    LOG

    Dec 31 15:00:09 unknown user.notice root: Remainder of init script starting
    Dec 22 08:04:58 unknown user.notice root: WAN UP Script will execute after 30sec please wait.....
    Dec 22 08:05:29 unknown user.notice root: Adlist host ping test FAILURE
    Dec 22 08:05:29 unknown user.notice root: exiting script....
    Dec 22 08:05:29 unknown user.notice root: adblock.conf is not populated !!
    Dec 22 08:05:34 unknown user.notice root: Adblock.conf is not populated with address !!
    Dec 22 08:05:45 unknown cron.warn crond[95]: time disparity of 19972385 minutes detected




    Data added to router via Tomato GUI


    Advanced --> DHCP / DNS --> Dnsmasq Custom Configuration
    Code:
    conf-file=/etc/adblock.conf
    

    Administration --> Scripts --> Init
    Code:
    sleep 3
    touch /etc/adblock.conf
    logger adlbock.conf created
    sleep 2
    logger Remainder of init script starting
    


    Administration --> Scripts --> WAN Up
    Code:
    ##################  BEGIN SCRIPT ####################
    
    logger WAN UP Script will execute after 30sec please wait.....
    sleep 30
    
    ## DEFINE VARIABLES
    
    adlistfile='/etc/adblock.conf'
    updtadlist='/tmp/update-adlist.sh'
    adlisthost='pgl.yoyo.org'
    adlisturl='http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
    numlines=0
    numbytes=0
    
    ## BEGIN update-adlist.sh CREATION
    if [ ! -x $updtadlist ] ; then
    cat > $updtadlist <<EOF
    #!/bin/sh
      if ping -c $adlisthost ; then
         logger Ad list host ping test SUCCESS proceeding download and entries test....
            
              if wget -O $adlistfile "$adlisturl" ; then
                 numlines=$(awk 'END { print NR }' $adlistfile)
                 lowamt=1700
                 highamt=300000
                 sleep 5
                 
                    if [ "$numlines" -lt "$lowamt" ] ; then
                       logger Adblock.conf contains less than 1700 entries check file !  
                       else
                             if [ "$numlines" -gt "$highamt" ] ; then
                                logger Adblock.conf has excessive number of lines ! check the file !!
                                else
                                logger Adblock.conf entry size within normal range download SUCCESS
                             fi
                     fi       
              
              else    
                    logger Adlist data host UNREACHABLE
                    logger exiting script....
                    exit
              fi
      
      else
        logger Adlist host ping test FAILURE
        logger exiting script....
        exit
      fi
      
      if ps | grep -q dnsmasq ; then
         service dnsmasq stop
         sleep 5
         service dnsmasq start
         else
            logger Restarting dnsmasq FAILURE
      fi
    EOF
      chmod +x $updtadlist
      /tmp/update-adlist.sh
      cru a UpdateAdlist "* * * * * $updtadlist"
      
      ## See how many lines and bytes adblock.conf contains and output values to logger
      if grep -q address $adlistfile ; then
         numlines=$(awk 'END { print NR }' $adlistfile)
         numbytes=$(ls -lg $adlistfile | awk '{ sum += $5} END {print sum }')
         sleep 5
         logger adblock.conf populated with $numlines lines and $numbytes bytes of data
        else
          logger adblock.conf is not populated !!
      fi
    
    else
      logger update-adlist.sh already exists and is executable will not re-run
    fi
    ## END update-adlist.sh CREATION
    
    
    ## BEGIN Testing the first host listed in adblock.conf on each WAN UP
      sleep 5
      if grep -q address $adlistfile ; then
         read var1 <$adlistfile
         echo -e $var1 > /tmp/testhost
         awkdhost=$(awk 'BEGIN { FS="/" } { print $2 }' /tmp/testhost)
         adblocktest=$(ping -c 1 $awkdhost | grep "from 127")
                 
            if [ -n "$adblocktest" ] ; then 
               logger First host "$awkdhost" unreachable - Ad blocking is working !!  SUCCESS
            else 
               logger Ad blocking is NOT working FAILURE
            fi
      
      else
        logger Adblock.conf is not populated with address !!
      fi
    
    ## END Testing the first host listed in adblock.conf on each WAN UP
    
    ##################  END SCRIPT ###########################
    
     
  38. guest

    guest Network Guru Member

    Please, don't everyone answer at once. :lol:


    Seriously though, I could use some assistance, and am appreciative if someone could help. :eek:
     
  39. mstombs

    mstombs Network Guru Member

    I also question whether all this should be in the WANUP - it gets run everytime the WAN is reconnected. Is 3 second wait in init script long enough?

    You should also check the script file that is created - I suspect you need to escape some of the $vars with \$vars, because the variable substitution will be made when the script is written, you want it when the script is run.
     
  40. Nox997

    Nox997 LI Guru Member

    "undo" the guide

    Hello!

    How can I "undo" the changes made in this guide? i.e. no domain blocking?

    Thanks,
    Nox
     
  41. itsjr

    itsjr LI Guru Member

    Thanks so much to you guys, I've got this working great.

    My next step is to point the blocked domains to a webserver on my local network to server up a blank.gif or page instead of the ad.

    So, here's what I got:

    WAN Up:
    Code:
    logger WAN UP Script will execute after 30sec please wait.....
    sleep 30
    
    ## DEFINE VARIABLES
    
    adlistfile='/etc/adblock.conf'
    updatescript='/tmp/update-adlist.sh'
    adlisthost='pgl.yoyo.org'
    adlisturl='http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
    pingtest=$(ping -c 1 $adlisthost | grep "from 193.110.91")
    
    ## BEGIN update-adlist.sh CREATION
    
    if [ ! -x $updatescript ] ; then
    cat > $updatescript <<EOF
    #!/bin/sh
    ## Why try to download if host unreachable
    ## Better to test first and if it is not down make two attempts
    if [ -n "$pingtest" ] ; then
    logger Adlist host ping test SUCCESS proceeding download attempts....
           if wget -O $adlistfile "$adlisturl" ; then
           logger Adlist data first download attempt SUCCESS
              elif wget -O $adlistfile "$adlisturl" ; then
              logger Adlist data second download attempt SUCCESS
           fi
    else 
    logger Adlist data host UNREACHABLE
    fi
    if grep -q address $adlistfile ; then
       logger adblock.conf file population SUCCESS
       service dnsmasq restart
       else
       logger adblock.conf file population FAILURE
    fi
    EOF
      chmod +x $updatescript
      /tmp/update-adlist.sh
      cru a UpdateAdlist "28 7 2 * 3 /tmp/update-adlist.sh"
      else
      logger update-adlist.sh already exists and is executable will not re-run
    fi
    
    ## END update-adlist.sh CREATION
    I'm no good with this stuff, where would the code go to grep 127.0.0.1 -> 10.0.1.201?

    Thanks again for all your hard work
     
  42. mstombs

    mstombs Network Guru Member

    Not sure what script you have started with, but that's not going to work!

    Problem 1. pingtest will be done once when defined, and the result written in the file.
     
  43. itsjr

    itsjr LI Guru Member

    ok, my bad. It was hard to follow along with all the clutter.

    Thing is, it is working. It blocks the ad domains perfectly. Points them to 127.0.0.1.


    What about this then. It was taken from yaqui on page 11 of this thread:

    Code:
    ## DEFINE VARIABLES
    
    adlistfile='/etc/adblock.conf'
    updtadlist='/tmp/update-adlist.sh'
    adlisthost='pgl.yoyo.org'
    adlisturl='http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext'
    numlines=0
    numbytes=0
    listhosttest=$(ping -c 1 $adlisthost | grep "from 193.110.91")
    
    ## BEGIN update-adlist.sh CREATION
    if [ ! -x $updtadlist ] ; then
    cat > $updtadlist <<EOF
    #!/bin/sh
      if [ -n "$listhosttest" ] ; then
         logger Ad list host ping test SUCCESS proceeding download and entries test....
            
              if wget -O $adlistfile "$adlisturl" ; then
                 numlines=$(awk 'END { print NR }' $adlistfile)
                 lowamt=1700
                 highamt=300000
                 sleep 5
                 
                    if [ "$numlines" -lt "$lowamt" ] ; then
                       logger Adblock.conf contains less than 1700 entries check file !  
                       else
                             if [ "$numlines" -gt "$highamt" ] ; then
                                logger Adblock.conf has excessive number of lines ! check the file !!
                                else
                                logger Adblock.conf entry size within normal range download SUCCESS
                             fi
                     fi       
              
              else    
                    logger Adlist data host UNREACHABLE
                    logger exiting script....
                    exit
              fi
      
      else
        logger Adlist host ping test FAILURE
        logger exiting script....
        exit
      fi
      
      if ps | grep -q dnsmasq ; then
         service dnsmasq stop
         sleep 5
         service dnsmasq start
         else
            logger Restarting dnsmasq FAILURE
      fi
    EOF
      chmod +x $updtadlist
      /tmp/update-adlist.sh
      cru a UpdateAdlist "<your interval here> $updtadlist"
      
      ## See how many lines and bytes adblock.conf contains and output values to logger
      if grep -q address $adlistfile ; then
         numlines=$(awk 'END { print NR }' $adlistfile)
         numbytes=$(ls -lg $adlistfile | awk '{ sum += $5} END {print sum }')
         sleep 5
         logger adblock.conf populated with $numlines lines and $numbytes bytes of data
        else
          logger adblock.conf is not populated !!
      fi
    
    else
      logger update-adlist.sh already exists and is executable will not re-run
    fi
    ## END update-adlist.sh CREATION
    
    
    ## BEGIN Testing the first host listed in adblock.conf on each WAN UP
      sleep 5
      if grep -q address $adlistfile ; then
         read var1 <$adlistfile
         echo -e $var1 > /tmp/testhost
         awkdhost=$(awk 'BEGIN { FS="/" } { print $2 }' /tmp/testhost)
         adblocktest=$(ping -c 1 $awkdhost | grep "from 127")
                 
            if [ -n "$adblocktest" ] ; then 
               logger First host "$awkdhost" unreachable - Ad blocking is working !!  SUCCESS
            else 
               logger Ad blocking is NOT working FAILURE
            fi
      
      else
        logger Adblock.conf is not populated with address !!
      fi
    
    ## END Testing the first host listed in adblock.conf on each WAN UP
    
    ##################  END SCRIPT ###########################
     
  44. mstombs

    mstombs Network Guru Member

    I see - I have made similar comments before, but never fixed the script. I don't actually use this one - I update the list manually via scripts stored on /jffs - so nothing happens on every start-up or every time the wan comes up. I'll have a look later but there is a general issue that the $vars will be executed/ replaced when the batch file is created. Not when it is run, as the author intended.
     
  45. itsjr

    itsjr LI Guru Member

    So, can I put a manually downloaded file on my CIFS share, and have tomato access it? I okay with manually downloading the file.
     
  46. mstombs

    mstombs Network Guru Member

    I'm sure you can load adblock files from /CIFS but you will break internet access if the router boots and fails to load the adblock conf if the CIFS not available. I use /jffs and get this problem on first boot after upgrading software etc.

    As stated above I prefer to update the adblock file manually (auto gets in the way when 'playing'), I use this little script, stored on /jffs/getadblock.sh

    Code:
    #!/bin/sh -x
    CNF=/jffs/adblock.conf
    ADS="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&mimetype=plaintext"
    AIP="0.0.0.0" # changes 127.0.0.1 to this globally
    wget -O - "$ADS"|sed "s/127.0.0.1/$AIP/">$CNF
    Change the "0.0.0.0" to your server if you have custom error pages

    I have

    Code:
    conf-file=/jffs/adblock.conf
    in my dnsmasq web screen config and use this manual command to restart dnsmasq to load new config if updated

    Code:
    service dnsmasq restart
     
  47. yaqui

    yaqui LI Guru Member

    Yes, it took me a long time to hunt down the correction for that!!! ^_^
    I haven't gone through and corrected all my scripts, the solution lays in the cat statement.

    I have mine currently set to just use a cron at some odd hour early in the morning. Don't have to worry about using jffs or any interruptions in service. Your situation may differ. :)
     
  48. mstombs

    mstombs Network Guru Member

    A solution is in your thread here

    http://www.linksysinfo.org/forums/showthread.php?t=54399

    something clever to do with those braces? In the scripts in this thread some variables should be evaluated when writing the file, other's when the file is executed. I recall you can put \$ to write the $ to the file.
     
  49. yaqui

    yaqui LI Guru Member

    yes you are correct, I see where they need to be placed now. some are incorrect.
     
  50. Kye-U

    Kye-U Addicted to LI Member

    Yaqui, gorgeous script! It uses minimal resources, yet it functions perfectly!

    Rommels, I'll post what needs to be done for the setup of this script:

    In Advanced => DHCP / DNS, paste this in the Dnsmasq Custom Configuration textarea:

    In Administration => Scripts => Init, paste the following:

    In Administration => Scripts => WAN Up, paste the following:

    Note: this is a customized version of Yaqui's script; all logger lines have been removed, as well as the counting of lines and the testing of the first host to see if it's working. I included a line to replace all 127.0.0.1 addresses to 0.0.0.0 to enhance performance (it has been bolded for clarity). To get the original script, go to Page 11 of this topic.

    Yaqui, mstombs: does this problem exist in this script?

    Thanks!
     
  51. mstombs

    mstombs Network Guru Member

    I think you are OK, the $vars are all fixed, they will be converted to their text values when the file is written. The problem is when you create and use variables in the "here" document - check the file that is created looks how you expect. Yaqui got to the bottom of this issue and explained how to avoid the $var substitution - but I've already forgotten the details...

    http://www.linksysinfo.org/forums/showthread.php?t=54399

    I am surprised you are using /etc for the conf file - I didn't know that was writeable!

    I should point out it is a major task to keep such a list up-to-date, I do see pgl.yoyo regularly updated but others question the management ie google cached copy of blocked domain for those of you using this list :-

    malwaredomainlist.com
     
  52. xcooling

    xcooling LI Guru Member

  53. mstombs

    mstombs Network Guru Member

    You use the Tomato written wan-up script stored in nvram so will not have this issue, but do you realize this is run every time the the wan reconnects? But also anything else in the wan-up will be run on the timed cron task! Also you do not support the powerful dnsmasq domain rather than host blocking feature.
     
  54. Kye-U

    Kye-U Addicted to LI Member

    Thanks!

    I've just checked the contents of the script file that's created, and all the variables have been written as their values.
     
  55. TexasFlood

    TexasFlood Network Guru Member

    I read this post a couple of weeks ago or so and it's been knocking around the back of my brain since. With domain blocking being more efficient in mind, I was looking at xcooling's script.

    xcooling's script can potentially download and combine the following sources:

    http://www.mvps.org/winhelp2002/hosts.txt
    http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts
    http://www.it-mate.co.uk/downloads/hosts.txt
    http://hostsfile.mine.nu/Hosts

    Due to memory considerations, the latter two are disabled by default, which is to say that the variables HSFSOURCE and HFPSOURCE are set to 0. Correct me if I'm wrong but the remaining sources - MVPS HOSTS file includes domains and subdomains whereas the PGL YOYO file is basically just domains. So by setting MVPSSOURCE to 0 and leaving only PGLSOURCE set to 1, xcooling's script is now blocking only domains.

    I'm running it like that now as a test. So far, although arguably less complete, it runs faster, takes less memory, and is easier to administratively tweak.

    Maybe the mvps list is better, I don't really know but unless somebody has a way to parse the mvps list down to domains, the yoyo list is a lot smaller and seem to be working ok for me so far. Might be worth including mvps as a backup, basically the opposite of what yaqui did. I'd rather have the speed & free memory right now but good to have a backup I suppose.

    Thanks xcooling for including an easy to use feature that I only just now realized was there!
     
  56. jochen

    jochen LI Guru Member

    There is a problem when processing the PGL Yoyo file. The hostformat "hosts" contains html code which is no properly stripped off. Use hostformat "nohtml" instead.
     
  57. pgl@yoyo.org

    pgl@yoyo.org LI Guru Member

    Actually, a better option is to use &mimetype=plaintext. See here for all the available options: http://pgl.yoyo.org/as/formats.php.
     
  58. peyton

    peyton LI Guru Member

  59. FattysGoneWild

    FattysGoneWild LI Guru Member

    Kinda confused. What script works say with MVPS hosts file? That is the 1 I want to add.
     
  60. tmr250z

    tmr250z LI Guru Member

    If you only want to use the MVPS hosts file, then I suggest the script from this thread: http://www.linksysinfo.org/forums/showthread.php?t=53133

    I've used it for years on my WRT54GL, it works perfectly for me.
     
  61. FattysGoneWild

    FattysGoneWild LI Guru Member

    I will take a look at that. Thanks so much. 1 more thing. I use Adblock Plus for firefox on 2 machines. Can we use that as well or does it put to much stress on the router?

     
  62. tmr250z

    tmr250z LI Guru Member

    I use Adblock Plus with Firefox too and I haven't noticed any problems using it with the hosts file on the router. If anything using the hosts file will put less stress on the router because you won't be downloading the ads that the hosts file blocks. And what the router's hosts file misses, ABP will block locally in Firefox, so its a nice layer protection as well.
     
  63. FattysGoneWild

    FattysGoneWild LI Guru Member

    I should have been more specific. Sorry. What I mean. Is there a Tomato script we can use for Adblock Plus with Rick's EasyList instead of having to use the add on in our browser. If so. By doing that and using the hosts script. Would it make the router work to hard? Also. I managed to put in the hosts mvps script and its working great. Very cool.

     
  64. tmr250z

    tmr250z LI Guru Member

    I have no idea, don't know if that's even possible. I think the Easylist formatting would have to completely change in order to use it on the router. But I'm glad MVPS script is working well for you.
     
  65. FattysGoneWild

    FattysGoneWild LI Guru Member

    You are probably right. I will stick to using ABP in FF.

     
  66. FattysGoneWild

    FattysGoneWild LI Guru Member

    Question. I am still using the following script.

    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
    echo -e "#!/bin/sh\nwget -O - http://www.mvps.org/winhelp2002/hosts.txt | grep 127.0.0.1 | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/hosts\nlogger DOWNLOADED http://www.mvps.org/winhelp2002/hosts.txt\nkillall -1 dnsmasq" > /tmp/dlhosts
    chmod 777 /tmp/dlhosts
    /tmp/dlhosts
    fi
    cru a Gethosts "45 23 * * 5 /tmp/dlhosts"

    I changed the date and time as suggested by OP. I would rather update it though manually instead of checking once a week. That way if a new hosts file happens to come out a day before or after. I do not have to wait a week for a update. But how should the script look to do this? And is there any other additional step to update the hosts file manually?
     
  67. srouquette

    srouquette Network Guru Member

    if you want to update manually and you only have this script in your WAN Up, comment the line with cru (put a #cru, it won't update), then log into your router with putty (telnet), execute /tmp/script_wanup.sh
     
  68. FattysGoneWild

    FattysGoneWild LI Guru Member

    Say if I have to reboot the router for any reason. The script will still automatically dl the latest version every time just like it is doing now correct?
     
  69. srouquette

    srouquette Network Guru Member

    Yes, if it's in the WAN Up script. The router runs this script every times it's connected to internet.

    the "cru" line is only here to schedule an automatic update. If you don't want automatic update, you can remove/comment it.
     

Share This Page