1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Auto DL Hosts File and Install...

Discussion in 'Tomato Firmware' started by mraneri, May 27, 2007.

  1. Bill_S

    Bill_S Network Guru Member

    I understand that if you can ping and get a response from a site that is on the list located here, http://winhelp2002.mvps.org/hosts.htm, that the script is not working (blocking). Am I correct or is there another way to test to see if the script is working. If I (we) have a specific way to test it would really be helpful.
    Thanks
     
  2. Bill_S

    Bill_S Network Guru Member

    I thought I had this mastered but now I find I cannot get this to work, I set the routers DHCP / DNS Server (LAN) to use internal DNS and I installed the script below in the WAN Up field. If I ping a site listed on the list I get the sites correct IP address not the 127.0.0.1 address.
    Here is the log showing the file was loaded:
    What I am doing wrong?
     
  3. Bill_S

    Bill_S Network Guru Member

    Ooops, I discovered my problem. I had the PC configured to use some public DNS servers not the routers. I am posting this in order to help others not make the same mistake. The PC has to be configured to obtain DNS server address automatically or use the routers IP address as the DNS server. Please correct me if I am wrong.
     
  4. teo yabgu

    teo yabgu Serious Server Member

    Hi,
    I have installed the script(the one with the whitelist) and it works like a charm. But there is one thing I couldn't figure out. I still get ads in gmail and I have check that the ads go to pagead2.googlesynication.com and I pinged that address I got back 127.0.0.1 so it seems working but actually it does not. Am I missing something? if somebody can help, would be appreciated.
    Btw, thanks for the great work on the script.
     
  5. mraneri

    mraneri LI Guru Member

    It's probably a cache problem, OR the ad isn't really being served from there.
    Clear your browser cache, and clear your DNS cache...

    Or, don't worry about it. Eventually it would disappear...
     
  6. teo yabgu

    teo yabgu Serious Server Member

    Thanks for fast reply mraneri. I have thought about that too and tried it with couple different computer in my house. I have tried with osx, windows and linux operating systems after I flush dns cache, still the same but as you said I will wait and meantime read more about it.
     
  7. teo yabgu

    teo yabgu Serious Server Member

    it has been more than a day and I have tried so many things but no result. I have found different versions of pixelserv and each one has different problem. The last pixelserv I am using now, downloaded through here.(pixelserv1.8) Ads are still coming up with gmail and I noticed when I use www.youtube.com, pixelserv works great but when I use https://www.youtube.com, ads are still blocked but pixelserv cannot replace the ads so "unable to load the page" warning shows up on the ads locations.(It works with safari but firefox and chrome still not working) any clue where I am doing it wrong. I have an Asus RT-N16 and using tomato 1.28.
     
  8. zbeyuz

    zbeyuz Serious Server Member

    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: started, version 2.66tomatofinalbf1 cachesize 10000
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset Tomato-helper auth
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: asynchronous logging enabled, queue limit is 10 messages
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq-dhcp[507]: DHCP, IP range 192.168.1.2 -- 192.168.1.254, lease time 1d
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: reading /etc/resolv.dnsmasq
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: using nameserver 208.67.222.222#53
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: using nameserver 8.8.8.8#53
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/hosts - 2 addresses
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/dnsmasq/hosts/hosts - 6 addresses
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.err dnsmasq[507]: failed to load names from /etc/blkhosts: No such file or directory
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq-dhcp[507]: read /etc/dnsmasq/dhcp/dhcp-hosts
    Apr 23 23:58:00 RT-C0C1C08823A0 daemon.notice miniupnpd[997]: version 1.8 started
    Apr 23 23:58:00 RT-C0C1C08823A0 daemon.notice miniupnpd[997]: HTTP listening on port 52888
    Apr 23 23:58:00 RT-C0C1C08823A0 daemon.notice miniupnpd[997]: Listening for NAT-PMP traffic on port 5351
    Apr 23 23:58:01 RT-C0C1C08823A0 daemon.info pppd[458]: System time change detected.
    Apr 23 23:58:01 RT-C0C1C08823A0 user.notice root: WAN UP Script Executing
    Apr 23 23:58:43 RT-C0C1C08823A0 cron.err crond[486]: time disparity of 22778938 minutes detected
    Apr 24 00:00:01 RT-C0C1C08823A0 syslog.info root: -- MARK --
    Apr 24 00:01:20 RT-C0C1C08823A0 user.notice root: DOWNLOADED ADBLOCK HOSTS FILE(s)
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/hosts - 2 addresses
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/dnsmasq/hosts/hosts - 6 addresses
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.err dnsmasq[507]: failed to load names from /etc/blkhosts: Permission denied
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.info dnsmasq-dhcp[507]: read /etc/dnsmasq/dhcp/dhcp-hosts


    Does anyone know why I can't get this script to work ?

    P/s: I am using Toastman Tomato firmware version: 1.28.0502.2 on Cisco E4200 v1.
     
  9. koitsu

    koitsu Network Guru Member

    chmod 777 /etc/blkhosts or possibly just chmod 644 /etc/blkhosts
     
  10. dontbotherme

    dontbotherme Network Guru Member

  11. eahm

    eahm LI Guru Member

    Why is my favorite anti-ads script no longer working? I haven't use it in a long time and I've just tested it again but it's not blocking anything. Testing on Shibby 116.

    Thanks.
     
  12. FattysGoneWild

    FattysGoneWild LI Guru Member


    Does the original script from OP still work with this new change with in MVPS Hosts File?!?!?
     
  13. mraneri

    mraneri LI Guru Member

    EDIT: 2/28/2014
    DISREGARD THIS POST!!! IT'S ALL WRONG!!!!! KEEP READING FOR THE SOLUTION....

    ________________________________

    The script still works with the MVPS change. The script does a search and replace for 127.0.0.1 replacing with 0.0.0.0. Given the MVPS change, that hosts file simply doesn't contain any matched searches so that's no problem.

    Here's the problem now:

    Code:
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    
    -                      4% |*                              | 21450  00:00:23 ETA
    -                     64% |*******************            |   329k 00:00:01 ETA
    -                    100% |*******************************|   505k 00:00:00 ETA
    Connecting to jamesisbored.com (162.243.37.88:80)
    
    -                     38% |***********                    |   242k 00:00:01 ETA
    -                    100% |*******************************|   628k 00:00:00 ETA
    Connecting to adaway.sufficientlysecure.org (91.250.99.24:80)
    wget: server returned error: HTTP/1.1 404 Not Found 
    Seems the hosts file that used to be here has been removed. Edit your script to remove the
    adaway.sufficientlysecure.org file from the download. See if that solves your problem.

    - Mike
     
    Last edited: Mar 1, 2014
  14. ipse

    ipse LI Guru Member

    Stumped
    Feb 27 18:55:00 Moog user.notice root: DOWNLOADED http://winhelp2002.mvps.org/hosts.txt
    Feb 27 18:55:00 Moog daemon.info dnsmasq[521]: read /etc/hosts - 1 addresses

    which is localhost

    Everything is the same as a mth ago when it was working
     
  15. mraneri

    mraneri LI Guru Member

    Paste your whole script. Just want to double check what's there. There are a couple of different ways to do this, and without going through this thread history, I can't be sure that the script I'm using now is EXACTLY the same as what you have. (Though it is fundamentally the same script.)

    Also, paste your dnsmasq config options here as well.
     
  16. ipse

    ipse LI Guru Member

    I did paste it again on router just to be sure

    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep 127.0.0.1 | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/hosts\nlogger DOWNLOADED http://winhelp2002.mvps.org/hosts.txt\nkillall -1 dnsmasq" > /tmp/dlhosts
    chmod 777 /tmp/dlhosts
    /tmp/dlhosts
    fi
    cru a Gethosts "45 23 * * 4 /tmp/dlhosts"

    --------------------------------
    cat /etc/hosts
    0.0.0.0 localhost

    --------------------------
    cache-size=2048
    strict-order
    server=64.140.114.xxx
    server=8.8.4.4
     
    Last edited: Feb 28, 2014
  17. mraneri

    mraneri LI Guru Member

    Maybe your firmware version changed the handling of the hosts file. I guess your hosts file is being overwritten after being downloaded. Try this:

    Change the script to save the file into etc/adhosts instead of etc/hosts.
    This keeps your regular hosts file intact.

    THEN, add:

    addn-hosts=/etc/adhosts

    to your dnsmasq custom configuration...

    Let me know if that works.
     
  18. ipse

    ipse LI Guru Member

    Thanks for the suggestion mraneri - unfortunately the problem seems related to actually downloading anything from that location:

    Feb 28 10:38:26 Moog user.notice root: WAN UP Script Executing
    Feb 28 10:38:32 Moog user.notice root: DOWNLOADED http://winhelp2002.mvps.org/hosts.txt
    Feb 28 10:38:32 Moog daemon.info dnsmasq[521]: read /etc/hosts - 2 addresses
    Feb 28 10:38:32 Moog daemon.info dnsmasq[521]: read /etc/dnsmasq/hosts/hosts - 25 addresses
    Feb 28 10:38:32 Moog daemon.info dnsmasq[521]: read /etc/adhosts - 1 addresses

    -------------------------
    cat /etc/adhosts
    0.0.0.0 localhost

    cat /etc/hosts
    127.0.0.1 localhost
    192.168.xxx.1 Moog Moog-lan

    --------
    I should have mentioned (but was typing from a phone) that I run Shibby 116 (and I had 112 before, same thing).

    --------------------------

    The script now reads:

    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep 127.0.0.1 | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/adhosts\nlogger DOWNLOADED http://winhelp2002.mvps.org/hosts.txt\nkillall -1 dnsmasq" > /tmp/dlhosts
    chmod 777 /tmp/dlhosts
    /tmp/dlhosts
    fi
    cru a Gethosts "45 23 * * 4 /tmp/dlhosts"
     
  19. mraneri

    mraneri LI Guru Member

    Figured it out...
    My previous statement which stated 127.0.0.1 changed to 0.0.0.0 was ok was actually incorrect...
    the grep 127.0.0.1 in the script filters out any lines that DON'T contain 127.0.0.1. Which is all of them. The intent was to filter out comments and other wasted lines and save precious RAM on your router.

    change
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep 127.0.0.1 | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/adhosts\nlogger DOWNLOADED http://winhelp2002.mvps.org

    to:
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep .0.0. | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/adhosts\nlogger DOWNLOADED http://winhelp2002.mvps.org

    Should fix your problem...
     
  20. ipse

    ipse LI Guru Member

    Will try tonight when I get home...I disabled remote mgmt of the router :)
    I wish paid services would have the same response time as you - trust me, I work in support ....

    Big Thank you, I think you nailed it - are you going to edit the original post for monkeys like me who copy/paste without parsing and dissecting what each command does?
     
  21. mraneri

    mraneri LI Guru Member

    Done... Edited this post and Post #120.

    You also may want to look at Post #120 if you need to implement a whitelist.
     
  22. ipse

    ipse LI Guru Member

    Worked like a charm... even without the dnsmasq option.
    Thanks again :)
     
  23. mraneri

    mraneri LI Guru Member

    Glad you got it going...

    - Mike
     
  24. koitsu

    koitsu Network Guru Member

    A general suggestion: I'd suggest moving the adhosts file into the /etc/dnsmasq/hosts directory, where it will be automatically read by dnsmasq. How that works: the stock dnsmasq.conf (at least on Toastman) contains this line:

    Code:
    addn-hosts=/etc/dnsmasq/hosts
    
    If you read the dnsmasq docs, you'll see that if addn-hosts points to a directory (which in this case it does), dnsmasq will read/parse all the files in that directory. Do not tinker with /etc/dnsmasq/hosts/hosts (that's a file maintained by the firmware/GUI/NVRAM options itself).

    Confirmation of said layout:

    Code:
    root@gw:/# ls -ld / /etc /etc/dnsmasq /etc/dnsmasq/hosts /etc/dnsmasq/hosts/hosts
    drwxr-xr-x  17 root  root  208 Feb 21 17:06 /
    lrwxrwxrwx  1 root  root  7 Feb 21 17:06 /etc -> tmp/etc
    drwxr-xr-x  4 root  root  80 Dec 31  1969 /etc/dnsmasq
    drwxrwxrwx  2 root  root  60 Dec 31  1969 /etc/dnsmasq/hosts
    -rw-r--r--  1 root  root  55 Feb 28 16:14 /etc/dnsmasq/hosts/hosts
    
     
  25. mraneri

    mraneri LI Guru Member

    Good idea. When I get some time, I'll edit my script and update my previous posts. Thanks for the tip.
     
  26. FattysGoneWild

    FattysGoneWild LI Guru Member

    Looking forward to the updated new script. Flashing to Tomato very soon!
     
  27. FattysGoneWild

    FattysGoneWild LI Guru Member

    Any news for the updated script? Thanks! :) I know OP is really busy. No disrespect or trying to push.
     
  28. FattysGoneWild

    FattysGoneWild LI Guru Member

    Is it okay to use the original script and it still work fine with out it being updated?
     
  29. mraneri

    mraneri LI Guru Member

    I finally updated the script. A minor change, really. If it's working for you, no need to change anything, but for those setting this up for the first time, the new version doesn't require you to add the config line to the dnsmasq options.

    I updated the original, basic script in post #1, the script with whitelist support in post #120, and the script which concatenates two hosts files in post #184.

    Use whichever you please.

    Thanks for your patience. I spent 4 of the last 6 weeks on business travel 8000 miles from home...
     
    FattysGoneWild likes this.
  30. mraneri

    mraneri LI Guru Member

    I can't believe there's still interest in this 7 year old script. Glad people still get use out of it. (Me too..)
     
    FattysGoneWild likes this.
  31. FattysGoneWild

    FattysGoneWild LI Guru Member

    Heh the script works wonderfully and its legend now! Thank you thank you THANK YOU! Installed it and it works great. :) 8k miles of traveling. Wow. Good times.
     
  32. mraneri

    mraneri LI Guru Member

    Heh.. 8k each way.. Two times... Total air miles 33,812...

    Really glad people find it useful.
     
  33. FattysGoneWild

    FattysGoneWild LI Guru Member

    Question. I set the date/time. Since the new mvp hosts file came out today. I decided to change the date/time again in the script. That way I would have the newest hosts file again sooner. BUT. When I changed the time and date in the script. Then hit save. It did not download at that specific time. Example I set the cru time for 30 10 2. (10:30am Tuesday) If I set it right? Believe I did. Maybe once the script is set. You cannot change the time/date with out a reboot? And from then on. It will download once a week?
     
  34. mraneri

    mraneri LI Guru Member

    The schedule is established at boot time. Changing the schedule and resaving the script doesn't cause the script to actually run. So your new schedule will only take effect at next reboot.

    If you want to run the script "on demand", just type "/tmp/dlhosts" (without the quotes) in the command box under the "Tools -> System" menu in the router.

    Or, just reboot. The script establishes the schedule at boot time, but it also runs 2 minutes after the boot is finished.
     
    FattysGoneWild likes this.
  35. S-F12

    S-F12 Network Newbie Member

    Hi all. I'm trying to get this running and it's not working for me on an E2000.

    I've tried many iterations of the script all to no avail. I'm currently trying:

    cat > /tmp/whitelist <<EOF
    whitelist1.com
    whitelist2.com
    EOF

    if [ ! -s /tmp/dlhosts ] ; then

    echo -e "#!/bin/sh\n(wget -O - http://winhelp2002.mvps.org/hosts.txt ; wget -O - http://jamesisbored.com/iphone/content/hosts.php) | grep .0.0. | grep -Fvf /tmp/whitelist | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/dnsmasq/hosts/blkhosts\nlogger 'DOWNLOADED ADBLOCK HOSTS FILE(s)'\nkillall -1 dnsmasq" > /tmp/dlhosts
    chmod 777 /tmp/dlhosts
    sleep 120
    /tmp/dlhosts
    fi
    cru a GethostsFriAM "23 3 * * 2 /tmp/dlhosts"


    I have -> dnsmasq -> custom configuration: addn-hosts=/etc/blkhosts


    When I enter the command:
    ls -l /tmp
    /tmp/dlhosts

    I get this:

    -rwxrwxrwx 1 root root 314 Aug 19 11:12 dlhosts
    drwxr-xr-x 2 root root 460 Dec 31 1969 etc
    drwxr-xr-x 3 root root 60 Dec 31 1969 home
    drwxr-xr-x 2 root root 40 Dec 31 1969 mnt
    -rwx------ 1 root root 543 Aug 19 11:12 script_wanup.sh
    drwxr-xr-x 2 root root 40 Dec 31 1969 share
    drwxr-xr-x 11 root root 220 Dec 31 1969 var
    -rw-r--r-- 1 root root 30 Aug 19 11:12 whitelist
    /tmp/dlhosts: line 2: can't create /etc/dnsmasq/hosts/blkhosts: nonexistent directory
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    Connecting to jamesisbored.com (107.170.129.17:80)

    Any idea where I'm going wrong?
     
  36. koitsu

    koitsu Network Guru Member

    I've bolded the relevant part in question. You almost certainly need to change that to /etc/blkhosts to be consistent with your addn-hosts line.

    As for the actual error you see: this happens because there is no /etc/dnsmasq or /etc/dnsmasq/hosts directory. /etc is a symlink to /tmp/etc (and /tmp is RAM). So if you really wanted to use /etc/dnsmasq/hosts/blkhosts as your blkhosts file, you would need to add a /bin/mkdir -p /etc/dnsmasq/hosts to the script (probably before the sleep 120).

    Also, not sure who wrote that script, but the sleep 120 thing is such a hack and unnecessary if the script was properly written (to make sure it's not run more than once) and placed into WAN Up instead. This is just me babbling/ranting though.
     
  37. S-F12

    S-F12 Network Newbie Member

    Ahh. I see. Well, ummm. That's all mostly over my head. Sorry.

    So I changed all of what I had before and just copied the script in the first post and I'm still getting the:

    /tmp/dlhosts: line 2: can't create /etc/dnsmasq/hosts/blkhosts: nonexistent directory
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)

    Error
     
  38. koitsu

    koitsu Network Guru Member

    Okay, I'll dumb it down even more:

    Change every occurrence of /etc/dnsmasq/hosts/blkhosts in that script to /etc/blkhosts and your problem should go away.
     
  39. S-F12

    S-F12 Network Newbie Member

    Makes sense. Now I don't get the error but nothing happens...... :rolleyes:


    -rwxrwxrwx 1 root root 300 Aug 20 14:18 dlhosts
    drwxr-xr-x 2 root root 460 Dec 31 1969 etc
    drwxr-xr-x 3 root root 60 Dec 31 1969 home
    drwxr-xr-x 2 root root 40 Dec 31 1969 mnt
    -rwx------ 1 root root 530 Aug 20 14:18 script_wanup.sh
    drwxr-xr-x 2 root root 40 Dec 31 1969 share
    drwxr-xr-x 11 root root 220 Dec 31 1969 var
    -rw-r--r-- 1 root root 30 Aug 20 14:18 whitelist
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    Connecting to jamesisbored.com (107.170.129.17:80)

    And it stays like that until the end of time as far as I can tell.


    And I still see advertisements!

    This was a lot easier when I was using DD-WRT. Really it was the only thing that was easier though.


    EDIT:


    Forget it. It's working!

    Wow. Thanks for the help. Maybe the OP should be updated? Why would this script not work for me but it seems to for everyone else?
     
    Last edited: Aug 20, 2014
  40. koitsu

    koitsu Network Guru Member

    Considering this thread is 3 pages long, a brief skim shows that people are editing it to their needs and simply not disclosing what they've edited unless something breaks. For example, I see some people using /etc/hosts (a really bad choice -- shouldn't mess with that file), and a couple others using a different path. Those who edit it correctly and get things working are very unlikely to say "thanks, it works!" along with their modified version, they just stay quiet and say nothing. Welcome to "tech support" on the Internet. ;-)
     
  41. mraneri

    mraneri LI Guru Member

    So, I'm the one who wrote the original script. A few points:

    I've tried to keep the script updated as I use it, and have updated the first post on multiple occasions, most recently 4 months ago, 7 years after I originally posted the script. (Yes.. the first post was from 2007.) Note the revision history at the beginning of the first post.

    The correct path at some level depends on which flavor and which version of Tomato you are running. The optimal path has changed over time as dnsmasq has been updated.

    As for the 120sec delay, you're right. A hack is exactly what it is. And, if you check the original post, it SHOULD be placed in WANUP... But, still, I found with my hardware, when my system came back from a power outage, the router's WAN link would be alive before the modem was fully online and the download would fail. In this case, you would be without ad blocking until the next scheduled download. So, following the KISS Principle, the easiest solution was the delay. And, yes, for the first 2 minutes after WANUP the first time the router is booted, you're not ad-blocked. A small price to pay for the simplicity.
     

Share This Page