Auto DL Hosts File and Install...

Discussion in 'Tomato Firmware' started by mraneri, May 27, 2007.

  1. Bill_S

    Bill_S Network Guru Member

    I understand that if you can ping and get a response from a site that is on the list located here, http://winhelp2002.mvps.org/hosts.htm, that the script is not working (blocking). Am I correct or is there another way to test to see if the script is working. If I (we) have a specific way to test it would really be helpful.
    Thanks
     
  2. Bill_S

    Bill_S Network Guru Member

    I thought I had this mastered but now I find I cannot get this to work, I set the routers DHCP / DNS Server (LAN) to use internal DNS and I installed the script below in the WAN Up field. If I ping a site listed on the list I get the sites correct IP address not the 127.0.0.1 address.
    Here is the log showing the file was loaded:
    What I am doing wrong?
     
  3. Bill_S

    Bill_S Network Guru Member

    Ooops, I discovered my problem. I had the PC configured to use some public DNS servers not the routers. I am posting this in order to help others not make the same mistake. The PC has to be configured to obtain DNS server address automatically or use the routers IP address as the DNS server. Please correct me if I am wrong.
     
  4. teo yabgu

    teo yabgu Serious Server Member

    Hi,
    I have installed the script(the one with the whitelist) and it works like a charm. But there is one thing I couldn't figure out. I still get ads in gmail and I have check that the ads go to pagead2.googlesynication.com and I pinged that address I got back 127.0.0.1 so it seems working but actually it does not. Am I missing something? if somebody can help, would be appreciated.
    Btw, thanks for the great work on the script.
     
  5. mraneri

    mraneri Network Guru Member

    It's probably a cache problem, OR the ad isn't really being served from there.
    Clear your browser cache, and clear your DNS cache...

    Or, don't worry about it. Eventually it would disappear...
     
  6. teo yabgu

    teo yabgu Serious Server Member

    Thanks for fast reply mraneri. I have thought about that too and tried it with couple different computer in my house. I have tried with osx, windows and linux operating systems after I flush dns cache, still the same but as you said I will wait and meantime read more about it.
     
  7. teo yabgu

    teo yabgu Serious Server Member

    it has been more than a day and I have tried so many things but no result. I have found different versions of pixelserv and each one has different problem. The last pixelserv I am using now, downloaded through here.(pixelserv1.8) Ads are still coming up with gmail and I noticed when I use www.youtube.com, pixelserv works great but when I use https://www.youtube.com, ads are still blocked but pixelserv cannot replace the ads so "unable to load the page" warning shows up on the ads locations.(It works with safari but firefox and chrome still not working) any clue where I am doing it wrong. I have an Asus rt-n16 and using tomato 1.28.
     
  8. zbeyuz

    zbeyuz Serious Server Member

    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: started, version 2.66tomatofinalbf1 cachesize 10000
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset Tomato-helper auth
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: asynchronous logging enabled, queue limit is 10 messages
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq-dhcp[507]: DHCP, IP range 192.168.1.2 -- 192.168.1.254, lease time 1d
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: reading /etc/resolv.dnsmasq
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: using nameserver 208.67.222.222#53
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: using nameserver 8.8.8.8#53
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/hosts - 2 addresses
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/dnsmasq/hosts/hosts - 6 addresses
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.err dnsmasq[507]: failed to load names from /etc/blkhosts: No such file or directory
    Jan 1 07:00:11 RT-C0C1C08823A0 daemon.info dnsmasq-dhcp[507]: read /etc/dnsmasq/dhcp/dhcp-hosts
    Apr 23 23:58:00 RT-C0C1C08823A0 daemon.notice miniupnpd[997]: version 1.8 started
    Apr 23 23:58:00 RT-C0C1C08823A0 daemon.notice miniupnpd[997]: HTTP listening on port 52888
    Apr 23 23:58:00 RT-C0C1C08823A0 daemon.notice miniupnpd[997]: Listening for NAT-PMP traffic on port 5351
    Apr 23 23:58:01 RT-C0C1C08823A0 daemon.info pppd[458]: System time change detected.
    Apr 23 23:58:01 RT-C0C1C08823A0 user.notice root: WAN UP Script Executing
    Apr 23 23:58:43 RT-C0C1C08823A0 cron.err crond[486]: time disparity of 22778938 minutes detected
    Apr 24 00:00:01 RT-C0C1C08823A0 syslog.info root: -- MARK --
    Apr 24 00:01:20 RT-C0C1C08823A0 user.notice root: DOWNLOADED ADBLOCK HOSTS FILE(s)
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/hosts - 2 addresses
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.info dnsmasq[507]: read /etc/dnsmasq/hosts/hosts - 6 addresses
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.err dnsmasq[507]: failed to load names from /etc/blkhosts: Permission denied
    Apr 24 00:01:20 RT-C0C1C08823A0 daemon.info dnsmasq-dhcp[507]: read /etc/dnsmasq/dhcp/dhcp-hosts


    Does anyone know why I can't get this script to work ?

    P/s: I am using Toastman Tomato firmware version: 1.28.0502.2 on Cisco E4200 v1.
     
  9. koitsu

    koitsu Network Guru Member

    chmod 777 /etc/blkhosts or possibly just chmod 644 /etc/blkhosts
     
  10. eahm

    eahm LI Guru Member

    Why is my favorite anti-ads script no longer working? I haven't use it in a long time and I've just tested it again but it's not blocking anything. Testing on Shibby 116.

    Thanks.
     
  11. FattysGoneWild

    FattysGoneWild LI Guru Member


    Does the original script from OP still work with this new change with in MVPS Hosts File?!?!?
     
  12. mraneri

    mraneri Network Guru Member

    EDIT: 2/28/2014
    DISREGARD THIS POST!!! IT'S ALL WRONG!!!!! KEEP READING FOR THE SOLUTION....

    ________________________________

    The script still works with the MVPS change. The script does a search and replace for 127.0.0.1 replacing with 0.0.0.0. Given the MVPS change, that hosts file simply doesn't contain any matched searches so that's no problem.

    Here's the problem now:

    Code:
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    
    -                      4% |*                              | 21450  00:00:23 ETA
    -                     64% |*******************            |   329k 00:00:01 ETA
    -                    100% |*******************************|   505k 00:00:00 ETA
    Connecting to jamesisbored.com (162.243.37.88:80)
    
    -                     38% |***********                    |   242k 00:00:01 ETA
    -                    100% |*******************************|   628k 00:00:00 ETA
    Connecting to adaway.sufficientlysecure.org (91.250.99.24:80)
    wget: server returned error: HTTP/1.1 404 Not Found 
    Seems the hosts file that used to be here has been removed. Edit your script to remove the
    adaway.sufficientlysecure.org file from the download. See if that solves your problem.

    - Mike
     
    Last edited: Mar 1, 2014
  13. ipse

    ipse LI Guru Member

    Stumped
    Feb 27 18:55:00 Moog user.notice root: DOWNLOADED http://winhelp2002.mvps.org/hosts.txt
    Feb 27 18:55:00 Moog daemon.info dnsmasq[521]: read /etc/hosts - 1 addresses

    which is localhost

    Everything is the same as a mth ago when it was working
     
  14. mraneri

    mraneri Network Guru Member

    Paste your whole script. Just want to double check what's there. There are a couple of different ways to do this, and without going through this thread history, I can't be sure that the script I'm using now is EXACTLY the same as what you have. (Though it is fundamentally the same script.)

    Also, paste your dnsmasq config options here as well.
     
  15. ipse

    ipse LI Guru Member

    I did paste it again on router just to be sure

    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep 127.0.0.1 | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/hosts\nlogger DOWNLOADED http://winhelp2002.mvps.org/hosts.txt\nkillall -1 dnsmasq" > /tmp/dlhosts
    chmod 777 /tmp/dlhosts
    /tmp/dlhosts
    fi
    cru a Gethosts "45 23 * * 4 /tmp/dlhosts"

    --------------------------------
    cat /etc/hosts
    0.0.0.0 localhost

    --------------------------
    cache-size=2048
    strict-order
    server=64.140.114.xxx
    server=8.8.4.4
     
    Last edited: Feb 28, 2014
  16. mraneri

    mraneri Network Guru Member

    Maybe your firmware version changed the handling of the hosts file. I guess your hosts file is being overwritten after being downloaded. Try this:

    Change the script to save the file into etc/adhosts instead of etc/hosts.
    This keeps your regular hosts file intact.

    THEN, add:

    addn-hosts=/etc/adhosts

    to your dnsmasq custom configuration...

    Let me know if that works.
     
  17. ipse

    ipse LI Guru Member

    Thanks for the suggestion mraneri - unfortunately the problem seems related to actually downloading anything from that location:

    Feb 28 10:38:26 Moog user.notice root: WAN UP Script Executing
    Feb 28 10:38:32 Moog user.notice root: DOWNLOADED http://winhelp2002.mvps.org/hosts.txt
    Feb 28 10:38:32 Moog daemon.info dnsmasq[521]: read /etc/hosts - 2 addresses
    Feb 28 10:38:32 Moog daemon.info dnsmasq[521]: read /etc/dnsmasq/hosts/hosts - 25 addresses
    Feb 28 10:38:32 Moog daemon.info dnsmasq[521]: read /etc/adhosts - 1 addresses

    -------------------------
    cat /etc/adhosts
    0.0.0.0 localhost

    cat /etc/hosts
    127.0.0.1 localhost
    192.168.xxx.1 Moog Moog-lan

    --------
    I should have mentioned (but was typing from a phone) that I run Shibby 116 (and I had 112 before, same thing).

    --------------------------

    The script now reads:

    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep 127.0.0.1 | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/adhosts\nlogger DOWNLOADED http://winhelp2002.mvps.org/hosts.txt\nkillall -1 dnsmasq" > /tmp/dlhosts
    chmod 777 /tmp/dlhosts
    /tmp/dlhosts
    fi
    cru a Gethosts "45 23 * * 4 /tmp/dlhosts"
     
  18. mraneri

    mraneri Network Guru Member

    Figured it out...
    My previous statement which stated 127.0.0.1 changed to 0.0.0.0 was ok was actually incorrect...
    the grep 127.0.0.1 in the script filters out any lines that DON'T contain 127.0.0.1. Which is all of them. The intent was to filter out comments and other wasted lines and save precious RAM on your router.

    change
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep 127.0.0.1 | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/adhosts\nlogger DOWNLOADED http://winhelp2002.mvps.org

    to:
    echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep .0.0. | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/adhosts\nlogger DOWNLOADED http://winhelp2002.mvps.org

    Should fix your problem...
     
  19. ipse

    ipse LI Guru Member

    Will try tonight when I get home...I disabled remote mgmt of the router :)
    I wish paid services would have the same response time as you - trust me, I work in support ....

    Big Thank you, I think you nailed it - are you going to edit the original post for monkeys like me who copy/paste without parsing and dissecting what each command does?
     
  20. mraneri

    mraneri Network Guru Member

    Done... Edited this post and Post #120.

    You also may want to look at Post #120 if you need to implement a whitelist.
     
  21. ipse

    ipse LI Guru Member

    Worked like a charm... even without the dnsmasq option.
    Thanks again :)
     
  22. mraneri

    mraneri Network Guru Member

    Glad you got it going...

    - Mike
     
  23. koitsu

    koitsu Network Guru Member

    A general suggestion: I'd suggest moving the adhosts file into the /etc/dnsmasq/hosts directory, where it will be automatically read by dnsmasq. How that works: the stock dnsmasq.conf (at least on Toastman) contains this line:

    Code:
    addn-hosts=/etc/dnsmasq/hosts
    
    If you read the dnsmasq docs, you'll see that if addn-hosts points to a directory (which in this case it does), dnsmasq will read/parse all the files in that directory. Do not tinker with /etc/dnsmasq/hosts/hosts (that's a file maintained by the firmware/GUI/NVRAM options itself).

    Confirmation of said layout:

    Code:
    root@gw:/# ls -ld / /etc /etc/dnsmasq /etc/dnsmasq/hosts /etc/dnsmasq/hosts/hosts
    drwxr-xr-x  17 root  root  208 Feb 21 17:06 /
    lrwxrwxrwx  1 root  root  7 Feb 21 17:06 /etc -> tmp/etc
    drwxr-xr-x  4 root  root  80 Dec 31  1969 /etc/dnsmasq
    drwxrwxrwx  2 root  root  60 Dec 31  1969 /etc/dnsmasq/hosts
    -rw-r--r--  1 root  root  55 Feb 28 16:14 /etc/dnsmasq/hosts/hosts
    
     
  24. mraneri

    mraneri Network Guru Member

    Good idea. When I get some time, I'll edit my script and update my previous posts. Thanks for the tip.
     
  25. FattysGoneWild

    FattysGoneWild LI Guru Member

    Looking forward to the updated new script. Flashing to Tomato very soon!
     
  26. FattysGoneWild

    FattysGoneWild LI Guru Member

    Any news for the updated script? Thanks! :) I know OP is really busy. No disrespect or trying to push.
     
  27. FattysGoneWild

    FattysGoneWild LI Guru Member

    Is it okay to use the original script and it still work fine with out it being updated?
     
  28. mraneri

    mraneri Network Guru Member

    I finally updated the script. A minor change, really. If it's working for you, no need to change anything, but for those setting this up for the first time, the new version doesn't require you to add the config line to the dnsmasq options.

    I updated the original, basic script in post #1, the script with whitelist support in post #120, and the script which concatenates two hosts files in post #184.

    Use whichever you please.

    Thanks for your patience. I spent 4 of the last 6 weeks on business travel 8000 miles from home...
     
    FattysGoneWild likes this.
  29. mraneri

    mraneri Network Guru Member

    I can't believe there's still interest in this 7 year old script. Glad people still get use out of it. (Me too..)
     
    FattysGoneWild likes this.
  30. FattysGoneWild

    FattysGoneWild LI Guru Member

    Heh the script works wonderfully and its legend now! Thank you thank you THANK YOU! Installed it and it works great. :) 8k miles of traveling. Wow. Good times.
     
  31. mraneri

    mraneri Network Guru Member

    Heh.. 8k each way.. Two times... Total air miles 33,812...

    Really glad people find it useful.
     
  32. FattysGoneWild

    FattysGoneWild LI Guru Member

    Question. I set the date/time. Since the new mvp hosts file came out today. I decided to change the date/time again in the script. That way I would have the newest hosts file again sooner. BUT. When I changed the time and date in the script. Then hit save. It did not download at that specific time. Example I set the cru time for 30 10 2. (10:30am Tuesday) If I set it right? Believe I did. Maybe once the script is set. You cannot change the time/date with out a reboot? And from then on. It will download once a week?
     
  33. mraneri

    mraneri Network Guru Member

    The schedule is established at boot time. Changing the schedule and resaving the script doesn't cause the script to actually run. So your new schedule will only take effect at next reboot.

    If you want to run the script "on demand", just type "/tmp/dlhosts" (without the quotes) in the command box under the "Tools -> System" menu in the router.

    Or, just reboot. The script establishes the schedule at boot time, but it also runs 2 minutes after the boot is finished.
     
    FattysGoneWild likes this.
  34. S-F12

    S-F12 Network Newbie Member

    Hi all. I'm trying to get this running and it's not working for me on an E2000.

    I've tried many iterations of the script all to no avail. I'm currently trying:

    cat > /tmp/whitelist <<EOF
    whitelist1.com
    whitelist2.com
    EOF

    if [ ! -s /tmp/dlhosts ] ; then

    echo -e "#!/bin/sh\n(wget -O - http://winhelp2002.mvps.org/hosts.txt ; wget -O - http://jamesisbored.com/iphone/content/hosts.php) | grep .0.0. | grep -Fvf /tmp/whitelist | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/dnsmasq/hosts/blkhosts\nlogger 'DOWNLOADED ADBLOCK HOSTS FILE(s)'\nkillall -1 dnsmasq" > /tmp/dlhosts
    chmod 777 /tmp/dlhosts
    sleep 120
    /tmp/dlhosts
    fi
    cru a GethostsFriAM "23 3 * * 2 /tmp/dlhosts"


    I have -> dnsmasq -> custom configuration: addn-hosts=/etc/blkhosts


    When I enter the command:
    ls -l /tmp
    /tmp/dlhosts

    I get this:

    -rwxrwxrwx 1 root root 314 Aug 19 11:12 dlhosts
    drwxr-xr-x 2 root root 460 Dec 31 1969 etc
    drwxr-xr-x 3 root root 60 Dec 31 1969 home
    drwxr-xr-x 2 root root 40 Dec 31 1969 mnt
    -rwx------ 1 root root 543 Aug 19 11:12 script_wanup.sh
    drwxr-xr-x 2 root root 40 Dec 31 1969 share
    drwxr-xr-x 11 root root 220 Dec 31 1969 var
    -rw-r--r-- 1 root root 30 Aug 19 11:12 whitelist
    /tmp/dlhosts: line 2: can't create /etc/dnsmasq/hosts/blkhosts: nonexistent directory
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    Connecting to jamesisbored.com (107.170.129.17:80)

    Any idea where I'm going wrong?
     
  35. koitsu

    koitsu Network Guru Member

    I've bolded the relevant part in question. You almost certainly need to change that to /etc/blkhosts to be consistent with your addn-hosts line.

    As for the actual error you see: this happens because there is no /etc/dnsmasq or /etc/dnsmasq/hosts directory. /etc is a symlink to /tmp/etc (and /tmp is RAM). So if you really wanted to use /etc/dnsmasq/hosts/blkhosts as your blkhosts file, you would need to add a /bin/mkdir -p /etc/dnsmasq/hosts to the script (probably before the sleep 120).

    Also, not sure who wrote that script, but the sleep 120 thing is such a hack and unnecessary if the script was properly written (to make sure it's not run more than once) and placed into WAN Up instead. This is just me babbling/ranting though.
     
  36. S-F12

    S-F12 Network Newbie Member

    Ahh. I see. Well, ummm. That's all mostly over my head. Sorry.

    So I changed all of what I had before and just copied the script in the first post and I'm still getting the:

    /tmp/dlhosts: line 2: can't create /etc/dnsmasq/hosts/blkhosts: nonexistent directory
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)

    Error
     
  37. koitsu

    koitsu Network Guru Member

    Okay, I'll dumb it down even more:

    Change every occurrence of /etc/dnsmasq/hosts/blkhosts in that script to /etc/blkhosts and your problem should go away.
     
  38. S-F12

    S-F12 Network Newbie Member

    Makes sense. Now I don't get the error but nothing happens...... :rolleyes:


    -rwxrwxrwx 1 root root 300 Aug 20 14:18 dlhosts
    drwxr-xr-x 2 root root 460 Dec 31 1969 etc
    drwxr-xr-x 3 root root 60 Dec 31 1969 home
    drwxr-xr-x 2 root root 40 Dec 31 1969 mnt
    -rwx------ 1 root root 530 Aug 20 14:18 script_wanup.sh
    drwxr-xr-x 2 root root 40 Dec 31 1969 share
    drwxr-xr-x 11 root root 220 Dec 31 1969 var
    -rw-r--r-- 1 root root 30 Aug 20 14:18 whitelist
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    Connecting to jamesisbored.com (107.170.129.17:80)

    And it stays like that until the end of time as far as I can tell.


    And I still see advertisements!

    This was a lot easier when I was using DD-WRT. Really it was the only thing that was easier though.


    EDIT:


    Forget it. It's working!

    Wow. Thanks for the help. Maybe the OP should be updated? Why would this script not work for me but it seems to for everyone else?
     
    Last edited: Aug 20, 2014
  39. koitsu

    koitsu Network Guru Member

    Considering this thread is 3 pages long, a brief skim shows that people are editing it to their needs and simply not disclosing what they've edited unless something breaks. For example, I see some people using /etc/hosts (a really bad choice -- shouldn't mess with that file), and a couple others using a different path. Those who edit it correctly and get things working are very unlikely to say "thanks, it works!" along with their modified version, they just stay quiet and say nothing. Welcome to "tech support" on the Internet. ;-)
     
  40. mraneri

    mraneri Network Guru Member

    So, I'm the one who wrote the original script. A few points:

    I've tried to keep the script updated as I use it, and have updated the first post on multiple occasions, most recently 4 months ago, 7 years after I originally posted the script. (Yes.. the first post was from 2007.) Note the revision history at the beginning of the first post.

    The correct path at some level depends on which flavor and which version of Tomato you are running. The optimal path has changed over time as dnsmasq has been updated.

    As for the 120sec delay, you're right. A hack is exactly what it is. And, if you check the original post, it SHOULD be placed in WANUP... But, still, I found with my hardware, when my system came back from a power outage, the router's WAN link would be alive before the modem was fully online and the download would fail. In this case, you would be without ad blocking until the next scheduled download. So, following the KISS Principle, the easiest solution was the delay. And, yes, for the first 2 minutes after WANUP the first time the router is booted, you're not ad-blocked. A small price to pay for the simplicity.
     
  41. serendrewpity

    serendrewpity Network Newbie Member

    I know this is extremely old and I apologize if this is against the rules. I know mraneri hasn't been here in 2 years but I wanted to say thank you.

    A while ago, I looked at a different solution to ad-blocking involving WANUP Scripts and it had a Certificate that had to be generated in a text file but the entire script was too big for the WANUP script section of Tomato so I had to break it up in other sections. There was a lot more details but in the end it didn't quite work to a point that required no user intervention.

    I gave up on it until about a year ago. Then I discovered some Anti-Ad-Spyware-Malware-botnet DNS servers and thought I was golden ... discovered I wasn't when I did some nslookup tests over time. I didn't look into it again until recently. This time I discovered this thread. My hope is that it won't be so hard for the next person to find this.

    Read everything but it's all here. I followed the instructions and it worked perfectly. I am running Tomato Firmware 131 K26ARM USB AIO-64K on a Netgear Nighthawk R7000. The 120 second sleep delay can probably be lowered by 30s or so since routers have advanced so much since 2014. My Router is a quad-core overclocked to 1.2Ghz. It boots up to a functioning state with this script enabled in 2:10 minutes. [this also includes connecting to a VPN service which causes DNSMasq to reload] My wireless clients aren't connecting for another minute after that.

    I'm sure another solution could have been implemented by performing some sort of connection checking to shorten or eliminate the delay but that seems overkill and what would be the benefit? So someone could say it's 'Elegant'?

    To me this is efficient and takes care of business. Much like Mike Tyson used to come into the ring. No flash. No flare. Sneakers, Shorts and a hoodie. Knocked you out and done! All business. Results are always more impressive than pomp.

    The most impressive thing is that this appears timeless and it is amazingly still relevant. Even if you're a scripting novice like myself when you look at the script you can see how to add additional wget commands to include more host file URLs. As it stands this script is blocking +30K sites. I'm sure other Host URLS could be included to double or triple that number. The routers of today can handle it.

    Thank you, thank you.
     
  42. ThaCrip

    ThaCrip New Member Member

    The following script actually works on Tomato v1.28.1815 (June 27th 2010) which is basically the newest release of Tomato firmware…

    basically the OP's original script needs to be changed to look like these for the script to work...

    *apparently I can't post the entire script as it should be because this site keeps complaining I am posting a url but if someone could allow me to post the entire working script I would do it*

    basically you need the OP's script to look like "grep .0.0. > /etc/hosts" and the other part that needs to be changed is instead of "/etc/dnsmasq/hosts/blkhosts\nlogger" it now should read "/etc/hosts/blkhosts\nlogger". (without the " of course)

    using that actually works unlike the script text in the topic. I am surprised someone missed that as it was a pain in the butt for me to figure out because until I did things that way, while the script technically attempted to work, as you could see the log file in the Tomato router, it never actually took effect as it was "0 addresses” but after doing what I did it now shows “12715 addresses”.

    NOTE: in order for the router HOSTS file to work you need to have the router assign your DNS server. so basically if your using automatic stuff like DHCP/DNS everything will work (which means most people will be fine since they likely have everything automatic) but on my main computer I was using a manual IP address so the router itself is not assigning a IP address to my computer and I was also using manual DNS servers and having manual DNS servers addresses setup in Windows stops the HOSTS file from working but you can easily fix it as you can still have a manual IP address if you prefer but instead of typing your DNS server simply type in whatever IP you use to access your router and then the HOSTS file on the router filters the computer properly.
     
    Last edited: Mar 7, 2018
  43. mraneri

    mraneri Network Guru Member

    I'm still alive. I have been inactive, as I was happily running with the 2014 version firmware, and busy with life. This script has continued to work for me for now more than 10 years. I'm glad people are still finding it useful. Some specific requirement caused me to upgrade to the latest Shibby ver140 build, and now I'm using AIO since I'm running on an RT-N16 with 32MB Flash. This has ad-blocking built in, so I'm trying that. Seems to be working similarly, and while not particularly based on my original script, is similar.

    I think it's hard to know exactly what paths are best to use, as different versions of firmware seem to be looking in different locations for stuff.

    Goals of my original script were to keep it as small as possible since we are trying to keep it in NVRAM, that, and the fact that my router almost never reboots are the reason I never worked to improve the 120 second delay. There were many different adblock scripts over the years. I think this one may have been the first one. I guess it's always been the simplest. Again, for more than a decade, it just worked.

    Anyway, I just wanted to pop in and say I'm still alive, and well, and still using Tomato. Thanks for kind words and appreciation expressed over the years. I'm glad you all found value in what I did.
     
    ThaCrip likes this.
  44. ThaCrip

    ThaCrip New Member Member

    While there are different versions the one I am using is basically the last more official version of it to my knowledge. so would it be best to update your post to include the fixes so that it works on "Tomato v1.28.1815 (June 27th 2010)" build? ; or is what you did overall better for most people and that someone can just read my fix to get it working should they happen to use v1.28.1815?

    but to state the obvious... thanks for this as I wanted to do this stuff years ago but could never figure out how to get them working as they would never list the entries in the log file to confirm it was working as I nearly gave up on it and somewhat accidentally figured it out as I think I got to partially credit Koisu for his post above where he said, "Change every occurrence of /etc/dnsmasq/hosts/blkhosts in that script to /etc/blkhosts and your problem should go away." and noticed others messing with the 'grep' thing so I took a guess.

    but I see others are generally doing the "grep 127.0.0.1" etc. should I not be using the "grep .0.0. > /etc/hosts"?

    just making sure what I am doing is okay is all as I have no knowledge of linux etc. thanks for your time.
     
  45. mraneri

    mraneri Network Guru Member

    I'm not sure the details, but I just double checked the adblock script I was using as of last week when I upgraded to shibby 140, and it still used the /etc/dnsmasq/hosts/blkhosts path. Before shibby, I was using Toastman version from 2014. So that Toastman from 2014 worked with /etc/dnsmasq/hosts/blkhosts. I guess at this point I would suggest someone try the script as posted and if it doesn't work, they can possibly check the logs to figure out whether they need to change the path.

    Thanks for the kind words from you and others. Makes me feel good that even after all these years, this is still valuable to people.

    0.0.0.0 would return nothing faster than 127.0.0.1 at least back in the Windows XP days. That and it's 2 bytes less per entry in RAM, and with limited RAM especially on some of the earlier routers, this was not insignificant. (I think I started on a Buffalo router with 16MB of RAM.)

    Anyway, as I said, worked for me for almost 11 years.
     
  46. koitsu

    koitsu Network Guru Member

    I can't speak for other firmwares, but on Toastman, as of this writing, dnsmasq will read any/all files placed in the /etc/dnsmasq/hosts/ directory, and assume them to be of the standard hosts syntax of "IP hostname" (where space can be one or more spaces, or one or more tabs). Proof of that directory and its contents (default):

    Code:
    root@gw:/tmp/home/root# ls -ld /etc/dnsmasq/hosts /etc/dnsmasq/hosts/hosts
    drwxrwxrwx    2 root     root            60 Dec 31  1969 /etc/dnsmasq/hosts
    -rw-r--r--    1 root     root            55 Mar  7 20:48 /etc/dnsmasq/hosts/hosts
    
    This is accomplished by the stock default directive addn-hosts=/etc/dnsmasq/hosts that is part of the firmware-generated /etc/dnsmaq.conf. Proof:

    Code:
    root@gw:/tmp/home/root# grep addn-hosts /etc/dnsmasq.conf
    addn-hosts=/etc/dnsmasq/hosts
    
    And quoting dnsmasq documentation (bold+underline emphases are mine):

    You will also find that there is a firmware-generated file called /etc/dnsmasq/hosts/hosts which contains some self-generated information taken from the GUI and some automated stuff. Example: when Basic -> Identification -> Hostname is set to "gw" and the router's LAN IP is 192.168.1.1, this is what you end up with:

    Code:
    192.168.1.1 gw
    {your-wan-ip} wan-ip
    {your-wan-ip} gw-wan
    
    Please do not tinker with the /etc/dnsmasq/hosts/hosts file -- the firmware maintains this itself. If you want to put your own files in that directory (ex. /etc/dnsmasq/hosts/mystuff), go right ahead -- dnsmasq will read it when given a SIGHUP signal (see dnsmasq documentation for verification; NOTES section, first paragraph).

    I haven't read the script, but a filename syntax of (minus quotes) "/etc/dnsmasq/blkhosts\nlogger" doesn't make sense for several reasons (putting newlines in a filename = bad!).

    My guess is the script is using echo -e or printf to echo some data into a shell script itself, in which case the filename is /etc/dnsmasq/blkhosts -- which won't be read by dnsmasq by default, so you'd need addn-hosts=/etc/dnsmasq/blkhosts in your Dnsmasq Custom Configuration in the GUI) -- and logger happens to be the command that logs the remaining portion of the line (omitted in reply #243) to syslog.
     
    Last edited: Mar 8, 2018
  47. mraneri

    mraneri Network Guru Member

    The original script was intedned to be pasted into the startup script. That script uses echo -e to create the shell script that runs 120 seconds later and once per week to download the adblock lists. That should explain why you see the escaped newline. The script has been shown to work, and filepaths aside, must be syntactically correct.
     
  48. ThaCrip

    ThaCrip New Member Member

    http://www.polarcloud.com/tomato ; I used the 'Tomato_1_28_ND.7z' file since I got a ASUS WL-520GU router which is 4MB flash with 16MB of RAM. that's what I currently have on my router and does not work with your script as it is on the initial post in here.

    any idea whether Toastman would work on that router? ; or am I better off not touching it? ; because I know with DD-WRT things can be a bit picky and I like the simplicity of Tomato etc. but looking at the Shibby stuff... apparently it lists my router there and says I can use k24 or k26 builds (seems k24 has Dec 25th 2014 is the newest build (i.e. tomato-ND-1.28.5x-124-VPN.trx from 'build5x-124-EN' folder (I assume EN means English language(?)))). are these generally stable because my current setup is nice and stable without any issues. but then again if these are good enough I may shift to them as then I can use your script as it is which is probably a bit safer than my random script modifications, right?

    but apparently I would imagine the Toastman/Shibby versions are superior to the older build I am using from 2010?

    one last thing... should I not be using the "grep .0.0. > /etc/hosts" entry? ; or is that not going to hurt anything?

    thanks for your time.

    p.s. I did try the script as posted but it simply does not load the MVPS HOSTS entries and checking the basic log file through Tomato's interface, I don't see anything obvious there either.

    SIDE NOTE: I do know I do have a older Linksys router laying around that I think has 8MB flash with 32MB of RAM though but it's collecting dust at the moment.
     
  49. koitsu

    koitsu Network Guru Member

    Asus WL-520GU is a 16MB RAM / 4MB flash MIPSr1 router. I have no idea how much NVRAM it has. This will not work with present-day Toastman firmware (maybe one from early 2016, but nothing recent; the stuff in the ND folder would be what you'd want).

    It might work with Shibby -- you'd want the K26 series (not K26RT-N), and would absolutely need to stick with the Mini firmwares, specifically tomato-K26-1.28.RT-MIPSR1-xxxx-Mini. I don't know if it'll work though, because the file size is incredibly close to 4096KB (the resulting .trx file is 3,997,969 bytes); there is some overhead on the flash as well. There are circa 2008 posts on forums talking about this router having problems with the WAN interface (specifically some internal switch/port mapping/naming problem), so YMMV. Several of the Asus WL-xxxx routers have "weirdness" like this (often requiring unique firmware builds for the exact model).

    You're going to have a very hard time finding good firmware support for this given the limited flash and RAM, and even more so if you think you're going to be able to run large adblocking lists on it (odds of it exhausting RAM and kernel panic'ing (crashing) the router = high).

    I'm going to stay out of the script discussion, because given the content of this thread, I'm amazed at how many botched pathnames/filenames/etc. there are in several posts. I think mraneri's script is a good idea and convenient, but the thread itself is filled with so much chaos that it's no wonder people have problems.

    P.S. -- You should not ever overwrite the /etc/hosts file on Tomato. Tomato maintains this file itself, and it is used by the system (libc resolver functions will use it for hostname lookups first, followed by going out and doing network-level DNS queries). If you overwrite it (which is what > /etc/hosts is doing) then you stomp what Tomato puts in it natively. dnsmasq will read /etc/hosts as well. Basically what I'm saying is: "adblock hostnames" should really go into their own file in the directory /etc/dnsmasq/hosts/ directory (and that file can safely be overwritten as needed). Present-day Tomato will read/parse all the files in the /etc/dnsmasq/hosts directory. Do not mess with /etc/hosts or the file /etc/dnsmasq/hosts/hosts (this is not a typo!).
     
  50. ThaCrip

    ThaCrip New Member Member

    So the Shibby v24 ain't a bit safer than v26 builds? ; as the file size on those are... "3.30 MB (3,465,216 bytes)" (i.e. tomato-ND-1.28.5x-124-VPN.trx) which is from Dec 25th 2014.

    also for the k26 I see... tomato-K26-1.28.RT-MIPSR1-132-Mini.trx which is from Oct 19th 2015... "3.61 MB (3,792,896 bytes)" ; so this won't have trouble fitting on the 4MB flash, correct?

    also, the free RAM in my current Tomato firmware interface after running nearly 1 year straight still had about 2MB left of free RAM... with that info, would I still have RAM issues with the MVPS HOSTS file on it? ; because it seems it only needs about 1MB of free RAM if i recall correctly to run the MVPS HOSTS file.

    but if I can get the shibby build running... then I can just straight up use the OP's script as is, correct? ; hence, I won't have to worry about modifying anything and all should be good.
     
  51. koitsu

    koitsu Network Guru Member

    I don't know what exactly you mean by "safer", so I can't answer the question. "v24" is Linux 2.4.x (kernel version), "v26" is Linux 2.6.x. Linux 2.6.x is a newer version of Linux than 2.4.x.

    Ideally, yes, but I make no promises. The only "small-capacity" routers I've ever used were the original WRT54G, WRT54GL, and WRTSL54GS. There have been many posts over the years of the firmwares becoming too large to fit on small-capacity routers, and the errors seen are either a) an upgrade attempt fails early on (i.e. no risk), b) the firmware claims to flash/update OK but then the router acts wonky (e.g. bricked). Don't forget to erase NVRAM to factory defaults too when changing firmwares -- failure to do that can/will result in problems.

    This is getting very off-topic from the thread at hand, so I won't be answering these types of questions going forward in this thread.

    I have no way at this point to calculate it. You could try it and find out. What's going to happen if you run out of memory is one of two things: 1) random things will stop working (ex. the web interface, DHCP, DNS, etc.) as a result of the Linux kernel killing off processes to try and make more memory available (this is called memory pressure, and the thing that's killing it is the Linux kernel, specifically what's called Linux OOM or Linux OOM killer), 2) router will reboot (a kernel panic, crashing due to memory exhaustion). We have seen both reported on this forum many times, and every time I've helped with it (it's hard to troubleshoot given the nature of the problem), the users literally disappear after I provide help.

    You can relieve these risks by using a firmware that offers USB support (the USB stack and related utilities result in larger firmware, however, so this may not be feasible on the WL-520GU despite it having a USB port), and using a small USB stick that includes a Linux swap partition + enabling swap through some custom commands in Scripts -> Init or through an autorun script (this also requires a Linux filesystem partition on the USB stick, not just swap). There are other forum threads on how to accomplish setting up swap on TomatoUSB, please search and use those. The end result will be a router that gets very slow during extensive memory pressure, but shouldn't crash or have things die off.

    I can't tell you for sure because this thread is a complete chaotic mess of tips/advice/stuff, so there is no "proper easy to use" version of it. You have to find the script, then read each of the posts/replies and modify the script appropriately. Here's what I can tell you right now:

    The destination filename for the hosts file that the script creates (read: the file that contains all the hostnames to block) SHOULD NOT be any of these:
    • /etc/hosts
    • /etc/dnsmasq/hosts/hosts
    • /etc/blkhosts
    It should be /etc/dnsmasq/hosts/filename_of_your_choice. A good example would be: /etc/dnsmasq/hosts/blkhosts
     
  52. ThaCrip

    ThaCrip New Member Member

    I should have been a bit clearer... I meant 'safer' as in less likely to fail on a older router like mine since it's a older kernel. plus, before I flashed, looking around online it seems people were using the K24 fine on it from articles back in 2012 etc. so I figured the build I used is a bit 'safer' than some of the slightly more recent builds.

    Well all appears to be good now in this regard, like you said above, because once I flashed "tomato-ND-1.28.7636Toastman-IPT-ND-Std.trx" to my ASUS WL-520GU and configuring the router to my liking I then applied the OP's script 'as is' (well I adjusted the time and day thing so we all don't hammer the MVPS HOSTS website all at the same time) on the initial page of this topic and all works perfect. 19.60KB used out of the 32KB on NVRAM side of things.

    so after applying the OP's script in the initial page and rebooting router, then I checked the log file on the router and it now shows...

    read /etc/hosts - 2 addresses
    read /etc/dnsmasq/hosts/blkhosts - 12715 addresses
    read /etc/dnsmasq/hosts/hosts - 3 addresses
    time disparity of 25343036 minutes detected

    so with this said... those who have older routers and can use the firmware I am using (or thereabouts) and there won't be any more confusion with the OP's script because it simply works straight up as he posted it. I am glad I upgraded my firmware as it just seems like a better version of what I was previously using so far and the script works as is.

    p.s. to play it extra safe before upgrading the firmware through Tomato's interface (the one from polarcloud.com/tomato website) I did a Administration > Configuration > Restore Default Configuration" and selected "Erase all data in NVRAM memory (thorough)". then after a reboot I went back to the "Administration > Upgrade" and selected the firmware file, which was "tomato-ND-1.28.7636Toastman-IPT-ND-Std.trx" (which is 3.21 MB (3,366,912 bytes)), and after upgrading I waited 3-5min to be safe, then went to the web interface page, and then did another 'Erase all data in NVRAM memory (thorough)" and after a reboot things looked a bit different as the default skin is different etc which probably means it was a good idea I cleared the NVRAM after the flash was successful because after reboot the interface still looked like the old firmware I was using from 2010. but now I can be confident I won't have any weird bugs due to a lack of properly clearing the NVRAM.

    SIDE NOTE: now I have over 7MB of RAM free at the moment where as after rebooting the firmware I was using from 2010 it was about 5.xMB free. also, seems DHCP is disabled by default and the router has no login info by default and there seems to be no wireless password by default now. so I had to turn on DHCP and enable the login information and wireless access password stuff with WPA2 Personal and AES stuff etc.
     
  53. koitsu

    koitsu Network Guru Member

    DHCP is disabled in Toastman by default (this is intentional; it's covered in the READ THIS CHANGELOG FIRST.txt file, although I don't know if he was including that back then. There are lots of very good reasons for it as a default). Wireless should have been set to some degree of defaults (particularly the SSID and security), but I could be wrong. I really can't remember code/bits from several years back. I really only focus on the Toastman-RT-AC and Toastman-ARM7 branches for more recent routers.

    Yup, you did the change/upgrade procedure correctly. Consider me a very happy camper -- a lot of people don't do the NVRAM reset when switching between firmwares (esp. when switching between authors' builds, ex. going between Toastman and Shibby), so I'm very happy you did it right. :)

    Which files dnsmasq read looks 100% good -- the /etc/dnsmasq/hosts/blkhosts file is what's relevant, as that contains your list of blacklisted hostnames that resolve to 127.0.0.1 or 0.0.0.0. You should expect to see less free RAM gradually over time (this is 100% normal on any *IX system) as the router gets used a bit more, as log files grow (they're kept in RAM but rotated out before they get too large), etc.. You can always look at the file to see what's in it (Tools -> System Commands -> cat /etc/dnsmasq/hosts/blkhosts), then try to do an nslookup or ping from a machine on your network for one of the entries to ensure you're getting back the correct IP that matches what's in the file.

    Enjoy!
     
    ThaCrip likes this.
  54. ThaCrip

    ThaCrip New Member Member

    Yeah, that stuff was thoroughly pounded in my head from the 'peacock' thread (i.e. https://www.dd-wrt.com/phpBB2/viewtopic.php?t=51486 ) back when I was using DD-WRT firmware years ago on my router.

    Thanks for that info.

    I will enjoy and thanks for the OP's time and the MVPS HOSTS script.

    p.s. I guess at this point, while everything is basically fine, I just wonder if newer Shibby builds are worth using over Toastman builds, since the Shibby builds are newer? ; but then again, I might just stick with what I got because as they saying goes, "if it ain't broke, don't fix it" ;) ; also, since DD-WRT recommended builds don't seem to change much (especially for these older routers) I wonder if there is any general recommended builds of the Toastman/Shibby versions? ; or are all of them pretty stable/reliable? ; I know you said you did not want to comment any further on this firmware stuff, so you can ignore this "p.s." section if you want to.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice