1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Auto DL Hosts File and Install...

Discussion in 'Tomato Firmware' started by mraneri, May 27, 2007.

  1. mraneri

    mraneri LI Guru Member

    EDITED 8/17/07: (01/27/09 - Added whitelist - read this post first, then go see post #120 in the same thread for instructions to add a whitelist.)
    EDITED 9/19/2011: Updated MVPS Hosts File URL location.
    EDITED 2/28/2014: Updated grep to handle 0.0.0.0 in the hosts file.
    EDITED 4/19/2014 - Updated path where hosts file is stored so it is automatically picked up by dnsmasq with no need for a configuration option.
    NOTE: Read this post and then see Post #120 and Post #184 for later versions of this script... All work. Choose the one that's right for you...

    Thought this may help some people... My Buffalo WRT-G54S downloads an ad-blocking hosts file into RAM and automatically installs it to be used by dnsmasq at bootup. Also, it stays updated by redownloading it weekly... So far, performance seems fine, though the HOSTS file at http://www.mvps.org/winhelp2002/hosts.htm has about 14000 entries, and takes about 1MB of your router's RAM. Make sure you have enough (I seem to have plenty left in my buffalo.)

    No need for JFFS, just do the following:
    Go to the Administration -> Scripts -> Wan Up script page
    Cut & Paste the following into the window (Read below about changing the cron time):

    Code:
    logger WAN UP Script Executing
    sleep 5
    test -s /tmp/dlhosts
    if [ $? == 1 ] ; then
        echo -e "#!/bin/sh\nwget -O - http://winhelp2002.mvps.org/hosts.txt | grep .0.0. | sed -e '2,\$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/dnsmasq/hosts/blkhosts\nlogger DOWNLOADED http://winhelp2002.mvps.org/hosts.txt\nkillall -1 dnsmasq" > /tmp/dlhosts
        chmod 777 /tmp/dlhosts
        /tmp/dlhosts
    fi
    cru a Gethosts "45 23 * * 5 /tmp/dlhosts"
    When the WAN comes up, this script checks to see if the downloading script is actually created yet. If the router was already running, and the WAN was previously up (i.e. you changed settings, or your WAN dropped out and came back), the script will already exist, which flags the hosts file has already been downloaded. If the router was just rebooted, the script doesn't exist yet (It is created in RAM). In this case, the script is echo'ed into a file. (Note that echo line one long line. it stops immediately before chmod, which is the first word on the subsequent line.) The chmod line makes that script executable, and the following line calls the script which does the download. More on that later. The last line, beginning cru schedules the download on a regular basis, to keep the hosts file updated. I think weekly is plenty. I RECOMMEND YOU CHANGE THE TIME IN THE cru STATEMENT, so we don't all hit the host download site at the same time. The above configuration triggers the download at 23:45 (11:45 PM) every Friday (0=Sunday, 5=Friday)... Change the 45, 23, and 5 to whatever you wish to have the download occur when you want.

    This is the script file that's created by the echo statement... The script that acutally downloads and installs the file. (This script is created by the script above, you don't have to paste it anywhere...)
    Code:
    wget -O - http://winhelp2002.mvps.org/hosts.txt | grep .0.0. | sed -e '2,$s/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' > /etc/dnsmasq/hosts/blkhosts
    logger DOWNLOADED http://winhelp2002.mvps.org/hosts.txt
    killall -1 dnsmasq
    The first line fetches the Hosts file, and filters out the lines that only contain comments. (Note, if you want to download some other hosts file, just change the url (2 locations) in the echo statement on the Wan Up script page.) This file is put in the right place where dnsmasq will look for it. [EDIT - 8/17/07 - This first line now filters comments at the end of the line, and also replaces 127.0.0.1 with 0.0.0.0 which seems to return faster on windows XP based systems, at least. Thanks to yaqui and sroquette for leading me to this enhancement.] The second line logs that the file was downloaded.
    The third line sends a signal to dnsmasq to re-read the hosts file. (Note, because of the -1, dnsmasq DOES continue running, and is not actually killed.)

    When it runs, you will see this in the log:
    Code:
    May 26 21:16:24 Router user.notice root: WAN UP Script Executing
    May 26 21:16:31 Router user.notice root: DOWNLOADED http://winhelp2002.mvps.org/hosts.txt
    May 26 21:16:32 Router daemon.info dnsmasq[104]: read /etc/hosts - 14194 addresses
    May 26 21:16:32 Router daemon.info dnsmasq[104]: read /etc/hosts.dnsmasq - 6 addresses
    I had a significant concern initially about the load this would place on the router. It seems there's not much of an effect, other than the ~1MB of memory required for the hosts file (with 14k hosts blocked) in the RAM filesystem and in dnsmasq. The dnsmasq documentation says that it has been tested (though not on a router, obviously) with hosts files with 1M records with no significant issues. Additionally, I can't detect any difference in DNS lookup performance.

    I hope this is helpful to people.

    - Mike
     
    Last edited: Apr 19, 2014
    eahm likes this.
  2. mraneri

    mraneri LI Guru Member

    Oh! Your computers will not use the router's dnsmasq (and will not use the router's hosts file) if you've instructed your router to pass your ISP's DNS servers directly to your PC.

    To make sure the hosts file in the router gets used:
    In Advanced -> DHCP/DNS, make sure "Use Internal Caching DNS Forwarder" is checked. (If it wasn't, check it, save changes, and reboot your PC.)

    Check the logs to see that "read /etc/hosts - 14194 addresses" is in there (with some similar number, depending on when you do this) and you should be set.


    if you want to be sure, from your PC, ping one of the sites on this list, and make sure the IP address comes up as 127.0.0.1.
    http://www.mvps.org/winhelp2002/hosts.txt
     
  3. yaqui

    yaqui LI Guru Member

    Wow nice scripting !
     
  4. Bonez116

    Bonez116 LI Guru Member

    Hi,

    have test your script on my Linksys WRT54G and it dosent work.
     
  5. mraneri

    mraneri LI Guru Member

    I can't help you without additional information.
    telnet into your router and execute the script manually. What do you see?

    also, look at your /etc/hosts file. How big is it before and after you run the script?

    type the script commands in one at a time in your telnet session and see what happens. Repeated here, this is the script that should run.

    Code:
    wget -O - http://www.mvps.org/winhelp2002/hosts.txt | grep .0.0. > /etc/hosts
    logger DOWNLOADED http://www.mvps.org/winhelp2002/hosts.txt
    killall -1 dnsmasq
     
    Last edited: Mar 1, 2014
  6. Mandrake333

    Mandrake333 LI Guru Member

    Looks pretty cool - just tried it on my WRT54GL... works like charm :)
     
  7. mraneri

    mraneri LI Guru Member

    Glad to hear. Still is reliable for me too, downloading and re-installing the updated hosts file weekly and on reboot. Still don't really see any performance impact, other than the memory used. But seems there is still plenty of RAM available.

    - Mike
     
  8. Edhel

    Edhel LI Guru Member

    Seems to be working for me also. Hopefully it updates itself automatically. :)

    I was wondering if it would be possible to do something similar with an ipfilter.dat file that could be downloaded from a web server also. Not really familiar with scripting, so any help is appreciated. The data structure of the one I have is different, so I'm not sure it could be used like this.

    Brief example of the data structure below...

    222.161.022.012 - 222.161.022.015 , 000 , CHANGCHUN CITY,POLIC SCHOOL,
    222.161.036.032 - 222.161.036.035 , 000 , CHANGCHUN CITY,CHENGXIANGGL GOVERNMENT,
    222.161.036.036 - 222.161.036.039 , 000 , CHANGCHUN CITY,NANGUANSW GOVERNMENT,
    222.161.047.024 - 222.161.047.031 , 000 , CHANGCHUN CITY,WST GOVERNMENT,
    222.161.058.012 - 222.161.058.015 , 000 , CHANGCHUN CITY,CXJS GOVERNMENT,
    222.161.058.068 - 222.161.058.071 , 000 , CHANGCHUN CITY,CXJS GOVERNMENT,
     
  9. mraneri

    mraneri LI Guru Member

    Downloading a file should be the easy part. Search for wget, or, login to your router, and type wget. You will see the quick usage instructions for wget.

    What to do with the file once you download it, I have no idea.

    - Mike
     
  10. Hypernova

    Hypernova LI Guru Member

    This doesn't work if your software uses a proxy since it's the proxy that does the resolving. Is there a way to import this thing for the Access Restriction list?
     
  11. mraneri

    mraneri LI Guru Member

    I doubt it. Sorry it doesn't help you.
     
  12. MiseryQ

    MiseryQ Network Guru Member

    Thanks, I'll try this.

    My wife hates when I block ads AT her computer (;
     
  13. Macskeeball

    Macskeeball LI Guru Member

    I want to thank you for sharing this script. As you can see in another thread I had been trying to do the same sort of thing before I found your thread, but wasn't having any luck. The reason I'm posting this is to link the two threads together, and because in response to my thread someone who provides another such hosts file has now made it available in dnsmasq format.

    As I understand it, blocking by domain name is more efficient than blocking by hosts, because it also covers subdomains. If someone can get a script that takes advantage of the newer format, that would be great. I've tried (and posted code in the other thread) but again have not had any luck.
     
  14. Macskeeball

    Macskeeball LI Guru Member

    In the other thread, mraneri, yaqui, and myself have now successfully implemented a more efficient method of blocking ads on the router. This new method blocks by domain name rather than by hostname (ie. all subdomains are blocked). The result is that a shorter (and thus less computationally intensive, which is important for these routers) yet more effective block list can be used. For instructions, read the howto. Enjoy.

    PS- Mraneri, you may want to make a note of this in this thread's first post, for people who will read it, see a howto, and ignore the other posts. Here's a link to this post
     
  15. mraneri

    mraneri LI Guru Member

    Actually, I am still using the hosts file, not the dnsmasq list. I like the MVPS list (have been using it for some time now (years - since before it was on my router) and have not investigated the thoroughness of the dnsmasq list.

    MVPS works good for me, and dnsmasq documentation makes reference to HOSTS file testing with a million entries (granted on a more powerful machine). I see no performance degradation, and my router is up for 44 days now. Last time the MVPS list was downloaded (a few days ago) it had 15022 hosts in the list. While I haven't done a comprehensive analysis, it seems the MVPS list is considerably more complete.

    I'm not saying one way is better than the other, and perhaps at some point in the future I will switch. But for now, I'm sticking with my original method, and the MVPS list.
     
  16. Macskeeball

    Macskeeball LI Guru Member

    Again, that's a good thing. The reason the MVPS list has more items is because it is a hosts file. Hosts files do not affect all subdomains, just whole hostnames. A host file cannot block *.doubleclick.net for example. That's the point of the dnsmasq list. It doesn't need a large number of what are essentially duplicates. Also, your router may have better hardware than others' do. I'm not that familiar with Buffalo routers.
     
  17. mraneri

    mraneri LI Guru Member

    Maybe not.

    I DO understand how it works. Looking through the list last night, however, I figured there were a bunch of sites that were not blocked with the shorter list... And I DON'T want to get into a "My way is better than your way" argument, and as such, the following will be my last post regarding this particular discussion (aside from answering specific questions).

    I wrote a program and compiled some data comparing the two lists.
    MVPS List currently blocks 15021 hosts.
    The proposed dnsmasq list has 1973 entries, which block more IP's than that because you don't need these "duplicates".

    Of the 15021 hosts in the MVPS list, 3972 of those hosts would be blocked by the 1973 entries in the dnsmasq list.

    This leaves 11049 hosts which show up in the MVPS list UNBLOCKED by the new method.

    Granted, it is impossible to tell how many hosts from the new list MVPS is NOT blocking (I'm sure there are a significant number).

    To each his own, and I'm glad you guys were able to get it working. I know there's quite a sense of accomplishment, and I don't mean to take any of that away from you or the other contributors, however, I will stick with MVPS.

    By the way, there's no reason you can't have both. You could easily make a script that downloads both lists, and installs the dnsmasq options to block with the short list AND the HOSTS file to block the old fashioned way. There would be quite a bit of redundancy in the lists, but it's doable. Don't know what the memory requirements would be.
     
  18. Macskeeball

    Macskeeball LI Guru Member

    Fair enough. Each have their ups and downs, and the MVPS list is indeed very comprehensive. In the end, they're both very good ways to do a very good thing, and that is to block ads from an entire LAN in an auto-updating manner with no additional setup and without wasting bandwidth downloading unseen ads. Cheers to that, and I really do appreciate the help you've been giving in both threads.
     
  19. mraneri

    mraneri LI Guru Member

    It's my pleasure to help.
     
  20. Macskeeball

    Macskeeball LI Guru Member

    Now that I think about it, would you mind sharing that program you wrote that did the analysis to determine the hosts in the MVPS list that are not covered by the pgl.yoy.org list? That could actually be useful as a base for a program that could really help the maintainer of the pgl.yoyo.org list find servers to add.

    Imagine a script that did the following:
    1. Downloads the MVPS list and the pgl.yoyo.org list (called "pgl list" after this)
    2. Strips out the extra stuff from both.
    3. Generates a version of the MVPS list containing only hosts that the pgl list does not cover
    4. Analyzes the list to determine for each domain name (blah.com), how many hostnames that are just different subdomains exist.
    5. Convert the hostnames in the new MVPS list into just domain names (ie. strip subdomains, and remove duplicates)
    6. Resort the list so that domain names that had the most duplicates (and thus the servers most likely to be pure ad servers) are at the top of the list

    The final output of that script could then be manually looked at by the pgl list maintainer to see if the servers in it fit his criteria for being in his list, at which point he could add them.
     
  21. mraneri

    mraneri LI Guru Member

    The program I wrote only works one way..

    For instance, you may want to block ads.microsoft.com, and popups.microsoft.com, but not www.microsoft.com. Some script may decide that since there are two sites within microsoft.com to be filtered, to just filter the whole domain. So, any script like this may cause sites to be blocked that shouldn't be blocked.

    My script was very simple, and in fact, I stripped all the extra junk out of the files in a text editor. I didn't write code to remove comments and all the 127.0.0.1's.

    Here's the code, written in Visual Basic, provided without support. (whine all you want about VB. It works and it's quick... and I haven't taken the time to learn perl, tcl, or some other alternative.) I don't want to get into a huge long discussion about processing this list.

    Notes:
    the pgl.yoy.org list is in hosts2.txt (cleaned, of course)
    the MVPS list is in hosts.txt
    the output file will contain two columns:
    the MVPS site, and the matching pgl.yoy.org site. If there is no match, the second column is blank (CompareList(0)="")
    Code:
        Dim CompareList() As String
        ReDim CompareList(0 To 10000) As String
        Dim i As Long
        
        Open "c:\hosts2.txt" For Input As #1
        For i = 1 To 10000
            If EOF(1) Then Exit For
            Line Input #1, CompareList(i)
            CompareList(i) = UCase(CompareList(i))
        Next
        Close #1
        ReDim Preserve CompareList(0 To i - 1)
        
        Dim CompareItem As String
        Open "C:\hosts.txt" For Input As #1
        Open "C:\hostsout.txt" For Output As #2
        Do While Not EOF(1)
            Line Input #1, CompareItem
            CompareItem = UCase(CompareItem)
            For i = UBound(CompareList) To 1 Step -1
                If Right(CompareItem, Len(CompareList(i))) = CompareList(i) Then Exit For
            Next
            Print #2, CompareItem + vbTab + CompareList(i)
        Loop
        Close #1
        Close #2
    
    BTW, if you can't follow this code, you may be getting in over your head.

    [EDIT: I added the resulting file, so you can see which sites are blocked and which ones aren't]
     

    Attached Files:

  22. Macskeeball

    Macskeeball LI Guru Member

    Yeah, I'm not a Windows guy. Just Mac and Linux in this house, heh. Thanks anyway.

    The script would only generate a list for a human to manually look over and check out. It wouldn't be a fully automatic thing, but it would greatly simplify the work that the person had to do.
     
  23. yaqui

    yaqui LI Guru Member

    After you do this:
    Code:
    wget -O - http://www.mvps.org/winhelp2002/hosts.txt | grep 127.0.0.1 > /etc/hosts
    Why does 'vi /etc/hosts' show all "^M" in there??
    Code:
    # This *must* be the first line: 127.0.0.1     localhost  #^M
    127.0.0.1  localhost^M
    127.0.0.1  ad.a8.net^M
    127.0.0.1  asy.a8ww.net^M
    127.0.0.1  www.aaa-livedoor.net #[Trojan-PSW.Win32.Maran.ei]^M
    127.0.0.1  www.abcsearcher.com #[Spamdexing][Microsoft.Strider]^M
    127.0.0.1  abc-search.info^M
    127.0.0.1  abloga.info #[Spamdexing]^M
    127.0.0.1  www.abx4.com #[Adware.ABXToolbar]^M
    127.0.0.1  www.acezip.net #[Win32/Adware.180Solutions]^M
    127.0.0.1  phpadsnew.abac.com^M
    127.0.0.1  a.abnad.net^M
    127.0.0.1  b.abnad.net^M
    127.0.0.1  c.abnad.net #[IE-SpyAd]^M
    127.0.0.1  d.abnad.net^M
    127.0.0.1  e.abnad.net^M
    127.0.0.1  t.abnad.net^M
    127.0.0.1  adv.abv.bg^M
    127.0.0.1  bimg.abv.bg^M
    127.0.0.1  www2.a-counter.kiev.ua^M
    127.0.0.1  accuserveadsystem.com^M
    127.0.0.1  www.accuserveadsystem.com^M
    127.0.0.1  gtcc1.acecounter.com^M
    127.0.0.1  gtp1.acecounter.com^M
    - /etc/hosts 1/15371 0%
     
  24. mraneri

    mraneri LI Guru Member

    I suspect because the file is formatted for windows. Windows uses CHR(13)+CHR(10) for newlines. Unix uses only (I think) CHR(10). So, on a unix system, the CHR(13)'s show up as ^M's. The hosts file still works fine. They don't hurt anything, and I haven't found an easy way to get them out (nor do I care to.. It's no big deal.)
     
  25. yaqui

    yaqui LI Guru Member

    It looks like an easy job for sed.
     
  26. mraneri

    mraneri LI Guru Member

    Still... Why bother?
     
  27. yaqui

    yaqui LI Guru Member

    It is difficult because of a problem I describe below:
    Code:
    #!/bin/sh
    file=/etc/hosts
    sed '
    s/^M$//
    s/^M$/ /
    ' "$file" > /tmp/sedout
    sedout file is now a correctly formatted unix file.

    I see the problem trying to get Ctrl-v and Ctrl-m into a script you that you could cut and paste, I don't see anyway to do that. You have to manually go in the shell and enter them. :(

    Edit: Here I figured this out, this is copy and pastable
    Code:
    ##Clean up and convert MVPS list to Unix format
    sed -i -e 's/[[:cntrl:]]$//' -e 's/[[:cntrl:]]$/ /' -e 's/[[:space:]]*#.*$//' -e '/^#/d' -e '/^$/d' /etc/hosts

    ## Or This in order
    Code:
    #Download list into /tmp/tmplist
    wget -O /tmp/tmplist $mvpsurl
    
    ## use this format for sed: sed -i -e /remove lines that start with #/ -e /remove all blank lines/ -e /remove comments at end of lines/
    
    sed -i -e '/^#/d' -e '/^$/d' -e 's/[[:space:]]*#.*$//' /tmp/tmplist
    cut -d ' ' -f1,2,3 /tmp/tmplist > /etc/hosts
    That should also work I think; because cut won't cut and copy the control characters, will it?
     
  28. RobNC

    RobNC Network Guru Member

    yet another mod - b/c no HTTPD on 127.0.0.1

    I made yet another mod to this... I'm running lighttpd on my linux server (and didn't want to overload the router), so I added this:
    Code:
    sed -i -e '2,$s/127.0.0.1/192.168.12.23/g' /etc/hosts
    
    where my LAN subnet is obviously 192.168.12.x/24 and the linux machine running lighttpd is 192.168.12.23. It sure speeds up firefox (since my local PC - running Windows XP - isn't running a web server) since 404 errors are a lot quicker to return than timeouts.

    I see that the /etc/hosts file on the router is changed, but maybe the cache needs to be flushed or something (or router rebooted).
     
  29. mraneri

    mraneri LI Guru Member

    This is a good idea. Any way to get the router to handle these requests? i.e. route to 192.168.12.1? what would the router return, though?

    By the way, the router's DNS cache is flushed with the killall -1 call. You may still have to flush the cache in your PC, because your PC is likely caching your routers DNS responses too. "ipconfig /flushdns" should do it, on a PC.

    - Mike
     
  30. RobNC

    RobNC Network Guru Member

    I thought about that, but I don't have anything on .1, and if I use the router's IP, it'd give me the login prompt I guess. This is probably the disadvantage here - something has to return something on port 80, otherwise it'll just hog up connections on the web browser waiting for the timeout.

    Thanks, I forgot about that being part of ipconfig in windows :)
     
  31. yaqui

    yaqui LI Guru Member

    Too bad you can't have httpd that is in Tomato return a custom 404.

    Or somehow get lighttpd running in there.
     
  32. mraneri

    mraneri LI Guru Member

    I updated my initial post with changes stemming from the discussion which started at THIS POST (Post 29 was the key). Thanks to yaqui and srouquette for leading me there.

    - Mike
     
  33. yaqui

    yaqui LI Guru Member

    If it is slowing down your machine (for single home-user PC) you may also want to try this :
    Quoted from MVPS.org
     
  34. mraneri

    mraneri LI Guru Member

    Yeah, I don't think this happens... Of course, the hosts file is not on the windows machine. As far as windows is concerned, there is not hosts based ad blocking going on. The windows maching is submitting DNS requests. The router is returning a bogus address. Windows has no idea that the request was intercepted from a hosts file on the router.

    I have seen some slowdowns on WinNT with large hosts files before. This was another motivation for me to come up with this. No more hosts files on windows PCs.. The router handling this completly circumvents this windows "defect"
     
  35. yaqui

    yaqui LI Guru Member

    Isn't dns caching still happening though? Shutting off the client stops all of that - no need for it on a single user - home based machine.
     
  36. mraneri

    mraneri LI Guru Member

    Caching was happening before you installed the hosts file on your router. Sure, you can turn off the cache on your PC. But even if you keep the cache on, your Windows PC should perform the same with or without the hosts file loaded on the router (except your PC doesn't download junk with the hosts installed.)

    I leave my cache on... Theoretically, it should be SLIGHTLY faster in general. No need for my windows box to keep requesting the ip of www.google.com, and then to have the router request the same info of my ISP if it isn't in it's cache.

    Caching is generally a good thing, remember. (unless you're real short on RAM on your windows PC...)
     
  37. rajeshh

    rajeshh LI Guru Member

    Thanks to this thread, i was able to implement the script you have in my dd-wrt firmware buffalo gr-125. Though i am having some trouble getting the cron to work etc...

    Anwyays I was also able to use this perl script: http://www.everything2.com/index.pl?node_id=1372598

    to run a server on one of my servers on the network so I see clean pages. It works flawlessly for almost all the pages i visit.

    Btw, I was having trouble on my opera browser on My Nokia 770...though mozilla works fine on all my other computers ( as well as minimo on the Nokia 770).
     
  38. speedy3k

    speedy3k LI Guru Member

    is it possible to modify the script to remove the ^M from the file?
     
  39. speedy3k

    speedy3k LI Guru Member

    nevermind...answered my own questions.

    you can add this to the script to remove the cr/lf (^M) from the file as well.

    tr -d '\015\032'
     
  40. anectine17

    anectine17 Network Guru Member

    Where are you finding these settings?
    To make sure the hosts file in the router gets used:
    In Advanced -> DHCP/DNS, make sure "Use Internal Caching DNS Forwarder" is checked. (If it wasn't, check it, save changes, and reboot your PC.)
     
  41. mraneri

    mraneri LI Guru Member

    Umm... Are you asking or repeating???

    Down the left side of your Router Config, click "Advanced" (Right under "Basic")
    Then click "DHCP / DNS"
    Then make sure the first option is checked.


    No???
     
  42. anectine17

    anectine17 Network Guru Member


    What Router config are you talking about? Where? In what router?
     
  43. Macskeeball

    Macskeeball LI Guru Member

    He's talking about the web interface that you can use to change settings on your router. His instructions are for the Tomato firmware, since this is the Tomato forum.
     
  44. anectine17

    anectine17 Network Guru Member


    OHHHHHHHHHH............DUHhhhhhhhhhhhh......Sorry, was thinking DD-WRT. My bad. I'll shut up now!! :redface:
     
  45. anectine17

    anectine17 Network Guru Member

    For what it's worth, I installed the script on my WRT54G-TM (special, custom version for T-Mobile) with DD-WRT v.24 RC4 installed, and it works GREAT!! I was already blocking browser ads by using the AdBlock extension in FireFox, but this script also blocks ads in IE as well as blocking Yahoo Messenger and chat ads!! Awesome!! Thanks!!
     
  46. mraneri

    mraneri LI Guru Member

    Glad it's working for you in DD-WRT too!
     
  47. padapa

    padapa Guest

    Great script is right! I'm getting ready to install it on DD-WRT as well. A couple of questions for you.

    It looks like the script could be modified to load a locally held script if I wanted to blend together several black lists.

    Could you point to a locally held page/image on the WRT and serve that up? Would that be as fast as the localhost redirect?

    Can you use this in concert with the built in "keyword" blocking to deliver a level of basic porn filtering functionality for the family's PCs?

    Thanks again for the work you have done.

    Padapa
     
  48. mstombs

    mstombs Network Guru Member

    The great thing about using dnsmasq with a conf file in this threads way is that it does do domain blocking - the single entry

    address=/adnad.net/127.0.0.1

    should block all of those sites without using wildcards. Some users find 0.0.0.0 as a redirect faster.
     
  49. Macskeeball

    Macskeeball LI Guru Member

    You may want to look into the parental controls offered by www.opendns.com
     
  50. dontbotherme

    dontbotherme Network Guru Member

    First off- thanks for the script... it's working great.

    Although sometimes (once in a blue moon) I come across a blocked site that I actually need to access.
    Is there a way to temporarily set the host file back to default? Or temporarily/permanently whitelist the particular blocked site?

    Thanks
     
  51. Macskeeball

    Macskeeball LI Guru Member

    It's actually "temporarily," not "temperately."

    To do what you want, you can turn on SSH on your router and then log into it from that, then tell it to do this: mv /etc/hosts /etc/hosts_disabled && service dnsmasq restart

    When you're done, do the reverse: mv /etc/hosts_disabled /etc/hosts && service dnsmasq restart

    For a more permanent method, it may be possible to use a dnsmasq config option to override the hosts file (which is just used by dnsmasq anyway). That could go in the Dnsmasq config field in Advanced -> DNS/DHCP. I'm not sure if that would work, but it may be worth a try.
     
  52. dontbotherme

    dontbotherme Network Guru Member

    It's been a long day... thanks.

    Thanks, it worked.

    After trying to reverse it, I get: -sh: service: not found

    Is this correct, or is something wrong? Also, I just realized I was using Telnet, instead of SSH... does this matter?
     
  53. Macskeeball

    Macskeeball LI Guru Member

    Apparently I made a typo, leaving out the "v" in "service." It should be: mv /etc/hosts_disabled /etc/hosts && service dnsmasq restart

    SSH is secure, while telnet is completely insecure. Telnet really shouldn't be used these days.
     
  54. mraneri

    mraneri LI Guru Member

    If you've only enabled via the LAN, and your LAN is secure, is it a big deal? I use WPA2 with a random 64 character key, and telnet is not enable via the WAN..
     
  55. PeterT

    PeterT Network Guru Member

    I agree TOTALLY... Certainly, in an environment which involves insecure network links (ie acroos the Internet, across Wireless linls, or even in a corporate environment where data sniffing might be occurring) use of clear text passwords is not advisible; however, within a home, wired environment, I see NOTHING wrong with use of straight telnet.
     
  56. Macskeeball

    Macskeeball LI Guru Member

    I just like to be in the habit of using the secure technology, especially in situations such as this where there is no benefit to using the insecure one.
     
  57. Slavedriver

    Slavedriver LI Guru Member

    Well, for it to be actually more extensible you could incorporate a separate whilelist which you would just grep -v from hosts file on each update.
     
  58. tempralflux

    tempralflux Network Guru Member

    ping test

    Can I say first nice script but did a ping test on _www.emporn.com and got Destination specified is invalid
    ping stats for 0.0.0.0 total packet loss but when I enter the url in explorer and firefox the page is loaded. Is this just 1 that got away or have i set some thinf wrong.

    Thanks.
     
  59. mraneri

    mraneri LI Guru Member

    Maybe it was in your cache from before you installed the script? Don't know. destination is invalid is the proper response now since we replace 127.0.0.1 with 0.0.0.0 in the script (it wasn't originally that way)...

    No idea why it still comes up? I figure either the web page is in your browser cache, or the ip is somehow still in your DNS cache. I would suspect the former given ping comes up invalid.
     
  60. tempralflux

    tempralflux Network Guru Member

    All caches are cleared and yes a strange one.

    Thanks for the reply.

    EDIT:

    just entered address=/emporn.com/127.0.0.1 in advanced dhcp/dns Custom Configuration

    and the site is blocked.
     
  61. eangulus

    eangulus Network Guru Member

    Hi everyone. Well after wasting a full day trying to find the best and lightest script that actually works I have finally happy with the one in this thread.

    I do have a question thou. I see that you guys redirect the ad domains to 127.0.0.1 and then the script changes that to the 0.0.0.0 which for those who don't know is not only faster on XP but also faster for Vista in IE and FF. What I don't like too much is the page errors in all the ad locations. Is it possible to point to my own webserver but not the root directory but a file eg. admessage.html where I can make a custom message to replace the ads with.

    If you can do this can someone please report how.
     
  62. mraneri

    mraneri LI Guru Member

    I don't think there's a clean way to do that. Anyone else come up with a solution, I'm interested, but I don't think it's feasible.
     
  63. u3gyxap

    u3gyxap Network Guru Member

    You can substitute 0.0.0.0 with the IP or domain name of what your server would be, then on the server setup a custom default page, custom error page and custom not found page, which in fact are images.
     
  64. eangulus

    eangulus Network Guru Member

    I understand that part, but is there anyway of setting it up to point directly to a directory or a file on the server.

    Reason being is that my current web server is setup with multiple clients on it under different directories and I have it so that when you visit it shows the directories and therefore I can get to what ever site I want.

    So really I would like to not loose the directory listing on my web server.

    If you really really can't do it, then can someone give me some ideas to setup a redirect or something of the sort so I can have both. I am running Ubuntu 7.10 Server with apache
     
  65. srouquette

    srouquette Network Guru Member

    The host file from http://www.mvps.org/ doesn't seem to work anymore.
    It crashes my router everytime I download it...
     
  66. mraneri

    mraneri LI Guru Member

    Works fine for me with the original script... Two lines have errors, but I don't really care. Everything works as expected. I'm still using the original script I posted. (Keep it simple principle applies...)

    Code:
    Feb 25 18:10:37 Router user.notice root: DOWNLOADED http://www.mvps.org/winhelp2002/hosts.txt
    Feb 25 18:10:37 Router daemon.err dnsmasq[7519]: bad name at /etc/hosts line 12961
    Feb 25 18:10:37 Router daemon.err dnsmasq[7519]: bad name at /etc/hosts line 12961
    Feb 25 18:10:38 Router daemon.info dnsmasq[7519]: read /etc/hosts - 16759 addresses
    Feb 25 18:10:38 Router daemon.info dnsmasq[7519]: read /etc/hosts.dnsmasq - 9 addresses
    
    P.S. I'm still on 1.14 if that matters. Up 28 days. (since a power outage)
     
  67. srouquette

    srouquette Network Guru Member

    heh, you're right, I may try again this one :)

    edit: yep, works great :)
    and I still have plenty of free space to add something else in wan up ^_^
     
  68. dontbotherme

    dontbotherme Network Guru Member

    Never mind...
     
  69. fun.k

    fun.k Addicted to LI Member

    ah! thanks a ton mike !

    i was trying to block ads with another script but i could not get it to work..

    yours worked right for me instantly :)
     
  70. mraneri

    mraneri LI Guru Member

    Glad you like it. Thanks.

    - Mike
     
  71. eggsped

    eggsped Addicted to LI Member

    Thanks for the great script Mike!!
     
  72. Snoopyee

    Snoopyee Addicted to LI Member

    Hey Mike,

    I'm having trouble getting your script to work on my WRT-54GL ver 1.1 I got recently. I put the exact txt from copying and paste into the WANup section of Advanced, but when I save it and reboot, the ads still show up.

    This is what I get when I go to the Logs:

    Jun 12 01:20:09 router user.notice root: WAN UP Script Executing
    Jun 12 01:20:11 router daemon.info dnsmasq[112]: DHCPREQUEST(br0) 192.168.1.135 00:1f:e1:a9:2c:b0
    Jun 12 01:20:11 router daemon.info dnsmasq[112]: DHCPACK(br0) 192.168.1.135 00:1f:e1:a9:2c:b0 mypc
    Jun 12 01:20:14 router daemon.info dnsmasq[112]: DHCPREQUEST(br0) 192.168.1.135 00:1f:e1:a9:2c:b0
    Jun 12 01:20:14 router daemon.info dnsmasq[112]: DHCPACK(br0) 192.168.1.135 00:1f:e1:a9:2c:b0 mypc
    Jun 12 01:20:20 router daemon.info dnsmasq[112]: DHCPDISCOVER(br0) 169.254.59.121 00:50:18:5a:80:c9
    Jun 12 01:20:20 router daemon.info dnsmasq[112]: DHCPOFFER(br0) 192.168.1.139 00:50:18:5a:80:c9
    Jun 12 01:20:20 router daemon.info dnsmasq[112]: DHCPREQUEST(br0) 192.168.1.139 00:50:18:5a:80:c9
    Jun 12 01:20:20 router daemon.info dnsmasq[112]: DHCPACK(br0) 192.168.1.139 00:50:18:5a:80:c9 homepc
    Jun 12 01:20:22 router user.notice root: DOWNLOADED http://www.mvps.org/winhelp2002/hosts.txt
    Jun 12 01:20:23 router daemon.info dnsmasq[112]: read /etc/hosts - 17631 addresses
    Jun 12 01:20:23 router daemon.info dnsmasq[112]: read /etc/hosts.dnsmasq - 1 addresses
    Jun 12 01:20:56 router cron.warn crond[91]: time disparity of 20220980 minutes detected

    Is there something I'm doing wrong?? Please help! I'm using Tomato 1.19.1463 and I do have the option 'Use Internal Caching DNS Forwarder' checked on in Advanced -> DHCP/DNS with everything else not checked. I hope that is enough information for you to troubleshoot this problem as I'm not much of a coder. Thx
     
  73. mraneri

    mraneri LI Guru Member

    Hmmm. The script is working. The log shows the file is downloaded an processed correctly, and was successfully interpreted by dnsmasq... Why your PC isn't using it, I'm not sure.

    on the PC which is still receiving ads, open up a dos window (click start-> run, and type "cmd" without the quotes). in the DOS window, type "ipconfig /all". You should have only 1 DNS server listed, and it should be 192.168.1.1, the IP of your router. If it's something else, OR there's more than 1, post the entire "ipconfig /all" response (you can obscure your MAC, or anything else private you need).

    If you DO have only your router's IP listed, and you re-verified "Use Internal Caching DNS Forwarder" is checked, I would reboot the router and the PC and try again.

    Note, all the above assumes you're using DHCP to assign IP addresses to the PC's on your local network. If you are not, you need to make sure the DNS servers you configure on your local PC point to your router, and not your ISP's servers.

    - Mike
     
  74. Snoopyee

    Snoopyee Addicted to LI Member

    I think I figured it out by your reply, Mike. I think I have the pc set with the opendns.com dns addresses instead of the router's address. I guess the next question is will your script work with the router set to the dns addresses of opendns.org? TIA

    I'm not at the router's location right now so I'll try fixing my dns address on the pc to not point to the dns address for opendns.com
     
  75. srouquette

    srouquette Network Guru Member

    yes it works.
     
  76. mraneri

    mraneri LI Guru Member

    Yes, and this is actually how I'm using it. Make sure you put the two opendns addresses in the static DNS options on the Basic->Network configuration screen. Then, make sure, in Advanced->DHCP/DNS that "Use Received DNS with static DNS" is UNCHECKED.

    Then, make sure your PC's have the static DNS option removed, and get the DNS from DHCP.

    All should work from there. (Then all services on all computers will also use OpenDNS.)

    - Mike
     
  77. yaqui

    yaqui LI Guru Member

    Still the best, concise way of ad-blocking :)
     
  78. xcooling

    xcooling LI Guru Member

  79. mraneri

    mraneri LI Guru Member

    Thanks, yaqui... I follow KISS principle.. (Keep it simple)
     
  80. pumpkin

    pumpkin LI Guru Member

    I just can say. Thanks
     
  81. johnjinsf

    johnjinsf Addicted to LI Member

    I also wanted to say thank you to mraneri for the HOSTS file on the router.

    I had Tomato 1.21 Speed Mod firmware (from Tourist in Paradise site) running on my Linksys WRT54G and changed it to include the script.

    There is only one thing that is an annoyance, rather than a problem.

    I usually read an online newspaper every day and it sems that where an advert has been blocked and the IE message displayed in its place, and when I want to go Back to the previous page, if say there had been 7 adverts blocked, I have to hit the Back key/button 7 times, or either click the "Recent Pages" button which displays the page I am on and each individual block - I hope I have explained that without confusing you.

    Has anyone else experienced this.

    Thanks again to mraneri for a superb posting

    John J
    San Francisco
     
  82. mraneri

    mraneri LI Guru Member

    No. Can't say I have. Do you have to click back multiple times on other websites? What website are you having trouble with?

    - Mike
     
  83. johnjinsf

    johnjinsf Addicted to LI Member

    Thanks for replying so quick.

    This is the only website where it has stood out - http://www.theage.com.au and only
    using Internet Explorer 7. Firefox 3.01i is OK.

    I grabbed some screen shots and merged them and will attempt to attach

    Thanks again
     

    Attached Files:

  84. mraneri

    mraneri LI Guru Member

    I get the same thing with IE6. I have no idea how that web site is doing it's ad insertion, but it's definitely weird. Sorry I can't help. When some other sites do zero delay redirects, I usually shift click links, to open in a new window. Otherwise, when I click back, I immediately get re-directed again. So by opening in a new window, when I want to go back, I just close it. You may want to get into a similar habit for this web site.

    Good luck, and sorry I can't help more.

    - Mike
     
  85. SM188

    SM188 Addicted to LI Member

    Code:
    Oct  9 12:25:20  user.notice root: WAN UP Script Executing
    Oct  9 12:25:21  user.info rcheck[168]: Activating rule 1
    Oct  9 12:25:30  user.notice root: DOWNLOADED http://www.mvps.org/winhelp2002/hosts.txt
    Oct  9 12:25:31  daemon.info dnsmasq[121]: read /etc/hosts - 15353 addresses
    Oct  9 12:25:31  daemon.info dnsmasq[121]: read /etc/hosts.dnsmasq - 0 addresses
    Oct  9 12:25:42  cron.warn crond[99]: time disparity of 20393005 minutes detected 
    
    So did I do it right? If I did any site you wanna show me so I can test this out?

    Thanks
     
  86. mraneri

    mraneri LI Guru Member

    Yup. The log looks perfect. It really is that easy...

    If you go to http://ad.doubleclick.net you should get a Server not found error message. If you get a webpage which says: "This is a DoubleClick advertising server" with some links, then the blocking is not working.

    In that case, in tomato configuration, go to Advanced -> DHCP/DNS and make sure "Use internal caching DNS forwarder" is checked.

    If it is, and ads are still not being blocked, make sure your PC is not configured with a static IP address. If it is, you've probably configured the DNS server on your local PC directly, and are bypassing your router altogether (for DNS).
     
  87. SM188

    SM188 Addicted to LI Member

    yup it works. I did the manual thing installing the individual host files on my comps/laptop. But I had one comp I didn't do it to and that site was blocked.

    Thanks
     
  88. rajeshh

    rajeshh LI Guru Member

    look at my post #37 for a perl script thats a simple http server that will just return a blank 1X1 png for all requests. I run this on a nslu2 slug and it works well, except at times, I will get a "Connection has been reset" error message in the places where the ad was...it seems maybe the slug is getting overhwhelmed or something. Earlier I used to run this on a more powerful linux box and that would be fine..
    Any ideas?
     
  89. johnjinsf

    johnjinsf Addicted to LI Member

    Have you seen the HostsMan product?
    I have not seen it working but I came across it the other day in an article.

    In case you have not heard about it, HostsMan is a freeware hosts file manager and editor and it allows for auto update.
    It is on all the 'usual' download sites but the home page is at http://www.abelhadigital.com/

    There is a forum at http://forum.abelhadigital.com/index.php?showforum=2 and that is where I asked if it would be possible to have an option for HostsMan to read/write to the router.
    The added benefit would be the ability to make changes to the file as needed. I had one site recently that I wanted to go to but it was in the router file, and so I just plugged the DSL modem into my laptop, bypassing the router for short time.

    Anyway the product owner wrote back today to say he'll put it on his list of "To Do's" but it will be a low priority.

    Just thought I would pass this information to you as something that may or may not happen at some point in time.

    Thanks

    John
     
  90. Dashiell

    Dashiell Network Guru Member

    If I may add my $.02

    I've been using this script from the OP in Tomato for a while now, it works really, really well! Simple and effective! One aspect I've noticed...


    The script changes all of the 127.0.0.1 entries to 0.0.0.0 in order to speed up the "timeouts" from a Windows (XP mostly) machine. This works reasonably well but every now and then I notice a workstation being a bit "slow to resolve." I've made one change to the setup (not the script) that cleared up the problem...

    Since the script changes all of the blocked pages to 0.0.0.0 it is actually (in tomato) pointing them toward what is defined on the "Basic: Network" page of tomato's gui as the WINS server (unless, of course you're actually using a wins server and have it set to something else... in which case this whole post does not apply to you ).

    I had a feeling (erroneously or not) that this might be the cause of the lengthened timeouts, so I simply changed the WINS setting in the menu to 0.0.0.1

    I haven't seen the "slow to respond/resolve" issue on any XP machine since then.

    Give it a try if you experience the same issue.
     
  91. johnjinsf

    johnjinsf Addicted to LI Member

    I use Static IP addresses and so the Basic Network screen does not show the WINS field because DHCP is not selected.

    I tried changing DHCP to yes, changing WINS to 0.0.0.1 as mentioned above, saving configuration, changing DHCP to no and saving configuration.

    I then changed DHCP to yes and WINS was back at 0.0.0.0 - it looks like the router changed WINS back to 0.0.0.0 when I reset DHCP to no.

    Can anyone offer a solution to get WINS to be 0.0.0.1 whilst using Static IP addresses?

    Thanks
     
  92. rajeshh

    rajeshh LI Guru Member

    Btw, I am not able to play certain videos from sites like nfl.com and such because ad.doubleclick.net is blocked. has anyone found a workaround? I googled, and the only solution seems to be to whitelist this domain....
     
  93. callous

    callous Network Guru Member

    does the router slow down much using a host file?
     
  94. mraneri

    mraneri LI Guru Member

    No. My router handles DNS requests with no apparent impact. The dnsmasq website also indicates dnsmasq has been optimized to work with large hosts files.
     
  95. callous

    callous Network Guru Member

    I ran the script and checked the log:

    Jan 18 21:17:58 unknown user.notice root: WAN UP Script Executing
    Jan 18 21:18:11 unknown user.notice root: DOWNLOADED http://www.mvps.org/winhelp2002/hosts.txt
    Jan 18 21:18:12 unknown daemon.info dnsmasq[116]: read /etc/hosts - 15540 addresses
    Jan 18 21:18:12 unknown daemon.info dnsmasq[116]: read /etc/hosts.dnsmasq - 8 addresses
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: user root: parse error at 18
    Jan 18 21:18:15 unknown cron.err crond[98]: time disparity of 20538858 minutes detected


    What's with the crom.err.crond[98] errors, and the time disparity of a zillion minutes?

    I also checked the host file residing in the router after this script of was run, and got a 0.0.0.0 somesite^M instead of 127.0.0.1 somesite^M

    I take it that it's not working for me?
     
  96. mraneri

    mraneri LI Guru Member

    I do not believe the cron errors have to do with the script, unless there was an error with how you edited the cron line to customize the download time. Note, the script completed 3 seconds before the errors, and the errors are coincident with the time disparity warning.

    Also, there is an element (sed) in the script which replaces 127.0.0.1 with 0.0.0.0, as some systems return the error page faster with 0.0.0.0 than 127.0.0.1. The script appears to be working on your machine.
     
  97. callous

    callous Network Guru Member

    ok thanks!
     
  98. smilespray

    smilespray Addicted to LI Member

    This works wonders for me, and THANKS for the elegant solution. However I have problems downloading podcasts from podkast.nrk.no. I have noted a block for ad.nrk.no, but I am not sure this is the problem.

    Running latest stock Tomato release.

    Dump from Terminal.app on a Mac:

    mymac:~ <userwithheld>$ host podkast.nrk.no
    podkast.nrk.no is an alias for nl.nrk.no.
    nl.nrk.no has address 160.68.207.168

    Have read through the entire thread but some of the posts went *swoosh* over my head.

    Any ideas?
     
  99. Dashiell

    Dashiell Network Guru Member

    I never thought to ask this, and maybe there's a fundamental answer I'm clueless about, but here goes:

    Does the number of entries in the hosts file conflict with dnsmasq cache settings? For instance, I run a simple dnsmasq script to utilize 1024 cache entries as opposed to the 150 default limit. Would that mean that only the first 1024 entries in the host file will be seen?

    My instinct tells me that the host file entries and the dnsmasq script are separate entities, with the cache file pulling addresses dynamically as the surfing is done. I guess I'm looking for confirmation of this.
     
  100. mraneri

    mraneri LI Guru Member

    I don't know which block is causing the problem, and frankly have little knowledge of current website/html technology. I can't help you. It's definitely likely that the fact that you have configured the router via this script to return bogus data for known ad sites is the cause of the problem with the podcasts. Short of disabling the script, I can't offer any assistance to your or similar problems.

    Sorry.

    - Mike
     

Share This Page