1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Script: Clean, Lean and Mean Adblocking

Discussion in 'Tomato Firmware' started by haarp, Apr 23, 2013.

  1. jerrm

    jerrm Network Guru Member

    OK - I had typed in a GET request before and got back a response, but get the seg fault if I just hit enter, or type in something that isn't in the format of text[space]text.

    It's not pretty, but running under inetd, that shouldn't really matter too much as inetd will just kick off a new instance if nullserv bombs due to a malformed request. Would definitely need to be better behaved if it had it's own daemon mode though.

    For now I'm staying with pixelserv.
     
  2. mstombs

    mstombs Network Guru Member

    An inetd kicks off a new instance always - but it is bad code to segfault - possible null pointer for variable 'file' on line 216, and 'path' line 220, but it shouldn't get that far should check 'method' not NULL and not 'GET' and return a 501 earlier ...
    We should discuss pixelserv v31 maybe in the other thread, but is there a problem with the blocks added by the script in this thread to which pixelserv v30 could reply more effectively? I have seen javascript errors but not a lot can do if the page expects variables to be set by the blocked domain!
     
  3. HunterZ

    HunterZ LI Guru Member

    The above lines are given in the example config in the first post of this thread, but it appears that the comma does not get stripped out properly:
    Code:
    wget: server returned error: HTTP/1.1 404 Not Found
    ADBLOCK: Failed: http://hosts-file.net/download/hosts.txt,
    
    I think the proper fix may be to take out the comma.
     
  4. Toink

    Toink Network Guru Member

    I've been using the All-U-Need Adblocking script for some years now. All my hosts and the pixelserv files are stored and pulled from my dropbox account via the public folder. My question is, can I just do the same with your adblock script without having to utilize jiffs or a USB dive? - I mean just paste the script in the WAN up part of my admin's scripts?

    Thanks!
     
  5. Bird333

    Bird333 Network Guru Member

    I think the script is too big (past 4k in size) to put in WAN up.
     
  6. Toink

    Toink Network Guru Member

    Ah! I forgot about that! :( Bummer.

    Thanks!
     
  7. HunterZ

    HunterZ LI Guru Member

    Some additional thoughts after having used this for a few days:
    • Several of the included lists download every time the adblock script is run, and do not store a last-updated timestamp. I guess maybe the server doesn't provide a last-update time? It would be good if the script could somehow update these only once every so many hours, instead of every time.
    • The magic that lets pixelserv run on an alternate IP seems to get blown away when I mess with IPv6 settings in Toastman's Tomato builds. I then have to kill pixelserv, manually remove the /var/run/adblock.pid file, and re-run the adblock script (or reboot I guess?) to get things working properly again. I don't expect that this should be addressed, as it's my own fault that it happens, but it may be nice if the script could check on (and repair) the status of the iptables stuff and pixelserv when it runs.
    • It takes a good minute or so to sort the blocklist on my router, but my Linux workstation on which the files are being stored anyways can do it almost instantly. It would be nice if there were an easy way to split the script duties in two, so that the blocklist generation could be handled by my workstation and the dnsmasq/pixelserv maintenance handled by the router. All-U-Need had this, but its implementation was too clunky for me to get working.
    • There are some blocklists that only come in .zip file format (e.g. http://www.malwaredomains.com/?page_id=29). It might be nice if this script had support for this, perhaps by piping wget to unzip?
    • Edit: Also, the iptables magic does not handle HTTPS gracefully. I'm not sure if this is a problem that can be solved, but a lot of ad servers seem to be using HTTPS now for some reason, and this results in long load times while the browser waits for a connection timeout. I tried changing the iptables rule from DROP to REJECT, but it doesn't seem to actually result in different behavior for some reason.
     
  8. HunterZ

    HunterZ LI Guru Member

    Update: I'm not experienced with iptables, but here are some script modifications to the iptables sections of stop() and the bottom of the main script that seem to help with the HTTPS issue:
    Code:
            iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT &>/dev/null
            killall pixelserv &>/dev/null
            ifconfig $BRIDGE:1 down &>/dev/null
            iptables -D INPUT -i $BRIDGE -p tcp -d $redirip -j REJECT --reject-with tcp-reset &>/dev/null
            iptables -D INPUT -i $BRIDGE -p all -d $redirip -j REJECT --reject-with icmp-host-prohibited &>/dev/null
    
    ...
    Code:
                    iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
                            iptables -I INPUT -i $BRIDGE -p all -d $redirip -j REJECT --reject-with icmp-host-prohibited
                            iptables -I INPUT -i $BRIDGE -p tcp -d $redirip -j REJECT --reject-with tcp-reset
                            iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
                    }
    
    I don't think the icmp-host-prohibited REJECT rule does much, but it's probably better than the original DROP rule. The tcp-reset REJECT rule seems to be the one that causes blocked HTTPS URLs to time out faster.


    Edit: Also, here is a script that you can use to get a list of the hashes used to create the lastmod- and source- files for each source. Change "/cifs1/adblock/config" to the path of your config file, if needed:
    Code:
    #!/bin/sh
    
    source /cifs1/adblock/config
    
    for s in $SOURCES; do
            echo "`echo $s | md5sum | cut -b -8`: $s"
    done
    
     
  9. srouquette

    srouquette Network Guru Member

    @Toink: that's my wanup script, to download adblock and pixelserv:

    Code:
    PREFIX="/tmp/adblock/" ## adjust this!
    
    if [ ! -d "$PREFIX" ]; then
      sleep 10
      mkdir "$PREFIX"
    fi
    
    echo '
    ### Settings ###
    PIXEL_IP="254" ## 0: disable pixelserv
    ## 1-254: last octet of IP to run pixelserv on (default=254)
    PIXEL_OPTS="" ## additional options for pixelserv
    BRIDGE="br0" ## bridge interface for pixelserv (default=br0)
    RAMLIST="0" ## keep blocklist in RAM (e.g. for small JFFS) (default=0)
    CONF="/etc/dnsmasq.custom" ## dnsmasq custom config (must be sourced by dnsmasq!)
    
    
    ### Sources (uncomment desired blocklists) [must be compatible to the hosts file format!] ###
    ## MVPS HOSTS (~600k) [default]:
    SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
    ## pgl.yoyo.org (~70k) [default]:
    SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
    ## Hosts File Project (~3M!):
    #SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    ## The Cameleon Project (~600k):
    #SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
    ## AdAway mobile ads (~20k):
    #SOURCES="$SOURCES http://adaway.sufficientlysecure.org/hosts.txt"
    ## hpHosts ad/tracking servers (~400k):
    #SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
    ## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list):
    #SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt, http://hosts-file.net/hphosts-partial.asp"
    ## MalwareDomainList.com (~40k):
    SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
    ## MalwareDomains.com (~360k):
    SOURCES="$SOURCES http://mirror1.malwaredomains.com/files/justdomains"
    
    
    ### Blacklist additional sites ###
    ## (add hostnames inside the quotes, space-separated, without http://) ##
    BLACKLIST=""
    
    ### Whitelist sites from blocking ###
    ## (add hostnames inside the quotes, space-separated, without http://) ##
    WHITELIST="feedburner.com intel.com goo.gl s3.amazonaws.com raptr.com"
    ' > "$PREFIX/config" && echo success
    
    
    wget http://server.com/tomato/adblock.sh -O - | tr -d "\r" > "$PREFIX/adblock.sh"
    wget http://server.com/tomato/pixelserv_v30 -O "$PREFIX/pixelserv"
    chmod +x "$PREFIX/adblock.sh"
    chmod +x "$PREFIX/pixelserv"
    
    
    "$PREFIX/adblock.sh"
    
    put adblock.sh (not the base64 encoded version, the clear one) on a server and specify the url to wget.
     
    Toink likes this.
  10. Toink

    Toink Network Guru Member


    Thank you srouquette!

    I can't seem to make it work when I pasted your script to my WAN up. I'm positive it's the adblock.sh where I'm failing. Sorry for being such a pain, can you point me to the right direction where I could get that 'clear one'?

    Also, I got a long list of 'Whitelist'... Anyway I could also pull the list from dropbox, the same way as the pixelserv and the hosts?

    I am also sending you my edited script based from yours via PM. If you can check where I failed, I would greatly appreciate it.


    Thank you for your time as always!
     
  11. srouquette

    srouquette Network Guru Member

    Toink likes this.
  12. HunterZ

    HunterZ LI Guru Member

    Improved HTTPS ad blocking can be achieved by using stunnel from entware to provide an SSL tunnel for pixelserv on port 443 of the redirection IP (thanks to the author of nullserv for mentioning it and providing an example of how to generate a cert):

    1. Install entware.
    2. Install stunnel ("opkg install stunnel"), which also installs openssl and a couple other dependencies.
    3. Create a self-signed SSL key:
    Code:
    openssl req -new -nodes -x509 -out /opt/etc/stunnel/stunnel.pem -keyout /opt/etc/stunnel/stunnel.pem
    4. Configure stunnel to run in daemon mode and redirect to pixelserv's IP (example here: http://pastebin.com/9PQSm8EP)
    5. Modify adblock.sh as follows:

    A. Modify "stop()" to look like this (note the additional iptables -D commands and killall stunnel):
    Code:
    stop() {
            elog "Stopping"
            rm "$CONF" &>/dev/null
     
            iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT &>/dev/null
            iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 443 -j ACCEPT &>/dev/null
            killall pixelserv &>/dev/null
            killall stunnel &>/dev/null
            ifconfig $BRIDGE:1 down &>/dev/null
            iptables -D INPUT -i $BRIDGE -p tcp -d $redirip -j REJECT --reject-with tcp-reset &>/dev/null
            iptables -D INPUT -i $BRIDGE -p all -d $redirip -j REJECT --reject-with icmp-host-prohibited &>/dev/null
     
            elog "Done, restarting dnsmasq"
            service dnsmasq restart
    }
    
    B. Modify the section near the bottom to look like this (notice the additional iptables -I commands and the if..else..fi block for stunnel):
    Code:
    if [ "$PIXEL_IP" != "0" ]; then
            if ps | grep -v grep | grep -q "$prefix/pixelserv $redirip"; then
                    elog "pixelserv already running, skipping"
            else
                    elog "Setting up pixelserv on $redirip"
     
                    iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
                            iptables -I INPUT -i $BRIDGE -p all -d $redirip -j REJECT --reject-with icmp-host-prohibited
                            iptables -I INPUT -i $BRIDGE -p tcp -d $redirip -j REJECT --reject-with tcp-reset
                            iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 443 -j ACCEPT
                            iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
                    }
                    ifconfig $BRIDGE:1 $redirip up
                    "$prefix/pixelserv" $redirip $PIXEL_OPTS
            fi
     
            if ps | grep -v grep | grep -q "stunnel"; then
                    elog "stunnel already running, skipping"
            else
                    elog "Starting stunnel"
                    stunnel &
            fi
    fi
    
    Notes:
    1. When trying to view redirected HTTPS URLs in a browser, you will likely get an untrusted certificate warning. This by itself is an improvement over dropping/rejecting the connections, because it happens instantaneously.
    2. You can add an exception for the certificate, which will then let you see via HTTPS what pixelserv normally serves up. The annoyance here is that you have to do it on a per-browser basis.
    3. I've noticed that sometimes my browser complains that the data received from pixelserv over HTTPS is invalid/corrupted. I have no idea what causes this, but I'm sure it's stunnel's fault because stunnel logs to /var/log/messages that it has sent a different number of bytes when the error occurs.
     
  13. Toink

    Toink Network Guru Member

    Code:
    ### Sources (uncomment desired blocklists) [must be compatible to the hosts file format!] ###
    ## MVPS HOSTS (~600k) [default]:
    SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
    ## pgl.yoyo.org (~70k) [default]:
    SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
    ## Hosts File Project (~3M!):
    #SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    ## The Cameleon Project (~600k):
    #SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
    ## AdAway mobile ads (~20k):
    #SOURCES="$SOURCES http://adaway.sufficientlysecure.org/hosts.txt"
    ## hpHosts ad/tracking servers (~400k):
    #SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
    ## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list):
    #SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt, http://hosts-file.net/hphosts-partial.asp"
    ## MalwareDomainList.com (~40k):
    SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
    ## MalwareDomains.com (~360k):
    SOURCES="$SOURCES [URL]http://mirror1.malwaredomains.com/files/justdomains"[/URL]
    I notice that if I add my own personal list of hosts, it still downloads the default ones. Using my own list and the old ALL-U-NEED adblock, I'm getting some 29700+ lists sorted. With this one, only some 18000+ are sorted, hence some ads are still showing (e.g. NBC News App for Windows 8)

    My sources (Hosts 1-4), doesn't load:

    Code:
    ### Sources (uncomment desired blocklists) [must be compatible to the hosts file format!] ###
    ## MVPS HOSTS (~600k) [default]:
    #SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
    ## pgl.yoyo.org (~70k) [default]:
    #SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
    ## Hosts File Project (~3M!):
    #SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    ## The Cameleon Project (~600k):
    #SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
    ## AdAway mobile ads (~20k):
    #SOURCES="$SOURCES http://adaway.sufficientlysecure.org/hosts.txt"
    ## hpHosts ad/tracking servers (~400k):
    #SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
    ## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list):
    #SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt, http://hosts-file.net/hphosts-partial.asp"
    ## MalwareDomainList.com (~40k):
    SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
    ## MalwareDomains.com (~360k):
    SOURCES="$SOURCES http://mirror1.malwaredomains.com/files/justdomains"
    ## Host1 [default]:
    SOURCES="$SOURCES http://db.tt/VgYecrA"
    ## Host2 [default]:
    SOURCES="$SOURCES http://db.tt/8utIr60"
    ## Host3 [default]:
    SOURCES="$SOURCES http://db.tt/gTOfSLq"
    ## Host4 [default]:
    SOURCES="$SOURCES http://db.tt/BSdy4XF"
     
  14. srouquette

    srouquette Network Guru Member

    yeah, I also noticed I was down from 32k to 17k, but I wasn't sure if it was a bug or something...
     
    Toink likes this.
  15. Toink

    Toink Network Guru Member

    ^This plus my preferred hosts wont load :(
     
  16. haarp

    haarp LI Guru Member

    This stupid forum software didn't notify me properly, making me think that this thread was dead. Ugh.

    Look at your links again. It can't work, they have BBcode in them, for some reason.

    It shouldn't discard anything. If it does, please give me a filter list that's affected, so I can have a look at it.

    Indeed. I fixed it, it was an oversight on my part.
     
  17. haarp

    haarp LI Guru Member

    a) Getting at the last-modified timestamp in the first place is already a hack and will not work everywhere. I can still have a look at it, if you can tell me which are affected, but no promises.
    I will not add additional logic to prevent refetching. There is no need to run this script hourly, as you seem to be doing. Once every week is plenty.

    b) I want my script to be as solid as possible, so if you could investigate your IPv6 problems for me, that would be very much appreciated.

    c) Not within the scope of my script, sorry. For something that is run maybe once every few days, this isn't necessary.

    d) The limited busybox environment on Tomato does not offer unzip. That's not possible.

    e) Can you add this to the bottom of the iptables rules and check if that helps?
    iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 443 -j REDIRECT --to-ports 80
    I do not want to introduce any new dependencies, and this might just do the trick (or it might make browsers shit their pants).
     
  18. Toink

    Toink Network Guru Member

    The BBCodes are only generated when I posted here. - I edited the code posted above.

    I of course, don't have it on my script :) Thanks :)
     
  19. HunterZ

    HunterZ LI Guru Member

    I'll try to post info on various issues I've had with the provided list of hosts when I'm at home.

    It's nothing that you should need to worry about. The adblock.sh-created iptables entries seem to get blown away as a result of hitting "Save" in the Tomato web GUI's IPv6 page. This just means that I have to re-run the adblock script once per "Save" operation to restore the custom iptables entries.

    I'll give it a shot.

    That may actually be the most ideal solution for running without stunnel if your goal is to have the situation resolve as quickly as possible without caring what the browser actually receives. I'm not convinced that stunnel adds much practical value over your suggestion anyways, given the issues/limitations I've experienced with the stunnel solution.

    If you want to play it safe, though, my reject-with-TCP-reset approach in post #208 works fairly well (and for more than just port 443).
     
  20. haarp

    haarp LI Guru Member

    The goal is to have the connection attempt hit pixelserv and praying that the browser accepts a sudden gif over non-HTTPS :p
    Otherwise, -j REJECT --reject-with tcp-reset would be preferable.
     
  21. HunterZ

    HunterZ LI Guru Member

    Based on discussions with nullserv's author, my guess is that the browser would likely toss the received data and notify the user it was invalid/corrupt.

    BTW, note that pixelserv was recently updated to incorporate some additional response types from nullserv (e.g. jpg, png, swf, text, etc.).
     
  22. Monk E. Boy

    Monk E. Boy Network Guru Member

    Yeah, before data is sent over port 443 it establishes an SSL session, so injecting non-SSL traffic into the middle of an encrypted session would at best result in invalid/corrupt, at worst if the client is running some kind of firewall/security it could detect it as a man-in-the-middle attack and display a warning. Reject is safer and probably the only option for encrypted traffic.
     
  23. HunterZ

    HunterZ LI Guru Member

    Sources info, as promised:
    • http://hostsfile.mine.nu/Hosts supports last-modified, but for some reason I seem to end up with a 0 byte source- file for it a fair percentage of the time. It may be worth downloading to a temp file and then testing whether the download was successful before copying it over the last known good one? Would probably not want to update the lastmod- timestamp in the case of a failure either.
    • http://sysctl.org/cameleon/hosts does not seem to support last-modified.
    • http://adaway.sufficientlysecure.org/hosts.txt also does not seem to support last-modified.

    I've commented out all three of these in order to get a more stable blocklist.
     
  24. haarp

    haarp LI Guru Member

    The next version will include optional file-based black and whitelist. I want to fix these issues before releasing tho:

    hostsfile.mine.nu supports lastmod and is working fine for me. If you end up with a 0 byte source-file, something else is fishy. What wget does is:
    Code:
    wget SOURCE -O - | tr -d "\r" | sed -e '/^[[:alnum:]:]/!d' | awk '{print $2}' | sed -e '/^localhost$/d' > SOURCEFILE
    replace SOURCE with the source you want to test and try it on your router. It should generate SOURCEFILE.

    http://sysctl.org/cameleon/hosts seems to support lastmod just fine. Can you verify it again?

    http://adaway.sufficientlysecure.org/hosts.txt's lastmod will be fixed in the next release.
     
    Elfew likes this.
  25. QSxx

    QSxx LI Guru Member

    Thank you for the great work and dedication to maintaining this very useful script :)
     
  26. HunterZ

    HunterZ LI Guru Member

    hostsfile.mine.nu works the 2-3 times I tested it just now, but as I mentioned before I don't get an empty file every time. I'll leave it enabled for a while and see if I can spot it happening again.

    I'm definitely not getting a lastmod- file for the sysctl.org blocklist. As a result, it redownloads the ~629KB file every time adblock.sh is run. This behavior is 100% consistent, so I will be disabling my use of this again.
     
  27. HunterZ

    HunterZ LI Guru Member

    I also noticed that the whitelist functionality takes a long time, partially for the following reasons:
    • Individual sed commands are run for each entry in the list, which each scan the entire list in order to remove what is likely to be a small number of entries.
    • It appears that a temporary copy of the blocklist is created for each entry in the list.
    I was able to get a version working that concatenates all whitelist entries into a single sed command using sed -e, but I was concerned about its robustness due to the fact that I had to use the 'eval' command, which has the side-effect of stripping all the quotes off of the parameters.

    I wonder if egrep -v would buy anything over sed for this use case?

    Other ideas:
     
  28. mstombs

    mstombs Network Guru Member

  29. jerrm

    jerrm Network Guru Member

    Have you considered replacing:
    Code:
    ln -s "$listprefix/blocklist" "$CONF" # dnsmasq ignores broken symlinks :)
    with:
    Code:
    echo "conf-file=$listprefix/blocklist" > "$CONF"
    It maintains the same functionality, and eliminates a duplicate copy of the list being in RAM.
     
  30. HunterZ

    HunterZ LI Guru Member

    As I predicted, hostsfile.mine.nu went 0-byte on me again:
    Code:
    ...
    -rwxr--r--    1 1000    1000            43 Jun 20 14:23 lastmod-293f9ffc
    -rwxr--r--    1 1000    1000            45 Jun 22 04:00 lastmod-3b41114e
    -rwxr--r--    1 1000    1000            45 Jun 20 14:30 lastmod-6cc47286
    -rwxr--r--    1 1000    1000            45 Jun 20 14:23 lastmod-c2934517
    -rwxr--r--    1 1000    1000            45 Jun 20 14:23 lastmod-da9bd190
    ...
    -rwxr--r--    1 1000    1000        247144 Jun 20 14:23 source-293f9ffc
    -rwxr--r--    1 1000    1000        24699 Jun 22 04:00 source-3b41114e
    -rwxr--r--    1 1000    1000            0 Jun 21 04:00 source-6cc47286
    -rwxr--r--    1 1000    1000        285735 Jun 20 14:23 source-c2934517
    -rwxr--r--    1 1000    1000        42121 Jun 20 14:23 source-da9bd190
    ...
    293f9ffc: http://hosts-file.net/ad_servers.asp
    3b41114e: http://www.malwaredomainlist.com/hostslist/hosts.txt
    6cc47286: http://hostsfile.mine.nu/Hosts
    c2934517: http://winhelp2002.mvps.org/hosts.txt
    da9bd190: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
    ...
    ADBLOCK: Download starting
    ADBLOCK: Unchanged: http://hosts-file.net/ad_servers.asp (Last-Modified: Thu, 20 Jun 2013 0:29:6 GMT)
    ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Tue, 21 May 2013 19:04:19 GMT)
    ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Fri, 21 Jun 2013 12:26:48 GMT)
    ADBLOCK: Unchanged: http://hostsfile.mine.nu/Hosts (Last-Modified: Wed, 09 May 2012 16:09:21 GMT)
    ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Wed, 05 Jun 2013 15:13:51 GMT)
    ADBLOCK: Filters unchanged
    ADBLOCK: Exiting
    
    Edit: I'm not even sure why the source-6cc47286 file got touched after a good initial download, given that the last-modified time is well over a month ago.
     
  31. Toink

    Toink Network Guru Member

    Has anyone tried the official Windows 8.1 Preview running IE11? I asked coz, this script as well as the ALL-U-NEED ad-blocking script, are causing some of the sites I frequently visit to redirect. Examples are:

    Engadget redirects to http://cdn.at.atwola.com/_media/uac/tcode3.html
    Linksysinfo.org redirects to http://rcm.amazon.com/e/cm
    windowsphonedaily.com redirects to http://rcm.amazon.com/e/cm
    wmpoweruser.com and mynokiablog.com redirects to http://seg.sharethis.com/getSegment.php

    Twice I did a clean install of Windows 8.1 Preview using the official ISO from MS; installed on a newly partitioned/formatted 500GB hard drive. - As soon as I installed Win8.1 I launch IE11 and tried the links above. I get redirected to those sites. :(

    When I disable the adblock script, I can get to these sites without issues. So I'm thinking there must be something incompatible with the script and IE11 - Beats me! >.<

    Using pixelsrv v32, E3000 running Toastman's 0502.8 VLAN

    EDIT:

    FIXED! I added them sites to my 'whitelist'....
     
  32. haarp

    haarp LI Guru Member

    Good idea, that's in place for the file whitelist now.

    Duplicate copy? You do know what a symlink is, right?

    Me neither. I can't trace this down.


    This makes no sense. But this mustn't happen under any cicrumstances. Obviously something is badly wrong here. Please upload your blocklist file somewhere without the whitelist in place.
     
  33. jerrm

    jerrm Network Guru Member

    Of course.

    The problem is the dnsmasq.custom file is appended to dnsmasq.conf. When using the conf-file directive in .custom, only the single line gets appended to dnsmasq.conf, not the entire blocklist.
     
  34. HunterZ

    HunterZ LI Guru Member

    I tested jerrm's idea:

    It does appear to reduce my RAM usage by the size of the blocklist, while still redirecting blocked IPs.


    And on an unrelated note, I noticed 3 copies of "nc" running for hosts-file.net. I guess these are hanging once in a while for some reason and getting cleaned up by the script? I just did an adblock.sh restart and don't see a new copy of "nc" running, so maybe it only happens when a source updates or something.


    Edit: I'm going to try to see if I can capture any info on that one source going to 0 bytes. I've now got an hourly cron job that runs adblock, and dumps its output and a long listing of source+lastmod file info to a timestamped log file.
     
  35. haarp

    haarp LI Guru Member

    Yep, I forgot how the custom dnsmasq config works. It's been a while...

    nc hangs? Interesting. Try adding -w30 to its arguments in the script
     
  36. HunterZ

    HunterZ LI Guru Member

    I can't reproduce the nc hang at this time, but if I see it again I'll do the tweak and see if it helps.


    Edit: Update on the 0-byte source file:
    Code:
    -rwxr--r--    1 1000    1000            27 Jun 27 09:56 /cifs1/adblock/lastmod-293f9ffc
    -rwxr--r--    1 1000    1000            45 Jun 22 04:00 /cifs1/adblock/lastmod-3b41114e
    -rwxr--r--    1 1000    1000            45 Jun 28 08:21 /cifs1/adblock/lastmod-6cc47286
    -rwxr--r--    1 1000    1000            45 Jun 20 14:23 /cifs1/adblock/lastmod-c2934517
    -rwxr--r--    1 1000    1000            45 Jun 20 14:23 /cifs1/adblock/lastmod-da9bd190
    -rwxr--r--    1 1000    1000        247144 Jun 28 07:40 /cifs1/adblock/source-293f9ffc
    -rwxr--r--    1 1000    1000        24699 Jun 22 04:00 /cifs1/adblock/source-3b41114e
    -rwxr--r--    1 1000    1000      1865994 Jun 28 08:21 /cifs1/adblock/source-6cc47286
    -rwxr--r--    1 1000    1000        285735 Jun 24 21:56 /cifs1/adblock/source-c2934517
    -rwxr--r--    1 1000    1000        42121 Jun 20 14:23 /cifs1/adblock/source-da9bd190
    ADBLOCK: Download starting
    Connecting to hostsfile.mine.nu (192.168.1.254:80ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Wed, 05 Jun 2013 15:13:51 GMT)
    ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Tue, 21 May 2013 19:04:19 GMT)
    )
    ADBLOCK: Unchanged: http://hosts-file.net/ad_servers.asp (Last-Modified: 20136991324)
     
    ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Fri, 21 Jun 2013 12:26:48 GMT)
    ADBLOCK: Downloaded
    ADBLOCK: Generating /cifs1/adblock/blocklist
    ADBLOCK: Config generated, 25433 unique hosts to block
    ADBLOCK: pixelserv already running, skipping
    ADBLOCK: Done, restarting dnsmasq
    ADBLOCK: Exiting
    -rwxr--r--    1 1000    1000            27 Jun 27 09:56 /cifs1/adblock/lastmod-293f9ffc
    -rwxr--r--    1 1000    1000            45 Jun 22 04:00 /cifs1/adblock/lastmod-3b41114e
    -rwxr--r--    1 1000    1000            45 Jun 28 08:21 /cifs1/adblock/lastmod-6cc47286
    -rwxr--r--    1 1000    1000            45 Jun 20 14:23 /cifs1/adblock/lastmod-c2934517
    -rwxr--r--    1 1000    1000            45 Jun 20 14:23 /cifs1/adblock/lastmod-da9bd190
    -rwxr--r--    1 1000    1000          1639 Jun 28 08:56 /cifs1/adblock/log.2013-06-28T08:56:01-0700.txt
    -rwxr--r--    1 1000    1000        247144 Jun 28 07:40 /cifs1/adblock/source-293f9ffc
    -rwxr--r--    1 1000    1000        24699 Jun 22 04:00 /cifs1/adblock/source-3b41114e
    -rwxr--r--    1 1000    1000            0 Jun 28 08:56 /cifs1/adblock/source-6cc47286
    -rwxr--r--    1 1000    1000        285735 Jun 24 21:56 /cifs1/adblock/source-c2934517
    -rwxr--r--    1 1000    1000        42121 Jun 20 14:23 /cifs1/adblock/source-da9bd190
    
    What this shows:
    • No "unchanged" message logged for hostsfile.mine.nu.
    • stderr shows suspicious "Connecting to" hostsfile.mine.nu" message.
    • Timestamp on corresponding lastmod-6cc47286 file does not change.
    • Size of source-6cc47286 goes to zero.
    Edit 2: I don't quite understand the following code, but it looks to me like you're piping stuff to commands that then redirect to $sourcefile even in the case that wget fails?
    Code:
            (
                    if wget $1 -O -; then
                            [ -n "$lastmod" ] && echo "$lastmod" > "$lmfile"
                            echo 0 >>"$listprefix/status"
                    else
                            elog "Failed: $1"
                            echo 1 >>"$listprefix/status"
                    fi
            ) | tr -d "\r" | sed -e '/^[[:alnum:]:]/!d' | awk '{print $2}' | sed -e '/^localhost$/d' > "$sourcefile"
    Edit 3: Added some logging to grabsource() after the "unchanged" check determines that the list should be downloaded:
    Code:
    ADBLOCK: grabsource(): force=, 1=http://hostsfile.mine.nu/Hosts, host=hostsfile.mine.nu, path=Hosts, lastmod=, lmfile=/cifs1/adblock/lastmod-6cc47286, sourcefile=/cifs1/adblock/source-6cc47286, oldlastmod=Last-Modified: Wed, 09 May 2012 16:09:21 GMT
    Connecting to hostsfile.mine.nu (192.168.1.254:80)
    So apparently hostsfile.mine.nu sometimes fails to return a lastmod value, and this causes the adblock script to think that it needs to redownload the blocklist. Apparently wget then fails too, and the source file gets clobbered.

    Suggested changes:
    • Fix $lastmod validity checking?
    • Don't allow $sourcefile to be touched if wget fails.
    Edit 4: I tried to fix clobbering of the source file by having wget output to a temp file and then post-processing that to the actual source file only if wget reports success, but that still didn't work because wget apparently doesn't report failure if it "successfully" downloads a 0-byte file?
     
  37. HunterZ

    HunterZ LI Guru Member

    I think I found the root of the 0-byte file problem:

    The source-6cc47286 list generated from hostsfile.mine.nu/Hosts contains an entry for mine.nu!
    Code:
    # grep mine.nu source*
    source-6cc47286:ganja.mine.nu
    source-6cc47286:ilim.mine.nu
    source-6cc47286:mine.nu
    source-6cc47286:kik.mine.nu
    This is causing the following series of events:
    1. adblock script downloads blocklist from hostsfile.mine.nu.
    2. mine.nu becomes blocked.
    3. adblock script runs again and clobbers hostsfile.mine.nu source file, and rebuilds blocklist without mine.nu blocking.
    4. go to step 1
    WTF?!


    edit: Also, I think I've implemented successful protection against 0-byte source files:
    Code:
    grabsource() {
            local host=$(echo $1 | awk -F"/" '{print $3}')
            local path=$(echo $1 | awk -F"/" '{print substr($0, index($0,$4))}')
            local lastmod=$(echo -e "HEAD /$path HTTP/1.1\r\nHost: $host\r\n" | nc $host 80 | tr -d '\r' | grep "Last-Modified")
    
            local sourcehash="$(echo $1 | md5sum | cut -c 1-8)"
            local lmfile="$listprefix/lastmod-$sourcehash"
            local sourcefile="$listprefix/source-$sourcehash"
            local tempfile="$listprefix/temp-$sourcehash"
    
            [ "$force" != "1" -a -e "$sourcefile" -a -n "$lastmod" -a "$lastmod" == "$(cat "$lmfile" 2>/dev/null)" ] && {
                    elog "Unchanged: $1 ($lastmod)"
                    echo 2 >>"$listprefix/status"
                    return 2
            }
    
            elog "grabsource(): force=$force, 1=$1, host=$host, path=$path, lastmod=$lastmod, lmfile=$lmfile, sourcefile=$sourcefile, oldlastmod=$(cat "$lmfile" 2>/dev/null)"
            elog `expr length "$lastmod"`
    
            if wget $1 -O "$tempfile" && [ -s "$tempfile" ] ; then
                    cat "$tempfile" | tr -d "\r" | sed -e '/^[[:alnum:]:]/!d' | awk '{print $2}' | sed -e '/^localhost$/d' > "$sourcefile"
                    [ -n "$lastmod" ] && echo "$lastmod" > "$lmfile"
                    echo 0 >>"$listprefix/status"
            else
                    elog "Failed: $1"
                    echo 1 >>"$listprefix/status"
            fi
    
            rm "$tempfile" &>/dev/null
    }
     
  38. jerrm

    jerrm Network Guru Member

    Sorry, I should have been following the thread closer - I had already reported this: http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-2#post-226746
     
  39. Serenus

    Serenus Networkin' Nut Member

    So I followed the entire step by step and got it to start up, but when I start it up it blocks all my internet connections (only google.com seems to work).

    I checked the logs and it keeps throwing the same error over and over:
    Jul 1 02:28:51 unknown user.debug init[1]: dnsmasq terminated unexpectedly, restarting.
    Jul 1 02:28:51 unknown daemon.crit dnsmasq[2733]: bad option at line 15 of /etc/dnsmasq.conf
    Jul 1 02:28:51 unknown daemon.crit dnsmasq[2733]: FAILED to start up

    I have an e3000 router running Toastman latest stable build 7502.
     
  40. HunterZ

    HunterZ LI Guru Member

    Serenus: Can you post the first 15-20 lines of /etc/dnsmasq.conf on your router?

    You can do this by going to Tools->System and typing the following command in the box and hitting Execute, then copy and paste the results here:
    Code:
    cat /etc/dnsmasq.conf
    It may be helpful to also do the same thing for /etc/dnsmasq.custom

    P.S. dnsmasq is complaining of an error in the config file that it's being given, which is prevent it from running. This in turn means that anyone on your LAN using your router as a DNS server will not be able to complete domain name lookups. It's likely that google only still worked for you because it was cached somewhere (likely in your test machine's OS or browser).
     
  41. Serenus

    Serenus Networkin' Nut Member

    There you go:

    cat /etc/dnsmasq.conf

    pid-file=/var/run/dnsmasq.pid
    resolv-file=/etc/resolv.dnsmasq
    addn-hosts=/etc/dnsmasq/hosts
    dhcp-hostsfile=/etc/dnsmasq/dhcp
    expand-hosts
    min-port=4096
    stop-dns-rebind
    rebind-localhost-ok
    interface=br0
    dhcp-range=tag:br0,192.168.2.1,192.168.2.230,255.255.255.0,1440m
    dhcp-lease-max=255
    dhcp-authoritative


    0.datacollector.coin.scribol.com
    0.r.msn.com
    005.free-counter.co.uk
    006.free-counter.co.uk
    007.free-counter.co.uk
    008.free-counter.co.uk
    008.free-counters.co.uk
    00fun.com
    011707160008.c.mystat-in.net
    061606084448.c.mystat-in.net
    070806142521.c.mystat-in.net
    090906042103.c.mystat-in.net
    092706152958.c.mystat-in.net
    0d7292.r.axf8.net
    0f36f3.r.axf8.net
    0koryu0.easter.ne.jp
    1.datacollector.coin.scribol.com
    1.googlenews.xorg.pl
    1.hot-dances.com
    1.michaelwilsonmusic.com
    1.ofsnetwork.com
    1.oz-over.com
    1.sharkadnetwork.com
    100-100.ru

    cat /etc/dnsmasq.custom

    cat: can't open '/etc/dnsmasq.custom': No such file or directory
     
  42. jerrm

    jerrm Network Guru Member

    Something is wrong with your adblock script, it is not formatting the entries correctly. All those domain entries should look like:
    Code:
    address=/00.moregoogle.bee.pl/192.168.0.254
    address=/00.moregoogle.osa.pl/192.168.0.254
    address=/000-101.org/192.168.0.254
    address=/0000.in/192.168.0.254
    address=/00002l8.previewcoxhosting.com/192.168.0.254
    and not list just the name by itself.

    Hard to say what is causing the errors, probably some sort of cut&paste or copy error when creating the script.
     
  43. Serenus

    Serenus Networkin' Nut Member

    Btw whenever I enabled JFFS it mentioned there was around 900KB available to be allocated, I'm not entirely sure how much space is required so I thought that maybe it wouldn't be enough since some of the list are way over 1MB, should I try and use a USB to store it?

    If so, how exactly should I do that?
     
  44. HunterZ

    HunterZ LI Guru Member

    1MB is almost certainly too little space.

    Personally I have Toastman's Tomato configured to mount a cifs (Windows samba/smb) share on my Linux workstation, and my adblock script stuff lives there.
     
  45. Serenus

    Serenus Networkin' Nut Member

    Reinstalled on the USB drive I had laying around and all seem to be working.

    Would you guys happen to have a test URL for ads?
     
  46. HunterZ

    HunterZ LI Guru Member

    google-analytics.com? ad.fly?
     
  47. m771401

    m771401 Reformed Router Member

    Quick question.

    Is the OP the most current iteration of this script/s?
    I ask because there is no timestamp for edits apparently.

    I would also like to thank you guys for being so damn smart!

    I am/was using the all-u-need script on a RT-N16. If this is better I would like to switch.
     
  48. HunterZ

    HunterZ LI Guru Member

    I believe the OP is the only place that the "official" version of the script resides. If you want to tweak it in various ways, there are some potentially useful posts scattered around the thread.


    I was also using all-u-need on my RT-N16 (and formerly my WRT54G), but had a number of gripes with it.

    This script addresses all of my major issues with all-u-need, so I highly recommend it if you have enough storage space somewhere (i.e. cifs, USB, or a sufficiently-large jffs partition).
     
  49. Cayennr

    Cayennr Reformed Router Member

    Forgive my lack of knowledge, but does this script remove ads from websites or it just places a white space instead? Please see the picture to see what I’m trying to understand. I’d like to use the script, but before I start I just wanted to make sure I understand what the script actually does. Last time I attempted to achieve something I completely screwed my configuration and I wasn’t able to access the router to revert it back. Luckily I had recent backup so I didn’t have to do the whole setup all over again. Oops…

    [​IMG]
     
  50. mstombs

    mstombs Network Guru Member


    As ever the answer is somewhere in between, if a simple image is blocked it can be replaced via pixelserv with a 1*1 image which allows the space to collapse, and you then only start to notice how effective it is when using a work/other internet connection! The browser could however be instructed to resize the image then you would get the effect on the left. When the browser request is unclear, perhaps because image is loaded via script which sets variables on blocked site - then you can get browser error messages/ broken image symbols or at worst incomplete page loads. So you have 2 choices - don't use site in question - or whitelist it and accept. More advert sites are moving to https which is a bigger load for their adservers but prevents interference or substitution of content. Other sites are using their own dns servers on first page request and embed IP addresses in web pages which avoids any dns poisoning ad blocking.

    AdblockPlus in compatible browser can do a better job - but a router solution protects all devices on your network, and if you see how big some of the ad site requests are you will appreciate the internet traffic reduction - and the privacy improvements blocking those hosts must give!
     
    Monk E. Boy likes this.
  51. Cayennr

    Cayennr Reformed Router Member

    Thank you for your response. That clears things up.
     
  52. HunterZ

    HunterZ LI Guru Member

    @Cayennr: Some web sites use frames and such to reserve specific-sized spaces on web pages for ads to appear in, and for these there will be a blank space when the ad is blocked. Other sites just put the ads inline, and the browser will not use anything more than the 1x1 pixel area of the adblock image.

    In general, I don't notice the areas where the ad is supposed to appear when it is blocked, unless an ad tries to load in a frame via https, as this sometimes results in a browser error appearing in the frame (I've found that serving fake ads over https is not truly feasible unless you want to use a proxy server on your LAN, which has its own issues).
     
  53. Cayennr

    Cayennr Reformed Router Member

    Thanks, definitely more clear now.
     
  54. rs232

    rs232 Network Guru Member

    I'm just testing this... A couple of questions if you don't mind:

    A) I spent sometime compiling a whitelist according to the old ADBLOCK script, this was fed to the script via a webserver. Is there any way to import a the .txt whitelist via URL with this new script?



    B) How do you verify if the script is running properly? adblock.sh status would help ;)


    C) When running it the last line I get after the lists are downloaded, is:

    Code:
    Jul  3 18:50:09 tomato user.notice ADBLOCK: Exiting
    good/bad?

    For reference this is the content of the log:

    Code:
    Jul  3 19:05:47 tomato user.notice ADBLOCK: Download starting
    Jul  3 19:05:47 tomato user.notice ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Sat, 29 Jun 2013 19:46:59 GMT)
    Jul  3 19:05:48 tomato user.notice ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Fri, 21 Jun 2013 12:26:48 GMT)
    Jul  3 19:05:48 tomato user.notice ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Tue, 21 May 2013 19:04:19 GMT)
    Jul  3 19:05:48 tomato user.notice ADBLOCK: Filters unchanged
    Jul  3 19:05:48 tomato user.notice ADBLOCK: Setting up pixelserv on 10.10.10.254
    Jul  3 19:05:48 tomato daemon.info pixelserv[2383]: /cifs1/adblock/pixelserv V30 compiled: Nov  6 2012 09:13:44 from pixelserv30.c
    Jul  3 19:05:48 tomato user.notice ADBLOCK: Done, restarting dnsmasq
    Jul  3 19:05:48 tomato daemon.notice pixelserv[2385]: Listening on 10.10.10.254:80
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2256]: exiting on receipt of SIGTERM
    Jul  3 19:05:48 tomato user.debug init[1]: 182: pptp peerdns disabled
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: started, version 2.67cs7 cachesize 4096
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset Tomato-helper auth
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: asynchronous logging enabled, queue limit is 5 messages
    Jul  3 19:05:48 tomato daemon.info dnsmasq-dhcp[2390]: DHCP, IP range 172.16.10.100 -- 172.16.10.127, lease time 7d
    Jul  3 19:05:48 tomato daemon.info dnsmasq-dhcp[2390]: DHCP, IP range 10.10.10.100 -- 10.10.10.127, lease time 7d
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: reading /etc/resolv.dnsmasq
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: using nameserver 194.168.8.100#53
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: using nameserver 194.168.4.100#53
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: using nameserver 208.67.220.220#53
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: using nameserver 8.8.4.4#53
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: using nameserver 8.8.8.8#53
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: read /etc/hosts - 3 addresses
    Jul  3 19:05:48 tomato daemon.info dnsmasq[2390]: read /etc/dnsmasq/hosts/hosts - 18 addresses
    Jul  3 19:05:48 tomato daemon.info dnsmasq-dhcp[2390]: read /etc/dnsmasq/dhcp/dhcp-hosts
    Jul  3 19:05:48 tomato user.notice ADBLOCK: Exiting
    
    I guess it's good because if I re-run the scrip I only get these logs:

    Code:
    Jul  3 19:07:26 tomato user.notice ADBLOCK: Download starting
    Jul  3 19:07:27 tomato user.notice ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Sat, 29 Jun 2013 19:46:59 GMT)
    Jul  3 19:07:27 tomato user.notice ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Tue, 21 May 2013 19:04:19 GMT)
    Jul  3 19:07:27 tomato user.notice ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Fri, 21 Jun 2013 12:26:48 GMT)
    Jul  3 19:07:27 tomato user.notice ADBLOCK: Filters unchanged
    Jul  3 19:07:27 tomato user.notice ADBLOCK: Exiting
    Thanks!
    rs232
     
  55. HunterZ

    HunterZ LI Guru Member

    I think the only way to be sure is to check how much memory dnsmasq is using with and without the adblock setup in place.

    You can "disable" the adblock by running 'adblock.sh stop', which will restart dnsmasq without the blocklist in place.
     
  56. haarp

    haarp LI Guru Member

    Sorry for all this inactivity. I've been very busy.

    The script got updated to 4.5, for changes see the first post. I hope everything works. You now get your whitelist and blacklist files.

    HunterZ, I honestly can't be bothered with mine.nu at the moment. That'll have to be fixed some other time.
     
  57. HunterZ

    HunterZ LI Guru Member

    Adding it to the default whitelist (as suggested by jerrm) is an easy fix.

    Thanks for the update. I'll switch over to it when I get a chance.
     
  58. haarp

    haarp LI Guru Member

    No, that's a workaround.
     
  59. HunterZ

    HunterZ LI Guru Member

    It's as valid as any other that you could come up with to try to address it client-side.

    The only real fix is to get mine.nu to remove themselves from their own blocklist (I still don't understand that one).

    Edit: I suppose some logic to implicitly/dynamically whitelist all configured blocklist sources would make the script more robust.
     
  60. haarp

    haarp LI Guru Member

    I'd say that's a separate issue. The real problem is that the script produces 0-byte files when it can't download a source, for whichever reason.

    And I see no way to fix that, without creating temporary files, which would increase memory consumption temporarily. If only there was a way to delay a pipeline until the first byte is read...
     
  61. HunterZ

    HunterZ LI Guru Member

    v4.5 looks good to me. Only mods I made to my local version were:
    • Removed comma from hosts-file.net SOURCES line (although I also commented it out).
    • Migrated my whitelist to a file (cool!).
    • Changed iptables rules to:
    Code:
    iptables -I INPUT -i $BRIDGE -p all -d $redirip -j REJECT --reject-with icmp-host-prohibited
    iptables -I INPUT -i $BRIDGE -p tcp -d $redirip -j REJECT --reject-with tcp-reset
    iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 443 -j ACCEPT
    iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
    • Changed iptables drop commands to:
    Code:
    iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT &>/dev/null
    iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 443 -j ACCEPT &>/dev/null
    killall pixelserv &>/dev/null
    ifconfig $BRIDGE:1 down &>/dev/null
    iptables -D INPUT -i $BRIDGE -p tcp -d $redirip -j REJECT --reject-with tcp-reset &>/dev/null
    iptables -D INPUT -i $BRIDGE -p all -d $redirip -j REJECT --reject-with icmp-host-prohibited &>/dev/null
    • Changed pixelserv launch commands to:
    Code:
    "$prefix/pixelserv" $redirip -p 80 $PIXEL_OPTS
    "$prefix/pixelserv" $redirip -p 443 $PIXEL_OPTS
    I could have migrated my 0-byte temp file workaround, but decided to try just whitelisting mine.nu for now.
     
  62. haarp

    haarp LI Guru Member

    Why run two pixelservs? Just redirect 443 to 80 via iptables, as I have proposed a while ago. Also, I'm assuming oyu need pixelserv v32 for that?
     
  63. HunterZ

    HunterZ LI Guru Member

    Yes, v32 is required for HTTPS responses and/or running on a specific port.

    I have been meaning to try some iptables suggestions that people have posted, but I haven't had time. I'm an iptables novice, so running 2 pixelserv's was an easy/quick solution for me that my RT-N16 has no problem handling.

    Anyways, the more interesting part of my iptables changes are that it rejects unhandled ports instead of dropping them, which causes blocks to resolve more quickly. By dropping instead of rejecting, you leave clients waiting for connections to time out.
     
  64. Monk E. Boy

    Monk E. Boy Network Guru Member

    Indeed, but keep in mind REJECT creates a higher load on the router since it involves creating and sending packets to the client instead of just redirecting the packet to /dev/null. Under normal circumstances I don't see a difference but with a LOT of connections (read: thousands) running through the router it can be an issue.
     
  65. JAC70

    JAC70 Networkin' Nut Member

    Just FYI to anyone interested, Toastman's OpenSSL still appears broken in STD version v1.28.7502.1 Use WinSCP to transfer the files manually.
     
  66. HunterZ

    HunterZ LI Guru Member

    I thought it was decided in the toastman thread that OpenSSL was only broken in some of the smaller versions of his firmware?

    You could always use the system command page of the web UI, or log in with a telnet client, or store the script on a cifs mount or USB drive as well...
     
  67. JAC70

    JAC70 Networkin' Nut Member

    I didn't see that. I'm using the STD version for e3000. The system command page is what generates the OpenSSL errors.
     
  68. rs232

    rs232 Network Guru Member

    4 things from myself:

    1) Thanks for adding the white/black list file support! Really!
    What is the correct syntax for it in the config file? The old ADBLOCK used:

    Code:
    WHITELIST="$WHITELIST http://webserver/whitelist.txt"
    Before I try... Is this compatible with your script?

    P.S. either ways can I suggest you add a quoted line in the default config referring to a white/blacklist file?

    2) Can I suggest you add support for adblock.sh start
    Just to make it compatible with rc scripts syntax

    3) Can I suggest you add support for adblock.sh status
    This would help indeed... and once again it's a rc script standard

    4) I'm wondering how difficult would it be to have a simple (ajax?) interface to edit the white/blacklist file so that users can add their own domains...
    to give an idea e.g. http://www.yvoschaap.com/weblog/ajax_inline_instant_update_text_20

    Thanks a lot for the script, good work as far as I can see!
    Regards
     
  69. haarp

    haarp LI Guru Member

    Check the start post, the config does mention how to use the files. Automatically downloading it is not supported, but if you want that you could add this to the bottom of your config:

    wget http://webserver/whitelist.txt -O - > "whitelist"
     
    HunterZ and rs232 like this.
  70. jerrm

    jerrm Network Guru Member

    Great script. Using it with the functionality basically untouched. I would like to make a few small requests.

    A personal preference is to keep data separate from binaries. Right now everything is dependent on the single $prefix variable. Would it be possible to break this up a little, making it a little easier to customize? The defaults could remain the same for the KISS install instructions.

    Add a block to the beginning of the script something like:
    Code:
    #path to script
    binprefix="$(cd "$(dirname "$0")" && pwd)"
     
    #path to list files
    prefix=$binprefix
     
    #pixelserv executable
    pixelbin=$binprefix/pixelserv
     
    #config file
    config=$binprefix/config
    
    Then we could reset everything except $config in the config file, and have an easy place to change the config file location, say to something like /opt/etc/adblock.conf, if desired.
     
  71. Bird333

    Bird333 Network Guru Member

    Along with this could a switch be added to disable iptables rules in the script for people who want to set those manually?
     
  72. QSxx

    QSxx LI Guru Member

    There appears to be a problem with new version of adblock (4.5). For some reason once it starts, it refuses to properly interpret switches (stop, restart, force...) and therefore ends up spitting pid error message.

    Code:
    ADBLOCK: Another instance found (/var/run/adblock.pid), exiting!


    If I'm not mistaken, this should appear only if the script was running AT the time you try to run it again, modify, stop, whatever... but it remains locked even after blocklist gets generated and dnsmasq restarted to apply it.

    Also, since I'm using both whitelist and blacklist in form of files - from the first post, it's unclear (or I'm blind) if i should comment $WHITELIST and $BLOCKLIST out or just keep them empty... and if i should specify something to the script that makes it understand i want to use files - or is it automagical :)

    Nvm... it's working... error was caused by wrong formatting of whitelist and blacklist files (DOS instead of UNIX) - it couldn't parse EOL properly
     
    Goggy likes this.
  73. haarp

    haarp LI Guru Member

    Very weird. It shouldn't mind that, and especially not fail in such a weird way.
     
  74. Monk E. Boy

    Monk E. Boy Network Guru Member

    Yeah I don't know why a superfluous <CR> character before the end of a line would matter but I don't know how the files are formatted... if it's an IP or DNS name then I suppose it could be trying to resolve that <CR>.
     
  75. jerrm

    jerrm Network Guru Member

    If you make any firewall changes in the gui, or iptables is reinitialized for some other config change, the adblock rules are lost.

    My fix was to put the adblock rules in a .fire autorun script as listed below. It seems to work well.

    I changed the block that sets up iptables and starts pixelserv from:
    Code:
    if [ "$PIXEL_IP" != "0" ]; then
            if ps | grep -v grep | grep -q "$prefix/pixelserv $redirip"; then
                    elog "pixelserv already running, skipping"
            else
                    elog "Setting up pixelserv on $redirip"
     
                            iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
                            iptables -I INPUT -i $BRIDGE -p all -d $redirip -j DROP
                            iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
                    }
                    ifconfig $BRIDGE:1 $redirip up
                    "$prefix/pixelserv" $redirip $PIXEL_OPTS
            fi
    fi
    to:
    Code:
    if [ "$PIXEL_IP" != "0" ]; then
            if ps | grep -v grep | grep -q "$prefix/pixelserv $redirip"; then
                    elog "pixelserv already running, skipping"
            else
                    elog "Setting up pixelserv on $redirip"
     
                            iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
                            mkdir /etc/config &>/dev/null
                            echo "#!/bin/sh
     
                            iptables -I INPUT -i $BRIDGE -p all -d $redirip -j DROP
                            iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
                            " >  /etc/config/99.adblock.fire
                            chmod +x /etc/config/99.adblock.fire
                            /etc/config/99.adblock.fire
                    }
                    ifconfig $BRIDGE:1 $redirip up
                    "$prefix/pixelserv" $redirip $PIXEL_OPTS
            fi
    fi
    and then in the stop() function add an
    Code:
    rm -f /etc/cfg/99.adblock.fire
     
  76. HunterZ

    HunterZ LI Guru Member

  77. bingo1105

    bingo1105 Networkin' Nut Member

    I agree that this would be preferable, but my router isn't cooperating:

    iptables -I INPUT -i br0 -p tcp -d 172.16.74.254 --dport 443 -j REDIRECT --to-ports 80
    iptables: No chain/target/match by that name

    I also tried:

    iptables -t nat -A PREROUTING -i br0 -p tcp -d 172.16.74.254 --dport 443 -j REDIRECT --to-ports 80
    iptables: No chain/target/match by that name

    Is this possibly a kernel issue? I can't see anything wrong with the syntax of those statements...
     
  78. srouquette

    srouquette Network Guru Member

    I don't remember where, but someone posted something like this:

    iptables -t nat -A PREROUTING -i br0 -p tcp -d 192.168.42.254 --dport 443 -j DNAT --to 192.168.42.254:80
     
    HunterZ likes this.
  79. jerrm

    jerrm Network Guru Member

    That's the command I use, added into the script it becomes:
    Code:
    iptables -t nat -A PREROUTING -i $BRIDGE -p tcp -d $redirip --dport 443 -j DNAT --to $redirip:80
     
    HunterZ likes this.
  80. bingo1105

    bingo1105 Networkin' Nut Member

    Awesome, thanks for that!
     
  81. HunterZ

    HunterZ LI Guru Member

    Thanks jerrm, but what would be the corresponding delete command for that rule?
     
  82. koitsu

    koitsu Network Guru Member

    Change -A to -D.
     
    HunterZ and jerrm like this.
  83. HunterZ

    HunterZ LI Guru Member

    Seems to work, although for some reason the rule doesn't show up via 'iptables -L'.
     
  84. bingo1105

    bingo1105 Networkin' Nut Member

    You have to run iptables -t nat -L to see the effect of the rule.
     
    HunterZ likes this.
  85. Almaz

    Almaz Serious Server Member

    Just installed the script but having a problem on my end. I'm using for testing only one SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt" I'm getting an error unknown user.debug init[1]: dnsmasq terminated unexpectedly, restarting.
     
  86. HunterZ

    HunterZ LI Guru Member

    I'll have to check my logs when I'm at home, but I think that might be OK. Dnsmasq is (and needs to be) killed by the adblock script so that it can restart with the blocklist as part of its new configuration.
     
  87. Almaz

    Almaz Serious Server Member


    Well in my case it keeps happening every couple of seconds and internet is not usable.
     
  88. HunterZ

    HunterZ LI Guru Member

    Well that's definitely bad. Maybe take a look at your /etc/dnsmasq.conf and /etc/dnsmasq.custom files via SSH, telnet, or the system commands page on the router's browser GUI and see if anything looks fishy.
     
  89. Monk E. Boy

    Monk E. Boy Network Guru Member

    Yeah, even the simplest of typos in the dnsmasq config can cause it to endlessly crash. Entering a - instead of = (or vice versa) is a recipe for a dead dnsmasq.
     
  90. Almaz

    Almaz Serious Server Member

    Can anyone tell me why DNSMASQ keeps crashing if I use host file from dropbox such as this one
    Code:
    SOURCES="$SOURCES http://dl.dropbox.com/u/32428671/m.txt?dl=1"
     
  91. HunterZ

    HunterZ LI Guru Member

    Works fine for me. Please put the following on pastebin and post links to them:
    • Log file (/var/log/messages)
    • Dnsmasq config files (/etc/dnsmasq.conf, /etc/dnsmasq.custom)
    • Adblock configuration ('config' file, wherever you have it stored)
     
  92. Almaz

    Almaz Serious Server Member


    The other host source with 1 line worked for me as well after rebooting a router, but I can bet if you enter this source, it'll crash dnsmasq and I just can't figure it out why. Please try this source

    Code:
    SOURCES="$SOURCES http://dl.dropbox.com/u/32428671/myhosts.txt?dl=1"
    
     
  93. HunterZ

    HunterZ LI Guru Member

    I'm not going to even try to load that, because it's not a properly-formatted HOSTS file:
    Code:
    127.0.0.1  http://www.cpalead.com/apply.php?ref=103479
    127.0.0.1  osirisdevelopment.com
    127.0.0.1  www.plimus.com
    127.0.0.1  fr.a2dfp.net
    127.0.0.1  cdn.mediafire.com/images/backgrounds/header/rockmelt_tabcontent.jpg
    127.0.0.1  is.luxup.ru
    127.0.0.1  www.rockmelt.com
    127.0.0.1  rockmelt.com
    127.0.0.1  http://cdn.mediafire.com/images/ad-loader.gif
    You've got URLs and even "http://" prefixes in there, instead of just hostnames. Dnsmasq works on hostnames, not URLs, so you're feeding the Adblock script the wrong kind of data.
     
    Almaz likes this.
  94. Almaz

    Almaz Serious Server Member

    Thank you very much HunterZ, I'll fix it right now.
     
  95. ShinichiYao

    ShinichiYao Reformed Router Member

    Here is a way to use Adblock Plus List
    Code:
    wget -O - http://adblock-chinalist.googlecode.com/svn/trunk/adblock-lazy.txt |
    grep ^\|\|[^\*][^\/]*\^$ |
    sed -e 's:||::' -e 's:\^::' > /YOUR/PREFIX/HERE/blacklist
    /YOUR/PREFIX/HERE/adblock.sh restart
     
  96. Darkvader

    Darkvader Serious Server Member

    :) First off grate work with the Adblocking script and many thanks it seem to be working very well for me.

    Clean, Lean and Mean Adblocking 4.5
    Tomato Firmware 1.28.0000 MIPSR2-112 K26 USB Big-VPN (Shibby)
    Belkin Play Max / N600 HD (F7D4301/F7D8301) v1
    Runing off usb flash

    Please: Can some one explain to me how I can load my owen PREFIX/host.txt file into the update part of the script, so when the script updates the blacklist it adds my host.txt file to the blacklist as well. The idear is to get Shibby to put a configurable link in the "Web Usage" page so you can update your owen host.txt file.

    Thanks
     
  97. HunterZ

    HunterZ LI Guru Member

    I don't think I understand what you're asking, but in case it helps:

    Support was recently added for adding custom blacklist entries via a "$PREFIX/blacklist" file.
     
  98. Darkvader

    Darkvader Serious Server Member

    :) Thanks for a quick responce.
    I found my problem when I ran "adblock.sh" the blocklist did not update from the change made in the blacklist
    but when I ran "adblock.sh force" the blocklist was update and this was the resolute.

    address=/zxkewwcw.biz/192.168.1.254
    address=/zz.cqcounter.com/192.168.1.254
    address=/zzz.clickbank.net/192.168.1.254
    address=/tmda.tmcdn.co.nz
    /192.168.1.254
    address=/secure-nz.imrworldwide.com
    /192.168.1.254
    address=/images.travelbug.co.nz
    /192.168.1.254

    ps: I am using windows notepad to edit the blacklist.
     
  99. HunterZ

    HunterZ LI Guru Member

    The script is not able to detect when the blacklist is modified. The latest version is pulled in whenever the blocklist is regenerated from the configured sources, which occurs if any of the following happen:
    • "adblock.sh force" (as you noted)
    • Any of the configured sources is redownloaded (due to an update, or due to not supporting lastmod tracking)
    • The blocklist file (not to be confused with the blacklist file) is deleted

    Edit: My point is that a web GUI could force the blocklist to be regenerated when blacklist changes are saved by running 'adblock.sh force' or deleting the blocklist and running adblock.sh.
     
  100. Darkvader

    Darkvader Serious Server Member

    Yep I understand and thanks :)
     

Share This Page