1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Script: Clean, Lean and Mean Adblocking

Discussion in 'Tomato Firmware' started by haarp, Apr 23, 2013.

  1. Darkvader

    Darkvader Serious Server Member

    Hi HunterZ were u did not understand my post ↓ I will try to explain again.

    Please: Can some one explain to me how I can load my owen PREFIX/host.txt file into the update part of the script, so when the script updates the blacklist it adds my host.txt file to the blacklist as well. The idear is to get Shibby to put a configurable link in the "Web Usage" page so you can update your owen host.txt file.

    This is my problem: I am not a programer and I have dyslexia which is a pain i the ass "oh" and Im am Windows user so I copy and past Domain Name from Web Usage page in to a windows txt file that has line end of "CRLF" which mess up the blocklist.
    maybe this will make more sense and maybe u can fix my attempt to script.
    Thanks

    #!/bin/sh
    PREFIX="/tmp/mnt/sda1/adblock/" ## Adjust this "/tmp/*/*/"!
    if [ ! -f "$PREFIX"blacklist ] ## To check if File blacklist exists
    then
    echo "" > "$PREFIX"blacklist ##Mack new File blacklist
    fi

    tr -d '\015' < "$PREFIX"MyHosts.txt > "$PREFIX"convert ## Remove the CR from windows file
    mv "$PREFIX"blacklist "$PREFIX"tmp ## Rename blacklist File to temp file
    echo "" > "$PREFIX"blacklist ## Mack new blacklist File
    cat "$PREFIX"convert "$PREFIX"tmp > "$PREFIX"blacklist ## Copy Files to new blacklist File
    mv "$PREFIX"blacklist "$PREFIX"tmp ## Rename blacklist File to temp file
    echo "" > "$PREFIX"blacklist ## Mack new File blacklist
    cat "$PREFIX"tmp | sort -u > "$PREFIX"blacklist ## Remove duplicates in blacklist File

    ##load blacklist it blocklist and restart adblocker.
     
  2. Almaz

    Almaz Serious Server Member

    Darkvader:

    If I understand you correctly you wish to add your hosts file to the block list.

    Create a dropbox free account where you can host files. Make a Public Folder unless it's already there by default. Just in case if you don't have an account here is a Link to get free account

    Create your own hosts file, for example "Darkvader_Hosts_File" and put it in your dropbox public folder. From DropBox website right click on your file "Darkvader_Hosts_File" and click on "Copy Public Link"

    The url should look like
    https://dl.dropboxusercontent.com/u/3212312671/Darkvader_Hosts_File Now Change the URL To look like

    http://dl.dropbox.com/u/3212312671/Darkvader_Hosts_File?dl=1 Make sure to keep digits the same, which is your account number. Note: HTTPS don't work with a script only HTTP.

    You can test it, if it works using SSH or run a script from a firmware such as

    if you don't get any error then everything is good and working

    Now add your own hosts file to a CONFIG file as

    Restart your router and that's it.


    If you are running everything from USB media connected to a router then you can put path in sources to your Darkvader_Hosts_File.
     
  3. darksky

    darksky Networkin' Nut Member

    @haarp - You should really consider getting a free github account on which to host your code. Much nicera nd more professional than pastebin :p As a bonus, others who happen across your code may have suggestions for improvements and/or bug fixes. Github is a great platform for this dialog. Feel free to PM me if you wish.

    Also, what is the license under which adblock.sh is available? MIT?

    Thank you for the script by the way. Playing with it now.
     
    Last edited: Aug 13, 2013
  4. darksky

    darksky Networkin' Nut Member

    @haarp - I think I found a bug. If the user defines has ./blacklist and wants to add more to the list, adblock.sh does not grab the new entries.

    Example:
    1) The script ran and dnsmasq is setup.
    2) I add 's1.2mdn.net' to ./blacklist
    3) I update but the new host is not in the list.
    Code:
    # ./adblock.sh restart
    ADBLOCK: Stopping
    ADBLOCK: Done, restarting dnsmasq
    .
    Done.
    ADBLOCK: Download starting
    <SNIP>
    ADBLOCK: Filters unchanged
    ADBLOCK: Done, restarting dnsmasq
    ..
    Done.
    ADBLOCK: Exiting
    
    # grep s1.2mdn.net blocklist
    (nothing is returned)
     
  5. HunterZ

    HunterZ LI Guru Member

    @darksky: I don't believe that the script currently tracks changes in the whitelist/blacklist files. Deleting the blocklist (not to be confused with the blacklist) and re-running the adblock script is probably the best way to pick up the change, unless you want to manually make the changes to the blocklist file and then restart adblock.
     
  6. fun.k

    fun.k Addicted to LI Member

    guys, i have a simple question (seems to me that it is more related to routing than the adblock script, but i'm just an artist):

    can this script work on a tomato AP that is connected to a potato main router?

    setup details:

    Thomson TG585v8 (@192.168.1.254 doing aDSL, DHCP, DNS, the works)-->WRT54GLv1.1 running v1.28.7634-Toastman-IPT-ND-MiniIPv6 (@192.168.1.1 doing Wireless only on the upper floor)

    basic_network.png advanced_dhcp.png

    if i manually set the IP (and choose the tomato AP as gateway&dns) on the wireless laptops, adblock works. if i leave them to auto/dhcp, then the main router serves IPs, and the script does not work.

    manual_ip_works.png
     
    Last edited: Aug 28, 2013
  7. HunterZ

    HunterZ LI Guru Member

    @fun.k: The script only works on whatever router you are using as a DNS server on your network, because it works by configuring dnsmasq to map a list of domain names to a specified IP address.

    Also, why would you want to configure the Tomato as a gateway? At home, I have an Asus RT-N16 downstairs running Tomato as my gateway+everything at 192.168.1.1, and a WRT54G upstairs running Tomato in Access Point mode at 192.168.10.1 with DHCP disabled. Both routers use the same wireless SSID and password but run on different channels, so that clients can see and roam between both. All clients see and use 192.168.1.1 as the gateway even if connecting via the WRT54G.
     
  8. fun.k

    fun.k Addicted to LI Member

    thanks for confirming & clearing it up HunterZ :)

    i usually set up everything w/ tomatos (identical to your set up). however, i'm in a temporary location and have no access to a spare adsl modem (in order to use the tomato'd WRT54GL as a gateway/main router), so i gave this setup a try.

    thanks to everyone involved in this script, i'm so used to blocking ads w/ tomato that everywhere else i go and see ads all over makes me homesick (and hurry to enable the ABP extension in my browser).
     
  9. Badders44

    Badders44 LI Guru Member

    Is any one else getting failures with http://hosts-file.net/ad_servers.asp? It resolves to http://hosts-file.net/.\Cad_servers.txt which seems ok.

    Code:
    Aug 29 04:00:01 Router user.info sched[3267]: Performing scheduled custom #1...
    Aug 29 04:00:01 Router user.notice ADBLOCK: Download starting
    Aug 29 04:00:02 Router user.notice ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Mon, 12 Aug 2013 08:22:13 GMT)
    Aug 29 04:00:02 Router user.notice ADBLOCK: Unchanged: http://sysctl.org/cameleon/hosts (Last-Modified: Thu, 08 Aug 2013 00:31:30 GMT)
    Aug 29 04:00:02 Router user.notice ADBLOCK: Unchanged: http://adaway.sufficientlysecure.org/hosts.txt (Last-Modified: Sat, 20 Jul 2013 18:49:11 GMT)
    Aug 29 04:00:03 Router user.notice ADBLOCK: Failed: http://hosts-file.net/ad_servers.asp
    Aug 29 04:00:03 Router user.notice ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Tue, 27 Aug 2013 07:56:29 GMT)
    Aug 29 04:00:03 Router user.notice ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Sat, 24 Aug 2013 13:18:55 GMT)
    Aug 29 04:00:03 Router user.notice ADBLOCK: Filters unchanged
    Aug 29 04:00:03 Router user.notice ADBLOCK: Exiting
     
  10. HunterZ

    HunterZ LI Guru Member

    Badders44 likes this.
  11. Darkvader

    Darkvader Serious Server Member

    Hi there is it possible to monitor what web sites are been block by the Adblocker and if so can you explain how to do it.
    thanks
     
  12. HunterZ

    HunterZ LI Guru Member

    In your Tomato GUI, go to Advanced->DHCP/DNS and add the following line to the Dnsmasq Custom Configuration box:
    Code:
    log-queries
    Then hit Save, then monitor the router logs and you will see the result of *every* DNS lookup performed by the router. You will be able to tell if blocking is happening by looking for results that map to your configured redirection IP address.

    You will probably not want to leave this setting in place for long, as it will likely fill up the log file pretty quickly.
     
  13. JAC70

    JAC70 Networkin' Nut Member

  14. HunterZ

    HunterZ LI Guru Member

    Are you sure the problem isn't that you also need to whitelist them without the www prefix?
     
  15. JAC70

    JAC70 Networkin' Nut Member

    Yep, tried that, thanks, but they still both resolve to my Pixelserv IP.
     
  16. Runkle

    Runkle Networkin' Nut Member

    excellent script, many thanks. how can I donate?

    also, is the source posted somewhere? I would like to reverse engineer to improve my skills.
     
  17. darksky

    darksky Networkin' Nut Member

    Is there an echo in here :)

     
  18. HunterZ

    HunterZ LI Guru Member

    Pasting and executing the encoded script just causes it to decode itself to the real one at $PREFIX/adblock.sh, so just take a look at that.
     
  19. darksky

    darksky Networkin' Nut Member

  20. JoeDirte

    JoeDirte Serious Server Member

    Just wanted to say thanks for this. I've been using this for about a week and it's great! Love that I don't get ads when watching Hulu+ or Netflix. The pixelserv is nice too. I like blank banners versus 404 error banners I used to get when using a local mvp hosts file. Nice to have it all on the router and updated automatically. Good stuff!
     
  21. remlei

    remlei Networkin' Nut Member

    As soon as I start using this script, the local DNS server stops responding (using google DNS by entering it manually on my computer's IP setting make it works again) and makes the router lag a lot as it uses lots of CPU process @500mhz, DNSmasq doesnt resolve anything, also there's a weird happening on my USB drive since it blinks a lot like nonstop (I triple checked that no one use the USB storage device except adblock) and it still blinks even after 30minutes. Still there's a internet access as I can still ping remote sites via their ip addresses.

    Here's are some logs see if there's somethings wrong I done in my side.

    Starting up adblock: http://pastebin.com/PTX6nkFU
    Router Logs adblock while process the blacklist and after it finished processing it: http://pastebin.com/TasdA99u

    CPU usage while running adblock
    Firmware: Tomato Firmware 1.28.0000 MIPSR2-112 K26 USB Nocat-VPN

    Any suggestions will be appreciated.
     
  22. koitsu

    koitsu Network Guru Member

    Provide full output from ps -w when the issue is happening. It's obvious some daemon or script (NOT the kernel) or process is taking up 92% CPU time.
     
  23. Beast

    Beast Network Guru Member

    I am still using v3.9e, can you tell me why I should update? What exactly are the benefits?
     
  24. rs232

    rs232 Network Guru Member

  25. Link2User

    Link2User Serious Server Member

    Hunterz how do i get Entware installed so i can block https? i've tried going through the guide but it comes up with errors /tmp/mnt/Opt# wget -O - http://wl500g-repo.googlecode.com/svn/ipk
    g/entware_install.sh , mkdir: can't create directory '/opt/bin': Read-only file system etc etc.

    my flash gets mounted to /tmp/mnt/Opt.

    If you have the time to write up a little guide?

    i'm running the latest shibby on a RTN-16
     
  26. Goggy

    Goggy Network Guru Member

  27. Goggy

    Goggy Network Guru Member

    Imho there is not really a need to install entware /stunnel ... - update to pixelserv v32 and follow this post
     
  28. Kye-U

    Kye-U Addicted to LI Member

    This is working very well with pixelserv v32 :)
     
  29. rs232

    rs232 Network Guru Member

    Lol it's even in the sample config file! I should have pay more attention, thanks for the input!

    Another question, why is my blocklist empty? Beside this the script doesn't seem to be working for me but I can't figure out what I'm doing wrong...
    as a test I'm trying to open in the browser one of the hosts from the lists I've enabled but I can still access it.

    Finally, do you see any problem in running the very same script from different devices (all pointing to the same /cifs share)?

    Code:
    root@unknown:/cifs1/adblock# /cifs1/adblock/adblock.sh
    Connecting to X.X.X.15 (X.X.X.15:80)
    ADBLOCK: Download starting
    ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Mon, 23 Sep 2013 16:38:04 GMT)
    ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Sat, 24 Aug 2013 13:18:55 GMT)
    ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Wed, 25 Sep 2013 21:37:11 GMT)
    ADBLOCK: Filters unchanged
    ADBLOCK: Exiting
    Code:
    root@unknown:/cifs1/adblock# ls -la
    drwxrwxrwx    2 110      100              0 Oct  3 11:53 .
    drwxrwxrwx    9 110      100              0 Oct  3 11:05 ..
    -rwxrwxrwx    1 110      100          4735 Jul 12 09:04 adblock.sh
    -rwxr--r--    1 110      100              0 Oct  3 11:42 blacklist
    -rwxr--r--    1 110      100              0 Oct  3 11:42 blocklist
    -rwxrwxrwx    1 110      100          2049 Oct  3 11:52 config
    -rwxrwxrwx    1 110      100            45 Oct  1 11:55 lastmod-3b41114e
    -rwxr--r--    1 110      100            45 Oct  3 11:42 lastmod-88096bb7
    -rwxrwxrwx    1 110      100            45 Jul  3 19:18 lastmod-9ff39132
    -rwxrwxrwx    1 110      100            45 Aug 28 11:32 lastmod-c2934517
    -rwxrwxrwx    1 110      100            45 Sep 23 19:13 lastmod-da9bd190
    -rwxrwxrwx    1 110      100          11720 Jul 12 09:05 pixelserv
    -rwxrwxrwx    1 110      100          32612 Oct  1 11:55 source-3b41114e
    -rwxr--r--    1 110      100        267904 Oct  3 11:42 source-88096bb7
    -rwxrwxrwx    1 110      100          3956 Jul  3 19:18 source-9ff39132
    -rwxrwxrwx    1 110      100              0 Sep 23 19:24 source-c2934517
    -rwxrwxrwx    1 110      100              0 Sep 23 19:24 source-da9bd190
    -rwxr--r--    1 110      100            363 Oct  3 11:53 whitelist
    
     
    Last edited: Oct 3, 2013
  30. Goggy

    Goggy Network Guru Member

    I'm using this script with whitlist, blacklist and a huge blocklist without a really noticeable delay (surfing, ...):
    ADBLOCK: Config generated, 328911 unique hosts to block

    :)
     
  31. jerrm

    jerrm Network Guru Member

    Delete the blocklist file and/or run the script with the force parameter. If the script believes the filters are unchanged and a blocklist file exists (even an empty one), it does not regenerate the file. Why you have an empty blocklist is another question, but see if we can force it first.

    Yes. The code makes no allowances for multi-user access. Multiple instances could trip over each other, and cause things like an empty blocklist file. A few options, mod the script so that only a single router actually does the blocklist update process, or to use something like router_name or lan_ipaddr to create a "data" folder for each router, or manually create a folder for each router and link to and run the script from the router's folder.
     
  32. rs232

    rs232 Network Guru Member

    Thanks for this! I've removed everything except adblock.sh/pixelserve/config. Then run adblock.sh force

    Now I get:

    Code:
    root@tomato:/cifs1/adblock# ls -la
    drwxrwxrwx    2 110      nas              0 Oct  3 15:58 .
    drwxrwxrwx    9 110      nas              0 Oct  3 15:15 ..
    -rwxrwxrwx    1 110      nas          4735 Jul 12 09:04 adblock.sh
    -rwxr--r--    1 110      nas              2 Oct  3 15:58 blacklist
    -rwxr--r--    1 110      nas            24 Oct  3 15:56 blocklist
    -rwxrwxrwx    1 110      nas          2067 Oct  3 15:56 config
    -rwxr--r--    1 110      nas            45 Oct  3 15:56 lastmod-3b41114e
    -rwxr--r--    1 110      nas            45 Oct  3 15:56 lastmod-c2934517
    -rwxr--r--    1 110      nas            45 Oct  3 15:56 lastmod-da9bd190
    -rwxrwxrwx    1 110      nas          11720 Jul 12 09:05 pixelserv
    -rwxr--r--    1 110      nas          32612 Oct  3 15:56 source-3b41114e
    -rwxr--r--    1 110      nas        283292 Oct  3 15:56 source-c2934517
    -rwxr--r--    1 110      nas          42012 Oct  3 15:56 source-da9bd190
    -rwxr--r--    1 110      nas            363 Oct  3 15:58 whitelist
    The blocklist is not empty any more (though very small) and I can still browse addresses I can find e.g. in source-c2934517 (http://www.adbanner.ro/)
     
  33. jerrm

    jerrm Network Guru Member

    Blocklist should be 100K+ based on the source files. Have to wonder if there has been some sort of cut/paste error. Or maybe there is something in the whitelist file over-matching.

    Can you post the whitelist file and the confgen() section of the adblock script you're running?

    Edit: also post the generated blocklist file.
     
  34. rs232

    rs232 Network Guru Member

    Ok this is what I did:
    - re-wiped everything apart from pixelserv
    - recreated adblock.sh and config as per post #1 (copy+paste into GUI tools/system), and yes I do use cifs1/adblock/ too
    - run an adblock.sh force
    - tested --> it works!!!

    Then... I go adding again my custom white/black list to the bottom of the config file (whitelist only no blacklist):

    wget http://192.168.X.X/whitelist.txt -O - > "whitelist"

    Run:
    adblock.sh stop
    adblock.sh force

    But this way I run exactly in the same problem as per above (blocklist empty):

    whitelist:
    Code:
    1shoppingcart.com
    affiliatefuture.com
    affiliatewindow.com
    aweber.com
    awin1.com
    clickbank.net
    doubleclick.net
    friendlyduck.com
    gmail.com
    google.com
    infusionsoft.com
    letitbit.net
    linkshare.com
    liveperson.net
    lphbs.com
    mcssl.com
    nwolb.com
    rapidshare.com
    rover.ebay.co.uk
    rover.ebay.com
    tradedoubler.com
    videolan.mirrors.uk2.net
    vimeo.com
    vitacost.com
    webgains.com
    
    
    
    Where the bottom of the config file looks like this:
    Code:
    ### Blacklist additional sites ###
    ## (add hostnames inside the quotes, space-separated, without http://) ##
    BLACKLIST=""
    
    ### Whitelist sites from blocking ###
    ## (add hostnames inside the quotes, space-separated, without http://) ##
    WHITELIST="de.ign.com followerscounter.com redirectingat.com"
    
    ### Blacklist and Whitelist files (optional) ###
    ## create the files "blacklist" and "whitelist" with your hosts, one per line ##
    ## useful if you have many hosts in these (they generate faster aswell!) ##
    wget http://192.168.X.X/whitelist.txt -O - > "whitelist"
    

    So one question I have is: can the

    Code:
    WHITELIST="de.ign.com followerscounter.com redirectingat.com"
    and

    Code:
    wget http://192.168.X.X/whitelist.txt -O - > "whitelist"
    live together in the config file?
     
  35. Goggy

    Goggy Network Guru Member

    I wonder why you do not just 'try it yourself? Apart from that I see no point to use "WHITELIST" and "WHITELIST-FILES" parallel?
     
  36. rs232

    rs232 Network Guru Member

    It was a rhetoric question as this (use only wget or both) was tried and it doesn't work for me.

    What I noticed is: is the whitelist file is present in the directory and force a restart the blocklist will be empty.

    If I remove the whitelist and force a restart the blocklist file is filles about 750K of data
     
    Last edited: Oct 4, 2013
  37. rs232

    rs232 Network Guru Member

    Ok, the problem appears to be related to white/blacklist files with one or more blank lines. See my whitelist above it has two blank lines at the end. Of course I can remove these manually but since the file is edited my multiple people and in general have a blank line at the bottom of the file is not so uncommon. I thought it would be a good idea to have the code preventing this scenario.

    I've achieved this changing:

    Code:
      if [ -e "$prefix/whitelist" ]; then
      sort -u "$listprefix"/source-* | grep -v -f "$prefix/whitelist" > "$listprefix/blocklist"
    into

    Code:
      if [ -e "$prefix/whitelist" ]; then
      cat "$prefix/whitelist" | grep -v '^$' > whitelist.tmp | mv whitelist.tmp whitelist
      sort -u "$listprefix"/source-* | grep -v -f "$prefix/whitelist" > "$listprefix/blocklist"
    and

    Code:
      [ -e "$prefix/blacklist" ] && {
      cat "$prefix/blacklist" >> "$listprefix/blocklist"
    
    into

    Code:
      [ -e "$prefix/blacklist" ] && {
      cat "$prefix/blacklist" | grep -v '^$' > blacklist.tmp | mv blacklist.tmp blacklist
      cat "$prefix/blacklist" >> "$listprefix/blocklist"
    I guess not the smartest of the options but it works.

    rs232
     
    Last edited: Oct 4, 2013
    Goggy likes this.
  38. jerrm

    jerrm Network Guru Member

    Overmatching whitelist - thought that might be the problem.

    I'd add tr to remove extraneous whitespace too:
    Code:
    tr -d " \t"
    or replace the extra grep with sed to do both in one command:
    Code:
    sed 's/[ |\t]*//g; /^$/d'
     
    Last edited: Oct 4, 2013
    Goggy likes this.
  39. Link2User

    Link2User Serious Server Member

    Thanks for the feedback, I've been using it since I posted and it's doing really well, I just have to sort of my lists because a few forums won't show user pics.
     
  40. Ravi Baskaran

    Ravi Baskaran Reformed Router Member

    Does this script work with Google DNS settings? I have a Tomato USB by toastman firmware on linksys E3000 router. Unfortunately, if I turn on adblock.sh, it is not able to resolve any IP address :(.

    Any help is appreciated.
     
  41. Almaz

    Almaz Serious Server Member


    It should work fine with any DNS. Check your settings.
     
  42. Ravi Baskaran

    Ravi Baskaran Reformed Router Member

    I am unable to get this script working with my router :(. Here is what I see when I execute the script.
    Code:
    root@unknown:/jffs/adblock# ./adblock.sh force
    ADBLOCK: Download starting
    Connecting to www.malwaredomainlist.com (143.215.130.61:80)
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    -                    100% |***********************************************************************************| 52711  0:00:00 ETA
    -                    100% |***********************************************************************************|  551k  0:00:00 ETA
    Connecting to pgl.yoyo.org (95.172.9.82:80)
    -                    100% |***********************************************************************************| 67772  0:00:00 ETA
    ADBLOCK: Downloaded
    ADBLOCK: Generating /jffs/adblock/blocklist
    sed: write error
    sed: write error
    sed: write error
    sed: write error
    ADBLOCK: Config generated, 17281 unique hosts to block
    ADBLOCK: Setting up pixelserv on 192.168.1.254
    pixelserv[1240]: /jffs/adblock/pixelserv V31 compiled: Jun  1 2013 23:40:13 from pixelserv31.c
    ADBLOCK: Done, restarting dnsmasq
    
    Done.
    ADBLOCK: Exiting
    
    Also, after executing the command, ls -lah shows the following
    Code:
    root@unknown:/jffs/adblock# ls -lah
    drwxr-xr-x    2 root    root          0 Oct  7 16:53 .
    drwxr-xr-x    4 root    root          0 Jan  1  1970 ..
    -rwxr-xr-x    1 root    root        4.6K Oct  6 17:26 adblock.sh
    -rw-r--r--    1 root    root      338.7K Oct  7 16:53 blocklist
    -rw-r--r--    1 root    root        1.9K Oct  6 17:30 config
    -rw-r--r--    1 root    root          45 Oct  7 16:52 lastmod-3b41114e
    -rw-r--r--    1 root    root          45 Oct  7 16:52 lastmod-c2934517
    -rw-r--r--    1 root    root          45 Oct  7 16:53 lastmod-da9bd190
    -rwxr-xr-x    1 root    root      11.4K Oct  6 17:24 pixelserv
    -rw-r--r--    1 root    root      31.9K Oct  7 16:52 source-3b41114e
    -rw-r--r--    1 root    root      276.7K Oct  7 16:52 source-c2934517
    -rw-r--r--    1 root    root      40.9K Oct  7 16:53 source-da9bd190
    Again, any help is appreciated :).
     
  43. Kye-U

    Kye-U Addicted to LI Member

    Do you have ramlist = 1 in the config file? (you should if your router is low on storage space)
     
  44. Ravi Baskaran

    Ravi Baskaran Reformed Router Member

    I did not run our of space on jffs partition. df -h shows about 100 kb of free space available in this partition.

    Do you think anything else might have gone wrong?

    Sent from my Micromax A116 using Tapatalk 4
     
  45. Ravi Baskaran

    Ravi Baskaran Reformed Router Member

    One more thing I noticed on my router was that though /jffs partition is very small, other partitions seems to have generous space.
    Code:
    root@unknown:/tmp/home/root# df -h
    Filesystem                Size      Used Available Use% Mounted on
    /dev/root                6.2M      6.2M        0 100% /
    tmpfs                    30.0M      1.2M    28.8M  4% /tmp
    devfs                    30.0M        0    30.0M  0% /dev
    /dev/mtdblock3          576.0K    476.0K    100.0K  83% /jffs
    As you can see the /dev partition has 30 MB of free space. Is it possible to create a partition or resize the /dev partition? I can host the adblock.sh and the related files into this partition.

    Any pointers ??
     
  46. jerrm

    jerrm Network Guru Member

    100kb is a miniscule amount of free space. I'd bet a reasonable sum you are running out of space during the process. Have you tried setting RAMLIST=1 as suggested?

    Oversimplifying, devfs can't be used for normal files and is basically useless in this context. That 30M is RAM, the same RAM showing for tmpfs and on your router status page. Just use the /tmp folder, or as suggested RAMLIST=1.
     
  47. Almaz

    Almaz Serious Server Member

    You can do what I do. Copy 3 needed files to dropbox public folder "adblock.sh, config & pixelserve". Create your own script in WanUP so it'll download all 3 files needed for the job in /tmp/adblock folder. Chmod files and run them from a script. I'm using my own hosts file so it also gets downloaded from my dropbox folder. Everything works from Ram and you have more than enough space to get the job done. Also to save more RAM you can add at the end
    rm /tmp/adblock/source* It'll remove downloaded hosts file.
     
    Last edited: Oct 8, 2013
  48. Ravi Baskaran

    Ravi Baskaran Reformed Router Member

    Thanks you for the responses, so far. I set RAMDISK=1 in the config file and now it is working fine :).

    I have one additional question. How do I start the adblock.sh automatically, after a power outage?
    BTW, will pixelserv start automatically if it crashes? If not, how to restart pixelserv automatically after a crash/outage?
     
  49. Kye-U

    Kye-U Addicted to LI Member

    :)
     
  50. Goggy

    Goggy Network Guru Member

    Just to be sure if i got it right:
    Code:
    if [ -e "$prefix/whitelist" ]; then
    cat "$prefix/whitelist" | sed 's/[ |\t]*//g; /^$/d' > whitelist.tmp | mv whitelist.tmp whitelist
    sort -u "$listprefix"/source-* | grep -v -f "$prefix/whitelist" > "$listprefix/blocklist"
    Code:
    [ -e "$prefix/blacklist" ] && {
    cat "$prefix/blacklist" | sed 's/[ |\t]*//g; /^$/d' > blacklist.tmp | mv blacklist.tmp blacklist
    cat "$prefix/blacklist" >> "$listprefix/blocklist"
    Thx!
     
  51. remlei

    remlei Networkin' Nut Member

    weird things happen when coreutils package (optware) is installed when using this adblock scipt.

    this hapens while coreutils is installed.

    Code:
    [: ==: binary operator expected
    [: ==: binary operator expected
    ADBLOCK: 'update' not understood!
    ADBLOCK: Exiting
    [root@unknown root]$ /opt/adblock/adblock.sh restart
    [: ==: binary operator expected
    [: ==: binary operator expected
    ADBLOCK: Stopping
    ADBLOCK: Done, restarting dnsmasq
    ..
    Done.
    ADBLOCK: Download starting
    [: extra argument `=='
    [: extra argument `=='
    [: extra argument `=='
    --2013-11-15 13:22:00--  http://www.malwaredomainlist.com/hostslist/hosts.txt
    --2013-11-15 13:22:00--  http://winhelp2002.mvps.org/hosts.txt
    Resolving winhelp2002.mvps.org... --2013-11-15 13:22:00--  http://hosts-file.net/ad_servers.asp
    Resolving www.malwaredomainlist.com... Resolving hosts-file.net... 216.155.126.40
    Connecting to winhelp2002.mvps.org|216.155.126.40|:80... 143.215.130.61
    Connecting to www.malwaredomainlist.com|143.215.130.61|:80... 50.17.116.14
    Connecting to hosts-file.net|50.17.116.14|:80... [: extra argument `=='
    --2013-11-15 13:22:00--  http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
    Resolving pgl.yoyo.org... 95.172.9.82
    Connecting to pgl.yoyo.org|95.172.9.82|:80... connected.
    connected.
    HTTP request sent, awaiting response... HTTP request sent, awaiting response... connected.
    HTTP request sent, awaiting response... connected.
    HTTP request sent, awaiting response... 302 Object moved
    Location: .%5Cad_servers.txt [following]
    --2013-11-15 13:22:01--  http://hosts-file.net/.%5Cad_servers.txt
    Reusing existing connection to hosts-file.net:80.
    HTTP request sent, awaiting response... 200 OK
    Length: 566133 (553K) [text/plain]
    Saving to: `STDOUT'
    
    3% [===>                                                                                                                          ] 20,165      92.4K/s              200 OK
    Length: unspecified [text/plain]
    Saving to: `STDOUT'
    
        [<=>                                                                                                                          ] 0          --.-K/s              200 OK
    Length: 439116 (429K) [text/plain]
    Saving to: `STDOUT'
    
    0% [                                                                                                                              ] 0          --.-K/s              200 OK
    Length: 53659 (52K) [text/plain]
    Saving to: `STDOUT'
    
    100%[=============================================================================================================================>] 53,659      28.2K/s  in 1.9s
    
    2013-11-15 13:22:04 (28.2 KB/s) - written to stdout [53659/53659]
    
        [        <=>                                                                                                                  ] 67,533      19.1K/s  in 3.5s
    
    2013-11-15 13:22:05 (19.1 KB/s) - written to stdout [67533]
    
    96% [========================================================================================================================>    ] 422,909    75.3K/s  eta 0s      [: extra argument `=='
    100%[=============================================================================================================================>] 566,133    92.0K/s  in 6.1s
    
    2013-11-15 13:22:08 (90.2 KB/s) - written to stdout [566133/566133]
    
    100%[=============================================================================================================================>] 439,116    77.1K/s  in 5.6s
    
    2013-11-15 13:22:08 (77.1 KB/s) - written to stdout [439116/439116]
    
    --2013-11-15 13:22:09--  http://adaway.sufficientlysecure.org/hosts.txt
    Resolving adaway.sufficientlysecure.org... 91.250.99.24
    Connecting to adaway.sufficientlysecure.org|91.250.99.24|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 17431 (17K) [text/plain]
    Saving to: `STDOUT'
    
    100%[=============================================================================================================================>] 17,431      34.4K/s  in 0.5s
    
    2013-11-15 13:22:11 (34.4 KB/s) - written to stdout [17431/17431]
    
    [: ==: binary operator expected
    [: ==: binary operator expected
    ADBLOCK: Download failed
    ADBLOCK: Setting up pixelserv on 192.168.0.254
    pixelserv[4435]: /opt/adblock/pixelserv V31 compiled: Jun  1 2013 23:40:13 from pixelserv31.c
    ADBLOCK: Done, restarting dnsmasq
    ...........................................................
    Done.
    ADBLOCK: Exiting
    
     
  52. koitsu

    koitsu Network Guru Member

    More than likely coreutils installs [ (a.k.a. test) that doesn't use the same syntax as what comes with Busybox. Optware is known for stomping over $PATH and sticking its own stuff first, which causes certain problems.

    Please provide the full contents of the "adblock.sh" script here as an attachment and I can tell you what's wrong. To me, it looks like someone is using == for a comparison operator within test, which is wrong -- the comparison operator is =. But that's me making blind assumptions.
     
  53. gugalnica

    gugalnica Reformed Router Member

    Hello!

    Today I decided I will start blocking ads at router. I found your script and I followed the instructions, but it doesn't work. I have Optware installed on my WRT54GL, as I'm also running STunnel, Samba and Webserver. I installed this script on my USB, so my prefix is: /opt/adblock .

    When I start the script this is what I get:

    Code:
    /opt/adblock$ /opt/adblock/adblock.sh
    /opt/adblock/adblock.sh: line 112: awk: not found
    /opt/adblock/adblock.sh: line 112: awk: not found
    ADBLOCK: Download starting
    /opt/adblock/adblock.sh: line 136: awk: not found
    /opt/adblock/adblock.sh: line 136: awk: not found
    nc: bad address '80'
    /opt/adblock/adblock.sh/opt/adblock/adblock.sh: line 136: awk: not found
    : line 136: awk: not found
    /opt/adblock/adblock.sh: line 136: awk: not found
    /opt/adblock/adblock.sh: line 136: awk: not found
    nc: bad address '80'
    nc: bad address '80'
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    /opt/adblock/adblock.sh: line 136: awk: not found
    Connecting to pgl.yoyo.org (95.172.9.82:80)
    Connecting to www.malwaredomainlist.com (143.215.130.61:80)
    /opt/adblock/adblock.sh: line 136: awk: not found
    /opt/adblock/adblock.sh: line 136: awk: not found
    ADBLOCK: Failed: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
    ADBLOCK: Failed: http://winhelp2002.mvps.org/hosts.txt
    ADBLOCK: Failed: http://www.malwaredomainlist.com/hostslist/hosts.txt
    ADBLOCK: Download failed
    ADBLOCK: Exiting
    My config file is and everything else is same as it is on first page of instructions, only PREFIX is changed.

    Can you help me with what is wrong?

    Thanks in advance.
     
  54. remlei

    remlei Networkin' Nut Member

    hope you can fix it for us.
     

    Attached Files:

  55. koitsu

    koitsu Network Guru Member

    Please try this. All I did was change all occurrences of == to = in test/[ operators.

    P.S. -- This script is pretty awful in a lot of regards; lots of assumptions and accidents waiting to happen. Though part of the limitations come from Busybox sh.
     

    Attached Files:

  56. rs232

    rs232 Network Guru Member

    Gents, I've just noticed one important thing. Many common domains are blocked by adblock.
    I had problems with slow internet for sometime and finally figure out that it was the adblock script. My Internet experience changed completely when I added the following 3 domains to the white-list:

    www.geoplugin.net
    cdn.optimizely.com
    ssl.google-analytics.com

    and btw the first one is used by this forum! Before white listing it I was experiencing slow browsing and some time the page wasn't loading the .css

    One thing I don't understand though is: isn't the pixserve meant to avoid this scenario? Responding with a white pixel instead of having a connection timeout?

    Perhaps my pixserve is not working fine? I can ping .254 from the router itself but not from my laptop. Is this meant to be?

    Thanks!
    rs232
     
    Last edited: Nov 27, 2013
  57. jerrm

    jerrm Network Guru Member

    The script's default firewall rules block everything except port 80 to the .254 address, so ping doesn't work from the laptop.

    For the others, a lot of it depends on what is being requested and whether pixelserv handles it. For example, the default rules would just drop an ssl request without a reset. If you're running a newer version of pixelserv, and modified the script accordingly, it attempts to more gracefully fail the ssl connect attempt, but I'm not really sure how successful it is. If the file request is not a for an extension pixelserv supports, it may cause issues.
     
  58. jerrm

    jerrm Network Guru Member

    And some of the assumptions were no doubt necessary originally in an attempt to stay under the 4K nvram limit. Nvram-ability has since been abandoned.

    I've modified it quite a bit, but run a version at home that is still probably 90% what haarp has posted, and it performs pretty well. It's not something I would ever try putting in for a client though.
     
  59. rs232

    rs232 Network Guru Member

    Thanks for that! I've notice that 254 is actually responding to a web request so, thanks for clarifying this for me!

    About the pixelserve options... I guess I can add them in the config file, but where are they documented if I might ask?

    Many thanks!
     
  60. jerrm

    jerrm Network Guru Member

    There really isn't anything to do in the config file.

    Look at the pixelserv thread, but it's gotten a little fragmented. There is a compiled v32 that includes some ssl code, but only listens on a single port, so you either need two copies running, or an iptables redirect. A v33 after that posted to git that adds an attempt to decode URL in the query and redirect, still only listening on a single port. Another v34 from someone else there that will listen on both multiple ports, but without the URL decode - that one might need a conf option for the second port.

    I'm running the v33 git version I compiled myself.
     
  61. mstombs

    mstombs Network Guru Member

    Haha its the v33 in git that's 'from someone else', V32 on these boards skipped to V34 http://www.linksysinfo.org/index.ph...run-on-router-wrt54g.30509/page-3#post-235434 . I didn't understand the 'decode URL' functionality, so didn't break what I couldn't test, the second port not currently configurable - hardcode to default https 443 (which seems to break on dd-wrt routers), and yes this does cause different error messages on different browsers. Any advert using non-standard port or expecting specific replies from its ad-server will still cause web stall page loads. Its also possible to have more variety of null text responses (css etc), but I am not convinced any benefit, I am also not really sure different image types are really needed.

    I'm so used to adblocking at home - I want a version that runs on my Android phone when travelling please!
     
  62. jerrm

    jerrm Network Guru Member

    I know, poor phrasing, no offense intended. Just trying to point out the need to parse though the various iterations.

    I have noticed the url decode being of some benefit - a deal site where some of the links get to the "real deal" where they didn't without it. Wouldn't worry about it if the feature was missing.
     
  63. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    Hi, I installed the script and it works, Pixelserv is installed, but sadly I had to disable the adblocking because the speed of SOME pages loading is in the order of minutes!

    For an example dyndns.com when going to login page sometimes takes ages.
    I was able to pinpoint to somekind of https connection when from a denied host it's trying to load a .js.
    Don't know yet if the .js is the problem or the https connections, but makes my browsing close to unusable!

    I have a DD-WRT router also that uses the same technique minus the pixelserv and the exact same pages are working great, don't know if the script or some TOMATO CONFIG is the problem!
    Any help Ideas would be appreciated!
     
  64. darkknight93

    darkknight93 Networkin' Nut Member

    This issue is related to ads deployed via https instead of our blocked http served ads.
    I will upload a iptables script to redirect https to http site asap. Note: this will show error messages embedded in the websites instead/at the place where the ads were due https traffic can not be served By pixelserv in the current version we use
     
  65. darkknight93

    darkknight93 Networkin' Nut Member

    So here is the modification i did for redirecting https traffic:
    Code:
    [..]
    stop() {
     elog "Stopping"
     rm "$CONF" &>/dev/null
     killall pixelserv &>/dev/null
     ifconfig $BRIDGE:1 down &>/dev/null
     iptables -D INPUT -p all -d $redirip -j REJECT &>/dev/null
     iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT &>/dev/null
     iptables -t nat -D PREROUTING -p tcp -d $redirip --dport 443 -j DNAT --to $redirip:80 &>/dev/null
     iptables -D INPUT -s $redirip -p all -j logdrop &>/dev/null
     iptables -D FORWARD -s $redirip -p all -j logdrop &>/dev/null
     elog "Done, restarting dnsmasq"
     service dnsmasq restart
    }
    [..]
    
    if [ "$PIXEL_IP" != "0" ]; then
     if ps | grep -v grep | grep -q "$prefix/pixelserv $redirip"; then
     elog "pixelserv already running, skipping"
     else
     elog "Setting up pixelserv on $redirip"
    
     iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
     iptables -I INPUT -p all -d $redirip -j REJECT
     iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
     iptables -t nat -A PREROUTING -p tcp -d $redirip --dport 443 -j DNAT --to $redirip:80
     iptables -I INPUT -s $redirip -p all -j logdrop
     iptables -I FORWARD -s $redirip -p all -j logdrop
     }
     ifconfig $BRIDGE:1 $redirip up
     "$prefix/pixelserv" $redirip $PIXEL_OPTS
     fi
    fi
    
    
     
  66. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    It works very well, Thank you! Your mod should go in the main script, I just don't get how nobody was affected by this.
    One more thing I added &>/dev/null after logdrop (is it a good change??!!?) so the script looks actually like below, or it would throw an error after adblock.sh restart.
    Code:
    [..]
    stop() {
      elog "Stopping"
      rm "$CONF" &>/dev/null
      killall pixelserv &>/dev/null
      ifconfig $BRIDGE:1 down &>/dev/null
      iptables -D INPUT -p all -d $redirip -j REJECT &>/dev/null
      iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT &>/dev/null
      iptables -t nat -D PREROUTING -p tcp -d $redirip --dport 443 -j DNAT --to $redirip:80 &>/dev/null
      iptables -D INPUT -s $redirip -p all -j logdrop &>/dev/null
      iptables -D FORWARD -s $redirip -p all -j logdrop &>/dev/null
      elog "Done, restarting dnsmasq"
      service dnsmasq restart
    }
    [..]
    if [ "$PIXEL_IP" != "0" ]; then
      if ps | grep -v grep | grep -q "$prefix/pixelserv $redirip"; then
        elog "pixelserv already running, skipping"
      else
      elog "Setting up pixelserv on $redirip"
      iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
        iptables -I INPUT -p all -d $redirip -j REJECT
        iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
        iptables -t nat -A PREROUTING -p tcp -d $redirip --dport 443 -j DNAT --to $redirip:80
        iptables -I INPUT -s $redirip -p all -j logdrop &>/dev/null
        iptables -I FORWARD -s $redirip -p all -j logdrop &>/dev/null
      }
      ifconfig $BRIDGE:1 $redirip up
      "$prefix/pixelserv" $redirip $PIXEL_OPTS
      fi
    fi
     
    darkknight93 likes this.
  67. darkknight93

    darkknight93 Networkin' Nut Member

    Many thanks! I think I forgot there someth
     
  68. jerrm

    jerrm Network Guru Member

    Above assumes you have logging enabled.

    Should be noted this assumes v32 of pixelserv. V34 will listen on 443 without needing the redirect. I don't know what would happen with a version prior to v32.

    This still assumes a generic REJECT without --reject-with tcp-reset was the best action, but that has already been discussed here. Just adding a tcp rule with --reject-with tcp-reset takes care of most of the slow page loads without the ssl redirect (but I think the redirect with v32 or using v34 is better).

    The script is still completely broken if anything resets iptables, but no one seems to be bothered by that and my version has diverged enough that it is not an easy set of patches to post.
     
    Last edited: Nov 28, 2013
  69. QSxx

    QSxx LI Guru Member

    I would greatly appreaciate if you would post your version then (or create thread where you could develop it further).

    Fellow adblocking addict here :)
     
  70. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    jerrm, as you stated, I tried to update Pixelserv to v34 and revert the script to it's initial form, but, the slowdowns are still there...
    Reverting back to darknight93's mod cured it again, it seems v34 still doesn't catch https properly. I only tested dyndns.com.
    I use Shibby's mod, maybe Toastman or other Tomato Flavours have a different config.

    I tried finding other alternatives, but this is a very good script, mostly because it works from a separate file (Damn you 32k NVRAM), and because it's fast, permanent, keeping track of changes in host files...
    But, a new, improved version is always welcome!

    I attached this v4.5+ script if other people want it...
     

    Attached Files:

    Last edited: Nov 28, 2013
    darkknight93 likes this.
  71. mstombs

    mstombs Network Guru Member

    There "shouldn't" be any difference between diverting port 443 to port 80 and V34 listening on 443, but maybe the method attempted to reject the https connection just doesn't work or one of the other image changes is broken, but could it be a different port and the logdrop is more effective? Or maybe an earlier version of pixelserv is just closing the connection without starting a conversation?
    I'm afraid pixelserv v34 still seems to be working for me... been running since 14th October, current stats
    Code:
    Nov 24 20:30:01 rtn66u daemon.info pixelserv[13995]: 123300 req, 16802 err, 2086 gif, 112 bad, 94505 txt, 56 jpg, 76 png, 85 swf, 9578 ssl
    Nov 28 20:30:01 rtn66u daemon.info pixelserv[13995]: 135132 req, 18553 err, 2331 gif, 112 bad, 103050 txt, 58 jpg, 86 png, 92 swf, 10850 ssl
    If anyone has a specific problem site I can take a look with wireshark if you pm me the details.
     
    Last edited: Nov 28, 2013
  72. jerrm

    jerrm Network Guru Member

    With v34 you could not do a full revert of the script, you would need to change the rules to ALLOW port 443 to $redirip.

    Something like changing:
    Code:
    iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
    to
    Code:
    iptables -I INPUT -i $BRIDGE -p tcp -d $redirip -m multiport --dports 443,80 -j ACCEPT
     
    Last edited: Nov 29, 2013
  73. Frequenzy

    Frequenzy Networkin' Nut Member

    will this work with pixelserv 34. i'll just replace the original adblock.sh?
     
  74. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    I'm just a user jerrm, you already lost me...
    Anyway, it seems better to not redirect the 443 port and just catch it, but I'm not in the position to modify the script to reflect that, I just don't know how to do it.
    It seems so important the script knows how to catch HTTPS requests properly, that it should be implemented ASAP and posted on the main page, as most advertisers move to https connections.
    I'm still amazed that nobody was bothered by this, I only found one post that resembled mine on "All You Need" script thread.

    P.S.: @Frequenzy, Yes, it works I'm using v34 (v32 also) ATM and it works great!
     
  75. jerrm

    jerrm Network Guru Member

    Look again, the ssl issue has been discussed - with potential fixes - both in this thread and the pixelserv thread. mstomb's ssl updates to pixelserv are a result, at least in part, of those discussions.

    This script is haarp's project, not sure if he still considers it active or not. I for one am reluctant to "branch" off a new version with related support headaches.
     
    Last edited: Nov 29, 2013
  76. QSxx

    QSxx LI Guru Member

    Don't be :)

    This is a free community and a free forum. If you can help that community - you should. haarp said that he disallows interventions in t(his) script. If he had time he would probably fix that on his own.

    Branch away buddy :)

    P.S. Posting script on forum isn't considered a legal binding contract that makes you maintain / troubleshoot / support it too... AFAIK...
     
  77. jerrm

    jerrm Network Guru Member

    I'm hesitant because I know the script is not in the condition I would normally consider releasing into the wild. As koitsu said there are some issues with the original, and my mods are of the quick and dirty/personal use only variety.

    I'll look it over and see....
     
  78. Beast

    Beast Network Guru Member

    I run script v3.9e in the wanup with pixelsrv 3.4 located on a usb stick connected to the back of the RT-N16. I am still not clear as to what the advantage are of the newer version of the script.

    pixelserv[959]: 5819 req, 565 err, 58 gif, 0 bad, 5047 txt, 0 jpg, 0 png, 5 swf, 144 ssl
     
  79. jerrm

    jerrm Network Guru Member

    This was the original post and "Home" of my adblock script mod. To avoid confusion and aid support, a new thread has been started here: http://www.linksysinfo.org/index.php?threads/script-adblock-not-so-lean.72290/

    Please go to the link above for the latest version. This post will no longer be updated.

    THIS SCRIPT IS INTENDED FOR A MODERN TOMATO BUILD, FROM 2013 OR LATER. It makes use of multiple Tomato specific features. No real attempt has been made to be compatible with any other platforms.

    This is a mod of @Haarp's "Clean, Lean and Mean Adblocking." It's not so lean anymore, but the core functionality is still @Haarp's.

    The script is written to run on "stock" Tomato. Third party packages like Optware or Entware are NOT required.

    The script, release notes, and pixelserv can also be found at http://tomato-adblock.weebly.com.

    Web Interface
    upload_2015-9-8_21-57-13.png upload_2015-9-8_22-8-28.png
    The script supports and installs a web interface by default, it features:
    • start/stop adblock
    • display adblock status and stats
    • display pixelserv status and stats
    • click to add host(s) to whitelist/blacklist
    • edit adblock config
    • edit adblock whitelist/blacklist files
    The script runs under Tomato's standard admin http server and should be compatible with any recent build. An adblock link is added to the Tomato interface. The url for adblock is also output to the console and syslog. The url will usually be http://xxx.xxx.xxx.xxx/user/adbock.sh, inserting the appropriate router IP address.

    The pixelserv status function requires @HunterZ 's pixelserv v35.HZ8 or newer, otherwise an error message will be displayed.

    Storage
    The adblock scripts and install script assume there is persistent storage available, in the form of USB, JFFS, or CIFS. This version makes no attempt to live inside nvram storage constraints. For older units without persistent storage, some run the script downloading dynamically at run time into the tmp folder.

    Pixelserv
    Pixelserv is recommended, but not required. It will provide an improved user experience(fewer browser error messages), but ads will be blocked with or without it. Usage of pixelserv can be disabled by adding "PIXEL_IP=0" in the config file.

    The adblock script assumes pixelserv v32 or later to handle ssl requests. It should redirect 443 for v32 or appropriately allow 443 directly for pixelserv for v34 or later. Using one of @HunterZ 's latest versions is encouraged (https://github.com/HunterZ/pixelserv).

    Logging
    The web interface blocked/resolved hosts report listings require query logging be enabled in dnsmasq. Tomato does not do so by default.

    There are two ways to enable logging:
    1. In the Tomato GUI: add "log-queries" to "Advanced->DHCP/DNS->Dnsmasq Custom configuration" or...
    2. In the adblock config file add "dnsmasq_logqueries=1"
    Using the adblock config option is often preferable because loging can then be toggled on/off by editing the adblock config in the web interface.

    Log Location
    By default, dnsmasq will log to syslog. Dnsmasq is VERY noisy and can dump thousands of lines to syslog.

    If using syslog, make sure "Log Internally" is checked under "Administration->Logging" in the Tomato GUI. Also consider increasing "Max size before rotate" if there is not enough query history showing.

    Logging to a separate file is done by adding the dnsmasq "log-facility" option to "Advanced->DHCP/DNS->Dnsmasq Custom configuration."

    Scheduling Updates
    Including "cron" on the adblock command line, ie: adblock.sh cron, adds a daily job to the scheduler to update the list files.

    Default schedule is at 2:10am. Change the schedule by adding "schedule='10 02 * * *'" to the config file in standard crontab <"min hour day month week"> format.

    Config File
    The default settings file is adblock.ini. Not a very *nix-like name, but that is by design for reasons that don't really matter to anyone else.

    This is a change from @haarp's simply named "config" file, which I found too generic for my tastes. Adblock is still backward compatible with a file named "config" if it exists in the script folder and looks like an adblock file.

    Required Settings
    There is only one: SOURCES

    SOURCES must be defined for the lists to use. If using only manual blacklist entries then set SOURCES="".

    If pixelserv is not being used, then "PIXEL_IP=0" should also be set.

    All other @haarp options are still supported as well as many others, but are not required.

    See adblock.ini.readme for some of the more commonly set options. Additional options can be found by looking at the script source in the "Default values" section, but these rarely need to be touched.


    Recent Changelog Entries

    Full changelog can be found
    here.

    2015-11-11
    adblock - properly initialize redirip for recursive webui calls
    adblock - check/post error if both PIXEL_IP and redirip are set

    2015-11-08
    adblock - use appropriate netmask when setting up redirip
    install - update pixelserv to @HunterZ's V35.HZ13
    install - change from zip to tar archive (support older builds without unzip)
    install - reduce warnings/errors when installing to fat/cifs
    install - use copy if link doesn't work for pixelserv (for fat/cifs)

    2015-09-14
    adblock - add /mmc folders to config file search paths
    adblock/web - add warning text if dnsmasq logging is enabled without syslog enabled or log-facility set
    install - add support for K24 pixelserv build
    install - update pixelserv to @HunterZ's V.35HZ12 releases

    2015-09-06
    adblock - add most likely webscript url to log output
    adblock - add quietfire config option to disable firewall autorun syslog output, defaults to quietfire=1
    adblock - make adding link to tomato ui the default ( tomatolink=1 )
    adblock - whitelist/blacklist - better handling of comments
    adblock/web - force ps -w to better handle long paths
    web - change "resolved hosts" report logic, now shows requesting IP, see this discussion
    web - add option to use old "resolved hosts" report, set web_oldresolvedhosts=1 in config

    INSTALLING:
    Install via the code block below. The code block can be pasted into the Tomato Web GUI or ssh/telnet command line.

    This will download and install the adblock scripts and default config file. It also attempts to detect processor type and install the appropriate MIPS or ARM version of @HunterZ's pixelserv.

    Change PREFIX to match your install location. PREFIX must be a full path, it cannot be relative.

    If PREFIX is not defined, the script will first attempt to install to /opt/bin, then attempt to the first writable location from the following list: /opt/adblock, /jffs/adblock, /mmc/adblock, /cifs1/adblock, /cifs2/adblock.
    Code:
    # For a custom location uncomment and edit PREFIX value
    # export PREFIX=/opt/bin
    wget -O - http://goo.gl/GfA7cQ | sh
    
    Manual Install
    • Download script archive
    • Unzip files into desired folder
    • Copy sample config file adblock.ini.default to adblock.ini, place in same folder as script
    • Edit config file as needed
    • Download and install pixelserv to the script folder if desired
    • Run the script
    RUNNING:
    Adblock has traditionally been loaded from Tomato's wanup script, ie:
    Code:
    /opt/bin/adblock.sh cron
    The primary problem with the above approach is wanup can be triggered multiple times and repeatedly if there are connectivity issues or when the wan IP changes. Once adblock is initialized, adblock.sh does not need to be called repeatedly in these circumstances.

    The solution is to test if adblock is already enabled using the "adblock.is.loaded" test hostname:
    Code:
    nslookup adblock.is.loaded || /opt/bin/adblock.sh cron &
     
    Last edited: Mar 13, 2016
  80. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    It works flawlessly... My god, it's PERFECT! I really like the changes you brought to the script, it's cleaner, leaner and meaner!
    EDIT: I deleted the base64 scripts, as Jerrm posted a very easy method to get the script on the router!
    PixelServ Version 34, - pixelserv v34
    Also after install I copied my configuration file to /opt/etc/adblock.ini
     
    Last edited: Dec 2, 2013
  81. Goggy

    Goggy Network Guru Member

    Thx jerrm for "refining" the script! So far it's working but i have one question: the "99.adblock.fire" - file should be generated in /etc/config through adblock.sh? Currently i get a "ln: /etc/config/99.adblock.fire: No such file or directory" - error.
    Thank you for your help :)
     
  82. Goggy

    Goggy Network Guru Member

    Seems there is /opt missing in the path?
    # firewall autorun script
    fire=/etc/config/99.adblock.fire should be fire=/opt/etc/config/99.adblock.fire
     
  83. lockheed

    lockheed Reformed Router Member

    This script is awesome.
    However, I have a WRT54G v2, and apparently there is no way to write into permanent memory:
    Code:
    Filesystem Size Used Available Use% Mounted on
    /dev/root 2.6M 2.6M 0 100% /
    tmpfs 7.0M 1.2M 5.8M 18% /tmp 
    This means I have to run this entire script (all three parts) upon boot of the router. Which is why it has to be in one piece. Am I right?
     
    Last edited: Dec 1, 2013
  84. jerrm

    jerrm Network Guru Member

    No.

    I don't want the firewall autorun link surviving a reboot. There is no need for the rules if we don't run adblock. When/if adblock is run after boot it will re-establish the rules and the autorun link.
     
    Goggy likes this.
  85. jerrm

    jerrm Network Guru Member

    Does the config folder exist? In most of my scripts I create it "just in case," but missed it here. The folder exists at boot with my shibby versions, but that may vary.
     
    Goggy likes this.
  86. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    No, the folder in my shibby install does not exist so I modified the script to save the firewall to /tmp/etc/ only...
     
  87. jerrm

    jerrm Network Guru Member

    Actually there is another problem with the link, it overwrites itself if called from autorun. I modified the proc to adjust to pixelserv and didn't put things back correctly. Give me a minute.
     
  88. jerrm

    jerrm Network Guru Member

    MY VERSION OF SCRIPT UPDATED DEC-1-2013 18:43 EST
    Small but important change, please update if previously downloaded.


    Original post updated to with corrected version.
     
    Last edited: Dec 2, 2013
    Goggy likes this.
  89. jerrm

    jerrm Network Guru Member

    I checked, on Shibby 110 the config folder exists at startup, on 115 it doesn't. Not sure when it changed. We use this technique extensively, so another script has been creating the folder and I didn't notice the behavior had changed.

    The updated script posted Dec 1 creates /etc/config if it doesn't exist, along with fixing the link overwrite problem.
     
  90. Goggy

    Goggy Network Guru Member

    Good morning,

    i have another little problem: when creating the blocklist, my blacklist and whitelist - files are downloaded but not taken into account.
     
  91. JoeDirte

    JoeDirte Serious Server Member

    I tried upgrading to this latest adblock and pixelserv on Shibby 115 and I keep having issues with the /opt/etc/config/99.adblock.fire file. It's never created even with the updated script. I use /jffs/adblock/ as my prefix and it wasn't created there although it appears to be hard-coded to look in the /opt/etc folder. I ended up reverting to a previous version I backed up before attempting the upgrade.

    I never saw /etc/config get created either.
     
  92. Goggy

    Goggy Network Guru Member

    jerrm's latest version overcomes this problem - updated adblock.sh in his original post ...
     
  93. JoeDirte

    JoeDirte Serious Server Member

    That's the one I tried about 30 mins ago. I'll take another look...

    Double-checked. Same issue. I don't even have a /config folder under /etc. I'm going to stick with my older version for now since it works. :)
     
    Last edited: Dec 2, 2013
  94. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    I hear you, these are relatively simple scripts that do a whole lot. Downside is that it's not always working because of some stupid missing dir like before (it's almost impossible to cover every situation).
    I installed it tested it and it works well, but I studied the script a little to understand (at least on a minimal level) what it does.
    After that initial step it's a piece of cake. I catched the missing dir and fixed it myself.

    This is not a new script, but more of a MOD, it has some MORE configuration to do in the body of the script, contrary to the original script, it's not as easy to install...
    From this point forward there is more work to do to advance it, best thing is move all configuration to adblock.ini (or config file), but that is work that Jerrm doesn't want or need to do.
     
    Last edited: Dec 2, 2013
  95. Frequenzy

    Frequenzy Networkin' Nut Member

    just tried jermm mod script and it works. running shibby 114.
     
  96. Almaz

    Almaz Serious Server Member

    Probably a stupid question but from which ssl websites do you see advertising?

    Sent from my SGH-T999 using Tapatalk 2
     
  97. Frequenzy

    Frequenzy Networkin' Nut Member

    @jermm
    age2update=?

    can we set this to 0, to disable.
     
  98. Goggy

    Goggy Network Guru Member

    Not only for blocking ad-sites, also for blocking sites which are tracking, spying, distribute malware etc. Let's take Google Analytics as example ...
     
  99. jerrm

    jerrm Network Guru Member

    The only variable that cannot be set or overridden in the config file is the location of the file itself - $config. Remove the $config and $prefix assignments where commented and the script should fall back to haarp's original behavior.
     
    Last edited: Dec 2, 2013
  100. jerrm

    jerrm Network Guru Member

    Yes.
     

Share This Page