1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Script: Clean, Lean and Mean Adblocking

Discussion in 'Tomato Firmware' started by haarp, Apr 23, 2013.

  1. jerrm

    jerrm Network Guru Member

    Where are you saving them?
     
    Goggy likes this.
  2. jerrm

    jerrm Network Guru Member

    Don't know why it wouldn't be created with the updated script. We use the technique in literally dozens of scripts on dozens routers across multiple Tomato versions without issue.
     
  3. Goggy

    Goggy Network Guru Member

    Solved - had to wget both files with path (but it worked without with haarp's script) Oo
    wrong: wget http://my.link.com/whitelist.txt -O - > "whitelist"
    working: wget http://my.link.com/whitelist.txt -O - > /opt/adblock/whitelist

    So for me everythink working now :)
     
    Last edited: Dec 2, 2013
  4. Goggy

    Goggy Network Guru Member

    that was my problem - see my post above. I have to say that i stay with the original behavior - all files in on folder (/opt/adblock).
    As already said - its working now as expected *HAPPY* - thx for your help! :)
     
    Last edited: Dec 2, 2013
  5. JoeDirte

    JoeDirte Serious Server Member

    I don't know either.
     
  6. Almaz

    Almaz Serious Server Member

    Is anybody having any problems with Etrade.com? It seems to be it takes some time to open the page with adblocker but without AdBlocker it opens up almost instantly. DNS Client disabled and I clear all Temp files from the browser before testing. I had to put a bunch of links in whitelist to get the page open faster but it still takes a few seconds to open the page completely. I just can't figure it out if it's a router speed limitation or website scripts causing time out delays. Is it possible DnsMasq latest versions works slower compare to the older one?

    Script Blocker v4.5 with V32 pixelserve
    Router: E3000
    Shibby v115

    Here is the list I had to whitelist and I'm still playing with it just to get the page load faster.

    Code:
    etrade.122.2o7.net
    2o7.net
    122.2o7.net
    us.etrade.com
    etrade.com
    pagead2.googlesyndication.com
    ssl.google-analytics.com
    google-analytics.com
    casalemedia.com
    akamaiedge.net
    akamai.net
    www.googleadservices.com
    cm.g.doubleclick.net
    csi.gstatic.com
    gstatic.com
    js.dmtry.com
    worldnic.com
    ns42.worldnic.com
    ns41.worldnic.com
    service.maxymiser.net
    r.turn.com
    turn.com
    rfihub.com
    openx.net
    imgclck.com
     
    Last edited: Dec 2, 2013
  7. JoeDirte

    JoeDirte Serious Server Member

    I got the updated version working today. There was a post that said "Original post updated" but it links back to post # 237877 on page 4 of this thread. When I actually went back to the first post in this thread, my issue disappeared. Chalk it up to trying this last night when I was too tired to think clearly. :)

    All is well again. Thanks everyone!
     
  8. remlei

    remlei Networkin' Nut Member

    I tested @jerrm's adblock update and it seems works fine, at first it confuses me since you need to edit lots of things on this and specially that adblock.ini that im not really familiar of (yeah, later I found out its just a same config file you use in page 1, renaming it to adblock.ini)

    in jerms modified script, the things that I noticed is that the every page that I visit takes 3-5minutes finish load, something like stuck in processing googleadservices.com

    although this issue is not present in v4.5 (officially in page 1),you only just needs to update the pixelserv to v32 or later to fix the long web page load.

    also for whitelist and blacklist, it's better to have the native support from original v4.5 which you can load whitelist and blacklist file locally stored on the same directory, on @jerrm's version you need to fetch it with wget or need to grab it somewhere on the web or insert the domains in adblock.ini that make things messy.
     
  9. Almaz

    Almaz Serious Server Member

    I found a reason why Etrade.com and other sites takes a long time to load due to Googleads and DoubleClick were blocked. Can anyone tell me why would Googleads and DoubleClicks hosts would hog the loading of the page? Is it possible to keep them blocked without losing the performance.

    Sent from Tapatalk 2
     
  10. Goggy

    Goggy Network Guru Member

    @remlei / Alamaz
    For me it seem's that pixelserv isn't working and because of that your browser is running into a timeout.
    Have you checked that pixelserv is running? Open pixelserv's ip in a browser. With http there should immediately be a blank page, with https you should get an error (in FireFox you should get an "ssl_error_access_denied_alert", in IE something like site can not be displayed).

    jerrm's modificated script is working PERFECTLY so we have to look for the problem on your side ;-)
     
    Last edited: Dec 3, 2013
  11. remlei

    remlei Networkin' Nut Member

    yes, I checked it and pixelserv is running. Using @jerrm's script, As you say, pixelserv in http return a blank page but in https it throws a ssl_error_access_denied_alert. And it had the very long load time for webpage.

    the weird thing is that @haarp's script,pixelserv in http returns blank page and https returns infinite loading, the weird thing is it doesnt have the long load time for webpage problem.
     
  12. Frequenzy

    Frequenzy Networkin' Nut Member

    when using jerrm script, i noticed this on the logs when i turn off/on bw limiter or qos.

    -off
    Dec 3 20:43:12 router user.info rcheck[11501]: Activating rule 1
    Dec 3 20:43:12 router user.notice ADBLOCK: /etc/config/config not found!
    Dec 3 20:43:12 router user.notice ADBLOCK: Exiting 11
    -on
    Dec 3 20:44:01 router user.info rcheck[11609]: Activating rule 1
    Dec 3 20:44:01 router user.notice ADBLOCK: /etc/config/config not found!
    Dec 3 20:44:01 router user.notice ADBLOCK: Exiting 11

    adblock seems to continue to run though. using shibby 114.
     
  13. Almaz

    Almaz Serious Server Member


    I'm still using v4.5 and pixelserve v32. Pixelserve is working properly and I do get blank page. Any other reason why Googleads and DoubleClick slows down loading page?

    Sent from my SGH-T999 using Tapatalk 2
     
  14. jerrm

    jerrm Network Guru Member

    As stated in the script v32 is assumed, with a note what to comment out if using an earlier version.

    No you don't, they only need to exist in the $prefix folder - nothing needs to be downloaded. Read the script comments, if you want to use the executable folder then comment out where $prefix is redefined.
     
    Last edited: Dec 3, 2013
  15. jerrm

    jerrm Network Guru Member

    Good find, the firewall rules are not being run on a firewall restart if the config file is not explicitly pathed.

    Your not any worse off than haarp's original script, but I will fix it and post later.
     
  16. evala

    evala Reformed Router Member

    hello, thx for all the helpful information in the thread, BUT i reached a problem: ... edit: done everything from scratch and it works wonderfully. big tnx from a noob like me to all plp that make this adblocker possible.
     
    Last edited: Dec 3, 2013
  17. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    @Almaz: I tried Etrade.com, and it loads very fast, note I'm using jerrm's mod. Original script really had a problem (I think only on some configs) with HTTPS connections.
    You could try to install it but fallow closely the instructions from page 4, or alternatively for a quick fix just try DarkKnight's (?) redirecting fix, I attached it to one of my posts on page 4!

    @jerrm: After reading trough the last batch of comments, I really came close to understanding why you didn't wanted to release the script in the first place ;)
     
  18. Almaz

    Almaz Serious Server Member

    Thanks Mihai Olimpiu-Cristian. I just did a quick test. Put blocklist in dnsmasq directly without pixelserve and page does open quickly without any problems. Since I have been using Original V4.5 and pixelserve v32 I'll try a new script and see if it works better. Which Pixelserve do you recommend v32 or v34?
     
  19. Almaz

    Almaz Serious Server Member

    Just installed a new adblocker but I have another problem. I'm using OpenVPN on port 443. VPN must be on port 443. Is there anything I can do to fix the following problem?

    daemon.info pixelserv[1131]: /tmp/adblock/pixelserv V34 compiled: Oct 14 2013 00:55:22 from pixelserv34.c
    daemon.err pixelserv[1133]: Abort: Address already in use
     
  20. jerrm

    jerrm Network Guru Member

    I don't think you can disable or change port 443 in v32. mstombs will correct me if I'm wrong.

    Try binding OpenVPN to a single address - I think it's the local parameter in the conf file. Alternatively try using pixelserv v32 and see if the using the redirect works.
     
  21. Almaz

    Almaz Serious Server Member

    I already tried v32 and it works fine but SSL won't work. Just to let you know with your script and pixelserve v32 the problem I was having with Etrade.com is completely gone and that's awesome. Now I'll see what I can do to get adblock to work with SSL/pixelserve v34 and OpenVPN on port 443.
     
    Last edited: Dec 4, 2013
  22. jerrm

    jerrm Network Guru Member

    Getting OpenVPN to only bind and listen the needed IP is probably the "correct" answer. Try addining "local m.y.i.p" to the custom conf. There really isn't any need for OpenVPN to bind the $redirip address.
     
  23. mstombs

    mstombs Network Guru Member

    You cannot change the 443 in pixelserv v34 at runtime, but will be able to in next version (Github source), sorry its a fatal error, but you probably don't want ad requests directed to your VPN link so good to fix that anyway. Tomato webserver only listens on the single lan IP address which means same standard ports can be used by pixelserv on secondary IP address also belonging to router - this was still a problem in dd-wrt when I last looked.

    There may in future be need to handle different blocked hosts differently - for some may be better to REJECT connections at iptables level, rather than refuse the ssl connection with a simple code from pixelserv. Extended page loads are caused by web pages making repeated requests and waiting for responses, I've already seen pages that detect adblocking in use, and give you option to unblock or not to proceed, fair enough!
     
  24. Almaz

    Almaz Serious Server Member

    I'm using OpenVPN in TAP mode and I can't bind it to another IP. I don't know yet but is it possible to redirect OpenVPN from 443 to another local port using --lport? I guess I would need to play with Iptables to get it to work if even possible.
     
  25. mstombs

    mstombs Network Guru Member

    I don't know about OpenVPN, but you can use the DNAT target in the nat PREROUTING chain to divert specific ip/port combos to others, there are already examples in the scripts in this thread or here:-

    http://www.linksysinfo.org/index.ph...run-on-router-wrt54g.30509/page-3#post-229597

    Note that dnsmasq or hostfile adblocking works by re-directing host IPs via dns poisoning, web page scripts can then attempt to access services on any port on that IP address, ports 80/443 are just the most common.
     
  26. jerrm

    jerrm Network Guru Member

    MY VERSION OF SCRIPT UPDATED 2013-12-04 07:44 EST
    Important change when config is located in script folder.
    See changelog above.

    Please update if previously downloaded.

    My original post updated with current version and changelog.
     
    Mihai Olimpiu-Cristian and Goggy like this.
  27. zygmunt.78

    zygmunt.78 Reformed Router Member

    Hi All, first of all thx for this wonderful script - now I don't need to use adblock addons in browsers.
    I've noticed that in some blocklist you can get a line with " so to remove that you should add
    sed -i -e 's/"/ /g' "$listprefix/blocklist" to adblock.sh script a just before elog "Blocklist generated"

    cheers
     
    Last edited: Dec 5, 2013
  28. techlifeweb

    techlifeweb Reformed Router Member

    I'm using jerrm's latest script and testing the whitelist. The whitelist downloads as it should but it doesn't work right.
    Here is my test whitelist:
    atdmt.com
    googlesyndication.com
    doubleclick.net
    hulu.com

    Only the last entry gets applied. If I change the order, it doesn't matter, only 1 entry is applied (whitelisted) and it is the last one in the list I create. Has anyone else seen this?
     
  29. jerrm

    jerrm Network Guru Member

    First guess is the file has CRLF line endings.
     
    Last edited: Dec 7, 2013
  30. Kye-U

    Kye-U Addicted to LI Member

    This is running great for me, thank you!

    Note: I had to remove the "relocate config and list files to /opt/etc" section. FYI: /opt/etc does not exist in Shibby 1.28.0000 MIPSR2-112 K26 Max.

    I also added a "refresh" parameter to stop adblocking (e.g. kill pixelserv), and force an update of all lists. I'm using this to run every Sunday to make sure everything's running fine (as for some reason, pixelserv "dies" on me occasionally. Like when I visit doubleclick.com, it doesn't return a blank page, it will try to load but eventually time out)

    Code:
       "refresh")
         stop
         force="1"
         ;;
    
     
    Last edited: Dec 7, 2013
  31. lamp10

    lamp10 Reformed Router Member

    Thanks Haarp and Jerrm for all of your work. I think I have Jerrm's version up and running... I also removed the "relocate config and list files to /opt/etc" section.

    I see where some ads are removed, but I'm still getting ads on Google. I assume this is a matter of not having the required domains blocked. I'm using the following lists:

    http://winhelp2002.mvps.org/hosts.txt
    http://www.malwaredomainlist.com/hostslist/hosts.txt
    http://adaway.sufficientlysecure.org/hosts.txt
    http://hosts-file.net/download/hosts.txt
    http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext

    Is there better list I need to be using? My knowledge on this topic is limited and I do appreciate the help.
     
  32. The Master

    The Master LI Guru Member

    Hello i got this error message. What does this mean? I have a N16 Router.

    Dec 9 16:30:35 N16 daemon.crit dnsmasq[24964]: cannot fork into background: Cannot allocate memory
    Dec 9 16:30:35 N16 daemon.crit dnsmasq[24964]: FAILED to start up
    Dec 9 16:30:35 N16 user.debug init[1]: dnsmasq terminated unexpectedly, restarting.
     
  33. Almaz

    Almaz Serious Server Member

    I'm guessing it could be hosts file contains illegal characters. Just try a single hosts file and see if it works or out of space/ram
     
  34. reimer

    reimer Networkin' Nut Member

    Also would like to say thanks to Haarp and Jerrm for this script.

    As someone who has absolutely no programming knowledge though, I was a little confused with installing the new updated script by jerrm. With haarps script, I installed it to cifs1. So when jerrms scripts was defaulted to installing to /opt/etc, I wasn't sure how to get it installed properly. I'm using a shibby build.

    Perhaps someone can point out whether I did it properly or not. Maybe for anyone else that was confused, this might help them too.



    First, I installed pixelserv v34 using this and adjusted prefix to /cifs1/adblock
    http://pastebin.com/raw.php?i=ziiG4zcm

    Second, I installed the config using the original method in the OP. Also adjusted prefix to /cifs1/adblock

    Third, I copied jerrms script to notepad, edited it by removing the "relocate config and list files to /opt/etc" section, and added "binprefix="/cifs1/adblock/" since prefix was set to $binprefix. (clearly I don't know much about programming as you can tell)

    I then saved it as "adblock.sh" and placed it in /cifs1/adblock and then executed it in Tomato.


    This comes up "ADBLOCK: Running as /tmp/.wxlxtIqv", the hosts files are downloaded and a blocklist is generated. Ads are being blocked. It seems to have worked.

    But once I try executing "/cifs1/adblock/adblock.sh" again, this comes up
    /tmp/.wxGjOazy: line 5: /cifs1/adblock/adblock.sh: not found
     
  35. jerrm

    jerrm Network Guru Member

    That's actually by design. I wrestled with automatically searching the script folder, /opt/etc, /jffs/etc, /cifs1/etc, cifs2/etc and /etc for the config file. I have several scripts that do so, but there is no way I could leave the default name "config" if I did so, and folks would still have to edit the script for full compatibility.

    I decided to force folks to make a choice.

    About to post what will probably be my final update, I may rethink it.
     
  36. The Master

    The Master LI Guru Member


    Found the "BUG"!!

    Code:
    ## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list):
    #SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt"
    #SOURCES="$SOURCES http://hosts-file.net/hphosts-partial.txt"
    This two files make the Problems. :(
     
  37. jerrm

    jerrm Network Guru Member

    Shouldn't have been necessary, but shouldn't hurt either. $binprefix defaults to the folder where the script is located, so if the script is located in /cifs1/adblock, $binprefix should reflect that.

    Are you sure you actually saved the file? With that temp file name, it looks like you executed the script itself from GUI.

    Sorry, but I don't have the patience to re-create haarp's orginal cut and paste install methods.
     
    Last edited: Dec 9, 2013
  38. boogarn

    boogarn Reformed Router Member

    I didn't realize it wasn't meant to used this way, and also used your script with haarp's install method (modified prefix stuff and executed through the GUI). Lo and behold, everything seems to be working fine other than the whitelist sites from the config. Is there a problem to be expected that this install method will cause if used as-is, or should it be okay to use it like this?
     
  39. Goggy

    Goggy Network Guru Member

    The lists themselves are OK, the RT-N16 seems to have insufficient RAM for such a huge blocklist. I have also experienced this behavior on the N16.
    On my RT-AC66U everything works even if the above lists are included :) Massive blocklist with a little impact in dns-resolution:
    Code:
    ADBLOCK: Downloaded
    ADBLOCK: Generating /opt/adblock/blocklist
    ADBLOCK: Blocklist generated
    ADBLOCK: 432973 unique hosts to block
    ADBLOCK: Done, restarting dnsmasq
    
     
  40. jerrm

    jerrm Network Guru Member

    If you have an adblock.sh file that exists on the filesystem and you can execute, you should be OK.

    What confuses me with reimer's post is the "ADBLOCK: Running as /tmp/.wxlxtIqv" response. The only way I know how that could come about is if the entire text of the main script was pasted into the GUI System Command text area, and execute was pressed.

    The fact he also has "/tmp/.wxGjOazy: line 5: /cifs1/adblock/adblock.sh: not found" afterwards implies the script is not saved to the drive.

    A fix for whitelist sites from the config will be posted soon. It's a dumb bug I introduced. Work around for now is to create a whitelist file, even an empty one will do.
     
    Last edited: Dec 9, 2013
  41. techlifeweb

    techlifeweb Reformed Router Member

    Yep, that was it. Thanks.
     
  42. mstombs

    mstombs Network Guru Member

    This does happen if you make web gui config changes that redefine the lan bridge associated with the listening ip address. Only solution I know is to kill and restart pixelserv - but same web gui change probably also restarts dnsmasq?
     
  43. jerrm

    jerrm Network Guru Member

    MY VERSION OF SCRIPT UPDATED

    My original post updated with current version and changelog.

    SCRIPT UPDATED 2013-12-11 17:57 EST
    Please update if previously downloaded.


    Important whitelist processing bug fix.

    Read firewall rule notes if using VLANS or VPN.

    This finally gets to a single version I can personally run and post here. Barring true bug fixes, it will likely be my last update. This version searches for the config file.Should now be a no-edit replacement for haarp's version in most installs, but also supports custom locations.

    As noted in log, many changes - consider 2013-12-11 "beta" at this point, will keep 2013-12-04 zip file posted for now.
     
    Last edited: Dec 12, 2013
    Goggy and Kye-U like this.
  44. Bird333

    Bird333 Network Guru Member

    I was reading your change log and it looks awesome. I'm sure I'll get around to trying it soon. Can you tell me what rules get created with 'STRICT' and 'LOOSE'?
     
  45. jerrm

    jerrm Network Guru Member

    Strict adds "iptables -A $chain -d $redirip -j DROP" at the end of the adblock chain.
     
  46. Kye-U

    Kye-U Addicted to LI Member

    This may be a noob question, I read through your comments in the script as well, but if I connect through VPN, how would I be able to have doubleclick.com return a blank page? (e.g. blocked and served an empty text file by pixelserv)

    Right now, if I connect my Android phone to my wifi directly, doubleclick.com is blocked, but if I disconnect from WiFi, use my mobile data and connect through VPN, it isn't blocked, even if I run the "fire" command after connecting through VPN (if I understand it correctly)

    Thanks!
     
    Last edited: Dec 12, 2013
  47. reimer

    reimer Networkin' Nut Member

    Thank you jerrm. The new script is now much easier to install and works perfectly!
     
  48. jerrm

    jerrm Network Guru Member

    The client would have to be using the router as it's DNS server.

    If the client is not using the router's dnsmasq, it will resolve the true IP for the ad hosts - the firewall settings are not a factor.

    If the client is using the router's dnsmasq, then ad hosts would resolve to the pixelserv IP and the ads would never load. The firewall rules would only impact how efficiently things happen and how well protected the pixelserv IP is from the vpn.

    If the client is directing all traffic through the VPN, it could still be using non-router dns server addresses. Tomato's "intercept dns port" GUI option box does not intercept VPN connections, but you could craft a rule to do so.
     
    Kye-U likes this.
  49. JAC70

    JAC70 Networkin' Nut Member

    Thanks jerrm, your updates seem to have resolved the issue with Google's tracking links needing two back presses. It also seems to have fixed my Whitelist issues, just have to keep adding parasite referral domains in each time I discover one I need.
     
    Last edited: Dec 15, 2013
  50. ShinichiYao

    ShinichiYao Reformed Router Member

    Can I remove all SOURCES HOSTS online and use only blacklist?
     
  51. rootMBX

    rootMBX Serious Server Member

    Coming from ALL-U-NEED and went through installing this... Awesome script! VERY fast!

    A few caveats to be aware of when installing.

    1. In the following code snippet from the example config file in post 1:
    Code:
    #SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
    ## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list):
    #SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt, http://hosts-file.net/hphosts-partial.asp"
    a. Change the extensions from .asp to .txt. Wget complains.
    b. The comma between the two hyperlinks causes parsing errors. Remove it and leave a space between the two hyperlinks.

    2. The following list's domain is actually BLOCKED!... BY ANOTHER LIST (or even possibly by itself...)! LOL!
    Code:
    ## Hosts File Project (~3M!):
    #SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    
    Add "mine.nu" to your whitelist BEFORE your first run and all should be well. If you've already run the script, do an "adblock.sh clean" and try again with your modified whitelist.

    I originally tried to add the full domain name (hostsfile.mine.nu) and it was still being directed to pixelserv's IP. Seems just the root domain (mine.nu) was being blocked. Whitelist file worked perfectly.

    3. On a RT-N66U. Takes about 180 seconds to compile the blocklist. You'll have about 422k unique blocked domains using all the lists from post 1. Here are my JFFS stats after: TOTAL/FREE -> 22.25 MB / 12.78 MB

    4. It also seems that after enabling all of the blocklists, dropbox.com of all things doesn't work. The actual dropbox.com domains resolves, but something that facilitates it's SSL (https) is blocked. Tracert fails, but nslookup returns IP. I disabled this list (From Number 2):
    Code:
    ## Hosts File Project (~3M!):
    #SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    
    And dropbox.com started working again. This really is a troublesome block list.

    EDIT: http://hosts-file.net/download/hosts.txt <- Blocks "dropbox.com" and a few variants. I didn't find "dropbox.com" in the "Hosts File Project" so maybe this list was the only culprit. Either way, adding "dropbox.com" to whitelist fixed the issue.

    Questions:
    1. Should I expect my JFFS flash to be working after 5-10 years of running this script once a week?
    2. Does anyone have a "more updated/modern" group of block list hyperlinks? Or should I continue to use the list in the config file from the first post?

    Many thanks to the two authors of this amazing script for their time and to all those who have contributed!
     
    Last edited: Dec 17, 2013
  52. jerrm

    jerrm Network Guru Member

    The source naming and mine.nu issues have been reported before.

    Needing to whitelist the base domain ("mine.nu") is due to the way the address directive works - see my next update.

    I haven't seen the dropbox issue, but have long had the mine.nu list commented out.

    I think jffs should be fine at once a week, but the next update will reduce jffs/usb writes.

    I've been satisfied with the group of lists I've been using, so haven't gone looking.
     
    Goggy likes this.
  53. rootMBX

    rootMBX Serious Server Member

    Great! Thanks for the info. Less writes using more RAM? Or overall?

    Also, I've come across slow loading with some sites (even using pixelserv v34, which was supposed to solve this problem as some in the thread have said) such as speedtest.net. Lag disappeared when script was disabled.

    Is there any way that I can get rid of this lag without going through and finding out which domains need to be whitelisted?

    EDIT: NVM. Turns out moving the script and all of it's writes to /tmp/adblock (RAM) solved the lag with speedtest.net. It downloads it's modified self, config, and whitelist from pastebin, installs, and runs (cron) on any boot. I don't reboot often so it's a non-issue.

    Compiling the blocklist went from 180 seconds -> 101 seconds using the same number of lists.

    RAM Disk Usage:
    Total Used Available Use%
    Before: 124.9M 17.0M 107.8M 14% /tmp
    Was something like 80-85% free before
    All But 2 Lists:
    After: 124.9M 57.5M 67.3M 46% /tmp
    Total / Free Memory 249.73 MB / 178.11 MB (71.32%)
    All Lists:
    After: 124.9M 61.8M 63.0M 50% /tmp
    Total / Free Memory249.73 MB / 170.45 MB (68.25%)


    Thanks again!
     
    Last edited: Dec 17, 2013
  54. defiant

    defiant Reformed Router Member

    I'm getting google text ads showing too even though the following are in the blocklist:

    root@router:/jffs/adblock# cat blocklist | grep google
    address=/1.googlenews.xorg.pl/192.168.1.254
    address=/2.googlenews.xorg.pl/192.168.1.254
    address=/3.googlenews.xorg.pl/192.168.1.254
    address=/3906523995308773357-a-1802744773732722657-s-sites.googlegroups.com/192.168.1.254
    address=/4.afs.googleadservices.com/192.168.1.254
    address=/4.googlenews.xorg.pl/192.168.1.254
    address=/5.googlenews.xorg.pl/192.168.1.254
    address=/analytics-api-samples.googlecode.com/192.168.1.254
    address=/commondatastorage.googleapis.com/192.168.1.254
    address=/domains.googlesyndication.com/192.168.1.254
    address=/google-analytics.com/192.168.1.254
    address=/google.tucows.com/192.168.1.254
    address=/googleads.g.doubleclick.net/192.168.1.254
    address=/googleads2.g.doubleclick.net/192.168.1.254
    address=/googleadservices.com/192.168.1.254
    address=/googleapi.buzzwordll.biz/192.168.1.254
    address=/googlenews.xorg.pl/192.168.1.254
    address=/googlesyndication.com/192.168.1.254
    address=/pagead.l.google.com/192.168.1.254
    address=/pagead2.googleadservices.com/192.168.1.254
    address=/pagead2.googlesyndication.com/192.168.1.254
    address=/partner.googleadservices.com/192.168.1.254
    address=/partnerad.l.google.com/192.168.1.254
    address=/ssl.google-analytics.com/192.168.1.254
    address=/tpc.googlesyndication.com/192.168.1.254
    address=/video-stats.video.google.com/192.168.1.254
    address=/wintricksbanner.googlepages.com/192.168.1.254
    address=/www-google-analytics.l.google.com/192.168.1.254
    address=/www.e-googles.com/192.168.1.254
    address=/www.google-analytics.com/192.168.1.254
    address=/www.googleadservices.com/192.168.1.254
    address=/www.googlechrome2013.com/192.168.1.254

    Also, I guess it's not possible to block facebook sponsored items in the newsfeed? Might revert back to adblock
     
  55. mikester

    mikester Network Guru Member

    Good work! Worked perfectly on Asus RT-N66U...browsing is a lot faster now
     
  56. jerrm

    jerrm Network Guru Member

    MY VERSION OF SCRIPT UPDATED

    My original post updated with current version and changelog.

    SCRIPT UPDATED 2013-12-22 23:00 EST
    Please update if previously downloaded.


    Two "new" list generation methods to improve dnsmasq memory usage - thanks to srouquette
    Reduce jffs/usb writes if tmpfs is available
    Don't overwrite prior downloaded file if wget fails and $prefix space is available
    A few more tweaks
    See the notes.
     
    QSxx and Goggy like this.
  57. crandy2

    crandy2 LI Guru Member

    Thanks for this. I had been running your previous version without problems, this one seems to run fine as well. I only block malware domains, etc. So only about 2400 domains. My RT-16 has no lag at all.

    Happy Holidays to you, and thanks for the great work!


    Sent from my A500 using Tapatalk 4
     
  58. philipkim

    philipkim Reformed Router Member

  59. ShinichiYao

    ShinichiYao Reformed Router Member

    That's perfect meet my need, thanks!
     
  60. Goggy

    Goggy Network Guru Member


    Thank you for the update
    ! Just want to say that i have to chmod the blocklist-file to at least 644, otherwise i get a permission denied when dnsmasq wants to load the file (through zzz.adblock.hosts). I am using HOSTS-mode, a performance-increase for me ...
     
  61. jerrm

    jerrm Network Guru Member

    B
    Because the hosts file is read after dnsmasq has dropped root privileges. Where are the files stored? Have you changed the default umask? I can add a chmod, but would like to understand better.
     
  62. Goggy

    Goggy Network Guru Member

    Hi!
    Default umask i haven't changed. All the files are stored in /opt/adblock (=mounted usb-stick). For now i have inserted a "chmod 644 "$blocklist" between "rm -f "$tmpblocklist" &>/dev/null" and "trap - SIGQUIT SIGINT SIGTERM SIGHUP" in the confgen section.
     
  63. jerrm

    jerrm Network Guru Member

    What tomato version? I think 644 is the default mask for any version I've used. What mask is the file created with? What mask do you get if you "touch /opt/adblock/masktest" from the command line?
     
  64. Goggy

    Goggy Network Guru Member

    I am using Victec's latest build v1.2x on an AC66U. The above "touch" result in a file masked 0644. The files created within the script are masked 0600:

    [​IMG]

    Hmm ... is there anything i can make wrong? :(
     
  65. koitsu

    koitsu Network Guru Member

    Just a terminology complaint: the "mask" is not 0644. For a file that defaults to 0644 perms, the mask (umask) would be 0022.

    You should not have to add silly chmod commands within the script. Instead, simply adding a line at the top (or at relevant place) that says umask 0022 should be sufficient.

    I am not surprised, by the way, that different users with different setups have different default umasks. It's well-known that Optware and some other stuff messes with dotfiles and sets up a shell environment that has a completely different set of parameters/environmental parameters than a stock Tomato/Busybox shell. However, daemons launched from via Startup scripts/etc. may have a different umask than via CLI. So a call to umask 0022 within said script would be a wise choice.
     
  66. Goggy

    Goggy Network Guru Member

    @koitsu: that seems to adress the problem! I have entware installed (even if i do not use it). Added umask 0022 at the beginning of adblock.sh as suggested and every created file has 644 as permission. :)
     
  67. Goggy

    Goggy Network Guru Member

    Please allow me at last another dumb question :confused:
    Whats the difference between
    Code:
    [[ -x /opt/adblock/adblock.sh ]] && /opt/adblock/adblock.sh &
    and a simple
    Code:
    /opt/adblock/adblock.sh
    Thx!
     
  68. jerrm

    jerrm Network Guru Member

    The first checks to see if the script exists and is executable and if so launches it as a separate process and the calling shell does not wait for it to complete.

    The second will execute the script and wait for completion before proceeding to the next command.
     
    AndreDVJ and Goggy like this.
  69. jerrm

    jerrm Network Guru Member

    WIth entware, are you running bash or other shell?

    I'll load up a victek build and check it out this weekend. Where are you calling adblock.sh from - init, wanup, an autorun script, etc?

    I don't see any harm in adding an umask statement to adblock.sh, but want to recreate/test before posting anything.

    Actually, you probably don't really need to modify the adblock script, and could add the umask statement to whatever script is calling adblock.sh. An option anyway in the interim for anyone who doesn't want to tweak adblock directly. It could also just be added to the config file in most instances.
     
    Goggy likes this.
  70. koitsu

    koitsu Network Guru Member

    No, adding a umask statement to whatever script calls adblock.sh will not necessarily fix the problem.

    Sorry to sound short and terse but I really don't feel like explaining how fork+exec vs. system() plays into this, combined with how certain shells act when spawning a sub-shell (many people don't understand the latter, because it varies per shell and environment)... :/ Just not in a good mood right now and I always hate going over UNIX details on a forum about routers, haha. :)

    It's generally best to just stick a umask statement at the top of the script which does any kind of file creation.
     
    Goggy likes this.
  71. Goggy

    Goggy Network Guru Member

    Today i updated the script on two RT-N16. Both of them running Victec's v 1.2v - firmware, usb-stick is mounted under /opt. None of them has optware / entware installed. The setup is almost identically, the only difference is that one has static wan, the other dials in via pptp - nothing that has something to do with the file system.
    And now that what confuses me: on one the files are created with 0600, on the other one they are created with 0644 :confused:
    I am running adblock.sh on every router via wanup-script:
    Code:
    [[ -x /opt/adblock/adblock.sh ]] && /opt/adblock/adblock.sh &
    "umask 0022" inserted at the beginning of the script and also the "0600-rtn16" works as expected ...
     
  72. philipkim

    philipkim Reformed Router Member

  73. jerrm

    jerrm Network Guru Member

    Not sure what you're saying here. Are you saying the files are 0600 by default on the pptp unit, but are OK with "umask 0022" added, or you get 0600 files even if with umask 0022 in the script (which I would find very odd)?

    What type of wan connection was on the AC66u?
     
  74. Goggy

    Goggy Network Guru Member

    I will try to make it more clear (sorry for my poor english):

    adblock.sh without umask 0022:
    • First RT-N16: Victec v1.2v, static wan, no opt-/entware, stick mounted under opt, 0644 files by default
    • Second RT-N16: Victec v1.2v, pptp wan, no opt-/entware, stick mounted under opt, 0600 files by default
    • AC66U: Victec v1.2x, pppoe wan, entware installed but not used, stick mounted under opt, 0600 files by default

    adblock.sh with umask 0022:
    0644 permission by default on every router / every file. Just as it should be. No permission denied - problems ...

    Hope it's more clear now. Me wonders that the RT-N16's behave differently ...
     
  75. jerrm

    jerrm Network Guru Member

    No apologies necessary. Your much better with english than I could be with another language.

    Odds are the ppp code is changing umask. A very quick grep shows some instances, but I haven't really looked and see if wanup would be impacted. If it's just a ppp problem, then I'll just add the umask to the next update. If Victek had intentionally changed the default trying to tighten things up, I'd might try to honor that and only chmod the blocklist file. I'll load up Victek again this weekend to test anyway.
     
  76. jerrm

    jerrm Network Guru Member

    Basically the same concept, but pulls multiple source lists and combines them. OpenWRT equivalents are almost certainly out there somewhere.

    Script is only targeted for recent tomato, and I personally have no interest in expanding to OpenWRT at the moment. However, it shouldn't require too much work for OpenWRT.

    I'm not so familiar with cerowrt, but if it follows openwrt behavior for dnsmaq I'd start with:
    • set CONF=/etc/dnsmasq.conf for legacy and optimize modes
    • change hostlink to point to a file that will be seen by openwrt's dnsmaq - probably need to add an "option addnhosts" to openwrt's config, or create a dnsmasq.conf with the directive.
    • replace the "service dnsmasq restart" lines with the openwrt init.d equivalent.
    No doubt many other errors that will also need fixing.
     
  77. Scottmsu

    Scottmsu Reformed Router Member

    Hello guys, I'm new to tomatousb but I'm very interested in learning how to configure it. I would like to add this adblock feature but i'm stuck at one of the steps.

    -> I recommend other storage methods however, as JFFS is very limited in size (depends on which filter lists you'll ultimately use of course)
    • Designate a directory on your storage for adblock, e.g. /jffs/adblock/ (as seen by the router). Avoid spaces! This is the PREFIX.
    • Install pixelserv if desired (thread). Take the entire chunk of script in this link, adjust PREFIX at the top, paste it into the box on Tools->System and press Execute. This is also how you can update pixelserv in the future.
    How do I designate a directory in the tomatousb webclient? I decided to use the jffs storage option. Adjusting the prefix means to replace it with the designated storage directory right?

    Thanks!

    edit: i'm on tomatousb by shibby 1.28 cisco e3200
    edit2: I figured it out! I think it's working now. How do i disable adblock if I want to in the future?
     
    Last edited: Dec 28, 2013
  78. The Master

    The Master LI Guru Member

    ......

    i quote from the SCRIPT README:


    ## Options:
    ## 'force': force updating sources,
    ## 'stop': disable Adblock, 'toggle': quickly toggle Adblock on and off
    ## 'restart': restart Adblock (e.g. for config changes)


    My Example: /cifs2/adblock/adblock.sh toggle -> on/off
     
  79. Mojonba

    Mojonba Network Guru Member

    Guys,

    I am having problems with the blacklist. I have added yahoo.com to the config file as well as a blacklist file and it is not blocking the site. Any ideas?
     
  80. Mihai Olimpiu-Cristian

    Mihai Olimpiu-Cristian Serious Server Member

    Add a line at the end of the file..., it worked for me, it must be something in the parser... (at least in some older version)
     
  81. Mojonba

    Mojonba Network Guru Member

    At the end of the config file you mean?, what about the separate blacklist\whitelist files, any way of getting those to work? Thx
     
  82. Goggy

    Goggy Network Guru Member

    Separate blacklist / whitelist files working for sure. Adapt the config file to suit your needs - example:
    Code:
    ### Blacklist and Whitelist files (optional) ###
    wget http://URL-TO-YOUR/whitelist.txt -O - > /opt/adblock/whitelist
    wget http://URL-TO-YOUR/blacklist.txt -O - > /opt/adblock/blacklist
     
  83. Mojonba

    Mojonba Network Guru Member

    Goggy,

    Thanks for the info. If the files were stored locally in the jffs would the code be like this?

    Code:
    ### Blacklist and Whitelist files (optional) ###
    /jffs/adblock/whitelist
    /jffs/adblock/blacklist
     
  84. darkknight93

    darkknight93 Networkin' Nut Member

    jondlou likes this.
  85. gnoj123

    gnoj123 Reformed Router Member

    Thanks you for this script. Can you help me why is download failing, as I'm new to this.

    This is what I get in my log:

    Code:
    Dec 30 20:43:24 Tomato user.notice ADBLOCK: Running as /opt/op/adblock.sh
    Dec 30 20:43:25 Tomato user.notice ADBLOCK: Using config file /opt/op/config
    Dec 30 20:43:25 Tomato user.notice ADBLOCK: Ignoring extra config file /opt/adblock/adblock.ini
    Dec 30 20:43:25 Tomato user.notice ADBLOCK: Requested list mode is OPTIMIZE
    Dec 30 20:43:26 Tomato user.notice ADBLOCK: Config or script has changed - rebuilding list
    Dec 30 20:43:26 Tomato user.notice ADBLOCK: Download starting
    Dec 30 20:43:27 Tomato user.notice ADBLOCK: Failed: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
    Dec 30 20:43:28 Tomato user.notice ADBLOCK: Failed: http://winhelp2002.mvps.org/hosts.txt
    Dec 30 20:43:28 Tomato user.notice ADBLOCK: Failed: http://www.malwaredomainlist.com/hostslist/hosts.txt
    Dec 30 20:43:28 Tomato user.notice ADBLOCK: No source files found
    Dec 30 20:43:28 Tomato user.notice ADBLOCK: Exiting /opt/op/adblock.sh 3
     
  86. Mojonba

    Mojonba Network Guru Member

    The code above is not working. How can I get the files working but stored locally in the jffs partition?

    Thanks

    Update: I updated the script to jerrm's 2013-12-22 version and whitelist blacklist files are working perfectly.
     
    Last edited: Jan 1, 2014
  87. remlei

    remlei Networkin' Nut Member

    updated to jerm's latest mod script and it randomly throws this error

    Code:
    Jan  3 06:57:35 unknown daemon.crit dnsmasq[4032]: cannot read /opt/adblock/blocklist: No such file or directory
    Jan  3 06:57:35 unknown daemon.crit dnsmasq[4032]: FAILED to start up
    Jan  3 06:57:35 unknown user.debug init[1]: dnsmasq terminated unexpectedly, restarting.
    it seems that the script doesnt check if blocklist file is exist, specially if the host files were already there., it just checks the host files if its latest or not, if it's latest then it just skips generating blocklist file. Although I didint delete blocklist file, it seems that the script still deletes randomly.
     
  88. jerrm

    jerrm Network Guru Member

    If the blocklist is missing, it should force a rebuild. The only issue I know of is with the specific combination of HOST mode, a ppp wan connection, and adblock called from wanup. Next update will fix that.

    Update to the latest, post the script output of "adblock.sh debug" and your config file.
     
  89. Ingo Pan

    Ingo Pan Reformed Router Member

    Hi,

    I just installed your script and somehow i must be doing a big mistake. It´s not blocking ads at all.
    I created the 2 files adblock.sh & config on the USB space located at /tmp/mnt/MYLINUXLIVE/adblock/ .
    I chmod the script to 777 and ran it with sh adblock.sh and got the following:
    root@tingo:/tmp/mnt/MYLINUXLIVE/adblock# adblock.sh
    ADBLOCK: Download starting
    Connecting to pgl.yoyo.org (95.172.9.82:80)
    Connecting to www.malwaredomainlist.com (143.215.130.61:80)
    - 100% |********************************************************************************************************************| 67288 0:00:00 ETA
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    - 100% |********************************************************************************************************************| 57262 0:00:00 ETA
    - 100% |********************************************************************************************************************| 518k 0:00:00 ETA
    ADBLOCK: Downloaded
    ADBLOCK: Generating /tmp/mnt/MYLINUXLIVE/adblock/blocklist
    ADBLOCK: Config generated, 17657 unique hosts to block
    ADBLOCK: Done, restarting dnsmasq

    Done.
    ADBLOCK: Exiting


    I also tried copy/paste the script/config, which gives the same results.



    What i am doing wrong ? Thanks in advance.
     
    Last edited: Jan 4, 2014
  90. Goggy

    Goggy Network Guru Member

    You are using your router as (the only) dns-server? What about pixelserv?
     
  91. Ingo Pan

    Ingo Pan Reformed Router Member

    Pixelserv i set to "0" in the config file. About the DNS i am not sure. I am using fixed ones in my PC´s (OpenDNS) that also have fixed IP´s. I also my set own DNS Servers in the router. Thanks for the quick reply.
    I am also using a VLAN, which servers as guest wireless network.
     
  92. Goggy

    Goggy Network Guru Member

    You HAVE TO use the Router as your only DNS-Server. You can use the OpenDNS - DNS-Server in your Router as static DNS-Server.
    You SHOULD use PixelServ. Only then the Ads are exchanged through a transparent Pixel and you will have no Issues with slow loading of Sites etc.
    I have the Impression that you do not really understand how this Thing works ;)
     
  93. Goggy

    Goggy Network Guru Member

    @jerrm:
    Is it possible that setting up the cron-job in the script doesn't really have an effect?
    Code:
    # default cron schedule standard cru format
    schedule="30 5 * * *"
    cronid=adblock.update
     
  94. Kye-U

    Kye-U Addicted to LI Member

    You have to run it with the cron parameter: e.g. /jffs/adblock/adblock.sh cron

    To remove the cron-job, use the stop parameter.
     
    Goggy likes this.
  95. Ravi Baskaran

    Ravi Baskaran Reformed Router Member

    I am using the 12-22 build provided by jerrm. The script works brilliantly. Thanks for your hard work :).

    I have a whitelist file with about 30-40 domains in them. Does the script support updating of the whitelist files dynamically aka. Is it possible for me add new entries into this whitelist and get the script to update the whitelist files alone? or will have to run the adblock.sh all over again?
     
  96. Ingo Pan

    Ingo Pan Reformed Router Member


    So i set all NIC parameters to DHCP for my Network Cards ? Then it uses the Router as DNS, correct ?
    @edit: Got it working. As i am from germany an no native speaker, i translated the part saying "You have to use the router as your ONLY Dns server" wrong. I just set the Router´s internal IP as DNS Server and left the 2nd DNS router blank.

    As i am from germany, i´d like to use the easylist germany blocklist. Is there someone that could give me instructions of how to get the aprop. list into the router (so that the format is compatible). The static URL is https://easylist-downloads.adblockplus.org/easylistgermany.txt

    Thanks in advance and for your efforts, Ingo
     
    Last edited: Jan 5, 2014
  97. jerrm

    jerrm Network Guru Member

    Have to run the script again, but you could craft a script to remove new whitelist entries from the generated blocklist.
     
  98. jerrm

    jerrm Network Guru Member

    Not really possible. This dns method of blocking can rely only on host names. The list contains full and partial urls and even script snippets. One section appears to be just hosts/domains, you might be able to strip that portion out and use it.
     
  99. jerrm

    jerrm Network Guru Member

    MY VERSION OF SCRIPT UPDATED
    My original post updated with current version and changelog.

    SCRIPT UPDATED 2014-01-05 14:44 ESTPlease update if previously downloaded.

    Important fix for HOST mode and ppp WAN connections. File permissions were not correctly set.
    Honors gui log settings.
    Better handling of no-reboot shutdown/init sequences.

    From 2013-12-22:
    Two "new" list generation methods to improve dnsmasq memory usage - thanks to srouquette
    Reduce jffs/usb writes if tmpfs is available
    Don't overwrite prior downloaded file if wget fails and $prefix space is available
    A few more tweaks
    See the notes.
     
    rotorbudd, iommi and Goggy like this.
  100. Bird333

    Bird333 Network Guru Member

    I am finally ready to try jerrm's version of the script. My files are in /opt/downloads/adblock/. Do I just need to replace haarp's script with yours? Do I need to modify my config file at all?
     

Share This Page