Script: Adblock - not so lean

Discussion in 'Tomato Firmware' started by jerrm, Mar 13, 2016.

  1. jerrm

    jerrm Network Guru Member

    THIS SCRIPT IS INTENDED FOR A MODERN TOMATO BUILD, FROM 2013 OR LATER. It makes use of multiple Tomato specific features. No real attempt has been made to be compatible with any other platforms.

    This is a mod of @Haarp's "Clean, Lean and Mean Adblocking." It's not so lean anymore, but the core functionality is still @Haarp's.

    The script is written to run on "stock" Tomato. Third party packages like Optware or Entware are NOT required.

    The script, release notes, and pixelserv can also be found at http://tomato-adblock.weebly.com.

    Web Interface
    menu.png UI.png
    The script supports and installs a web interface by default, it features:
    • start/stop adblock
    • display adblock status and stats
    • display pixelserv status and stats
    • click to add host(s) to whitelist/blacklist
    • edit adblock config
    • edit adblock whitelist/blacklist files
    The script runs under Tomato's standard admin http server and should be compatible with any recent build. An adblock link is added to the Tomato interface. The url for adblock is also output to the console and syslog. The url will usually be http://xxx.xxx.xxx.xxx/user/adbock.sh, inserting the appropriate router IP address.

    The pixelserv status function requires @HunterZ 's pixelserv v35.HZ8 or newer, otherwise an error message will be displayed.

    Storage
    The adblock scripts and install script assume there is persistent storage available, in the form of USB, JFFS, or CIFS. This version makes no attempt to live inside nvram storage constraints. For older units without persistent storage, some run the script downloading dynamically at run time into the tmp folder.

    Pixelserv
    Pixelserv is recommended, but not required. It will provide an improved user experience(fewer browser error messages), but ads will be blocked with or without it. Usage of pixelserv can be disabled by adding "PIXEL_IP=0" in the config file.

    The adblock script assumes pixelserv v32 or later to handle ssl requests. It should redirect 443 for v32 or appropriately allow 443 directly for pixelserv for v34 or later. Using one of @HunterZ 's latest versions is encouraged (https://github.com/HunterZ/pixelserv).

    Logging
    The web interface blocked/resolved hosts report listings require query logging be enabled in dnsmasq. Tomato does not do so by default.

    There are two ways to enable logging:
    1. In the Tomato GUI: add "log-queries" to "Advanced->DHCP/DNS->Dnsmasq Custom configuration" or...
    2. In the adblock config file add "dnsmasq_logqueries=1"
    Using the adblock config option is often preferable because loging can then be toggled on/off by editing the adblock config in the web interface.

    Log Location
    By default, dnsmasq will log to syslog. Dnsmasq is VERY noisy and can dump thousands of lines to syslog.

    If using syslog, make sure "Log Internally" is checked under "Administration->Logging" in the Tomato GUI. Also consider increasing "Max size before rotate" if there is not enough query history showing.

    Logging to a separate file is done by adding the dnsmasq "log-facility" option to "Advanced->DHCP/DNS->Dnsmasq Custom configuration."

    Log Rotation
    Adblock does not handle log rotation itself. If using syslog, Tomato will rotate automatically. If using log-facility, consider @HunterZ's script: http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-8#post-250455

    Scheduling Updates
    Including "cron" on the adblock command line, ie: adblock.sh cron, adds a daily job to the scheduler to update the list files.

    Default schedule is at 2:10am. Change the schedule by adding "schedule='10 02 * * *'" to the config file in standard crontab <"min hour day month week"> format.

    Config File
    The default settings file is adblock.ini. Not a very *nix-like name, but that is by design for reasons that don't really matter to anyone else.

    This is a change from @haarp's simply named "config" file, which I found too generic for my tastes. Adblock is still backward compatible with a file named "config" if it exists in the script folder and looks like an adblock file.

    Required Settings
    There is only one: SOURCES

    SOURCES must be defined for the lists to use. If using only manual blacklist entries then set SOURCES="".

    If pixelserv is not being used, then "PIXEL_IP=0" should also be set.

    All other @haarp options are still supported as well as many others, but are not required.

    See adblock.ini.readme for some of the more commonly set options. Additional options can be found by looking at the script source in the "Default values" section, but these rarely need to be touched.


    Recent Changelog Entries

    Full changelog can be found
    here.

    2015-11-11
    adblock - properly initialize redirip for recursive webui calls
    adblock - check/post error if both PIXEL_IP and redirip are set

    2015-11-08
    adblock - use appropriate netmask when setting up redirip
    install - update pixelserv to @HunterZ's V35.HZ13
    install - change from zip to tar archive (support older builds without unzip)
    install - reduce warnings/errors when installing to fat/cifs
    install - use copy if link doesn't work for pixelserv (for fat/cifs)

    2015-09-14
    adblock - add /mmc folders to config file search paths
    adblock/web - add warning text if dnsmasq logging is enabled without syslog enabled or log-facility set
    install - add support for K24 pixelserv build
    install - update pixelserv to @HunterZ's V.35HZ12 releases

    2015-09-06
    adblock - add most likely webscript url to log output
    adblock - add quietfire config option to disable firewall autorun syslog output, defaults to quietfire=1
    adblock - make adding link to tomato ui the default ( tomatolink=1 )
    adblock - whitelist/blacklist - better handling of comments
    adblock/web - force ps -w to better handle long paths
    web - change "resolved hosts" report logic, now shows requesting IP, see this discussion
    web - add option to use old "resolved hosts" report, set web_oldresolvedhosts=1 in config

    INSTALLING:
    Install via the code block below. The code block can be pasted into the Tomato Web GUI or ssh/telnet command line.

    This will download and install the adblock scripts and default config file. It also attempts to detect processor type and install the appropriate MIPS or ARM version of @HunterZ's pixelserv.

    Change PREFIX to match your install location. PREFIX must be a full path, it cannot be relative.

    If PREFIX is not defined, the script will first attempt to install to /opt/bin, then attempt to the first writable location from the following list: /opt/adblock, /jffs/adblock, /mmc/adblock, /cifs1/adblock, /cifs2/adblock.
    Code:
    # For a custom location uncomment and edit PREFIX value
    # export PREFIX=/opt/bin
    wget -O - http://goo.gl/GfA7cQ | sh
    
    Manual Install
    • Download script archive
    • Unzip files into desired folder
    • Copy sample config file adblock.ini.default to adblock.ini, place in same folder as script
    • Edit config file as needed
    • Download and install pixelserv to the script folder if desired
    • Run the script
    RUNNING:
    Adblock has traditionally been loaded from Tomato's wanup script, ie:
    Code:
    /opt/bin/adblock.sh cron
    The primary problem with the above approach is wanup can be triggered multiple times and repeatedly if there are connectivity issues or when the wan IP changes. Once adblock is initialized, adblock.sh does not need to be called repeatedly in these circumstances.

    The solution is to test if adblock is already enabled using the "adblock.is.loaded" test hostname:
    Code:
    nslookup adblock.is.loaded || /opt/bin/adblock.sh cron &
     

    Attached Files:

    Last edited: Apr 13, 2016
  2. jerrm

    jerrm Network Guru Member

    Whitelist Basics

    The whitelist is a list of regular expressions. Any lines in the blocklist that match will be stripped.

    Consider a sample blocklist containing:
    www.somedomain.net
    www.doubleclick.net
    foo.doubleclick.net
    bar.doubleclick.net
    site1.www-doubleclick.net
    doubleclick.net
    doubleclick.net.acme-cdn.net
    www.anotherdomain.com


    To remove any entry that has "doubleclick.net" anywhere in the name, add doubleclick.net to the whitelist.
    www.somedomain.net
    www.doubleclick.net
    foo.doubleclick.net
    bar.doubleclick.net
    site1.www-doubleclick.net
    doubleclick.net
    doubleclick.net.acme-cdn.net
    www.anotherdomain.com

    To remove only "doubleclick.net" but block www.doubleclick.net, foo.doubleclick.net, bar.doubleclick.net, etc, then add ^doubleclick.net$ to the whitelist.
    www.somedomain.net
    www.doubleclick.net
    foo.doubleclick.net
    bar.doubleclick.net
    site1.www-doubleclick.net
    doubleclick.net
    doubleclick.net.acme-cdn.net
    www.anotherdomain.com

    To remove www.doubleclick.net, foo.doubleclick.net, bar.doubleclick.net, etc, but block "doubleclick.net" itself, add .doubleclick.net to the whitelist.
    www.somedomain.net
    www.doubleclick.net
    foo.doubleclick.net
    bar.doubleclick.net
    site1.www-doubleclick.net
    doubleclick.net
    doubleclick.net.acme-cdn.net
    www.anotherdomain.com

    Remember a dot "." in a regular expression will match any character, so there is a possibility of overmatch.
    www.somedomain.net
    www.doubleclick.net
    foo.doubleclick.net
    bar.doubleclick.net
    site1.www-doubleclick.net
    doubleclick.net
    doubleclick.net.acme-cdn.net
    www.anotherdomain.com

    If you want to make 100% sure you exactly match only "www.doubleclick.net" then escape the dots - add ^www\.doubleclick\.net$ to the whitelist.
    www.somedomain.net
    www.doubleclick.net
    foo.doubleclick.net
    bar.doubleclick.net
    site1.www-doubleclick.net
    doubleclick.net
    doubleclick.net.acme-cdn.net
    www.anotherdomain.com

    Whitelist vs LEGACY/OPTIMIZE/HOST Modes

    HOST Mode
    HOST mode generally interacts with the whitelist as one would expect based on the above description. Each entry in the blocklist represents an individual host. If a line in the blocklist matches the whitelist it is removed and the host is not redirected by dnsmasq.

    LEGACY/OPTIMIZE Mode
    The blocklist in LEGACY or OPTIMIZE is treated differently by dnsmasq. Essentially each line in the blocklist represents both a host and a domain.

    Consider a blocklist that contains:
    Code:
    www.doubleclick.net
    doubleclick.net
    If "www.doubleclick.net" is whitelisted, it will still be blocked. "Doubleclick.net" will remain in the blocklist, causing any host that ends in "doubleclick.net" to be blocked.
     
    Last edited: Mar 13, 2016
  3. jerrm

    jerrm Network Guru Member

    Placeholder
     
  4. jerrm

    jerrm Network Guru Member

    Placeholder
     
  5. jerrm

    jerrm Network Guru Member

    Last edited: Mar 13, 2016
    TucknDar and vincom like this.
  6. vincom

    vincom LI Guru Member

    ty, its time(finally) for its/your own thread
     
  7. aksaraff

    aksaraff New Member Member

    Thanks for forking the threads and the updated instructions! The only struggle I have at the moment is all the dirname constructs - my install of Tomato does not have the binary and so I am trying to figure out what the commands expand to so that I can get a script with the right locations before running it.
     
  8. roberthuang

    roberthuang Serious Server Member

    @jerrm

    Thank you so much to create a new thread for this script. This is definitely much easier for people to follow up, especially for those who never used the script before and want to give it a try.

    One suggestion, could you add a small description section of how to uninstall (remove) the script?
     
  9. aksaraff

    aksaraff New Member Member

    This is the latest output I get after fiddling around with the config files and editing the dirname entries in the script -

    Code:
    # ./adblock.sh
    ADBLOCK[13310]: Running as /jffs/jerm/adblock.sh
    ADBLOCK[13310]: Using config file /jffs/jerm//adblock.ini
    ADBLOCK[13310]: Requested list mode is OPTIMIZE
    df: ramfs: No such file or directory
    ln: /www/user/adblock.sh: No such file or directory
    ADBLOCK[13310]: ERROR - could not create web link /www/user/adblock.sh
    ADBLOCK[13310]: List not old enough to update
    ADBLOCK[13310]: Setting up 192.168.1.254 netmask 255.255.255.0 on br0:adblk
    ADBLOCK[13310]: Setting up pixelserv on 192.168.1.254
    ADBLOCK[13310]: pixelserv[13472]: clock_gettime() reports CLOCK_MONOTONIC not supported; switching to less accurate CLOCK_REALTIME
    ADBLOCK[13310]: pixelserv[13472]: /jffs/jerm//pixelserv version: V35.HZ13 compiled: Oct  6 2015 22:34:25 options: 192.168.1.254
    ADBLOCK[13310]: Writing File /etc/dnsmasq.custom
    ADBLOCK[13310]: CONF file /etc/dnsmasq.custom changed
    ADBLOCK[13310]: Restarting dnsmasq
    ADBLOCK[13310]: ..
    ADBLOCK[13310]: Done.
    ADBLOCK[13310]: Exiting /jffs/jerm/adblock.sh 0
    
    I downloaded the static version of the MIPS K24 pixelserv and rewired the soft link. The startup messages look good but `ps` shows that the binary isn't running which leads to all ad-requests being redirected to the home page of my router and the consequent login prompt.
     
  10. roberthuang

    roberthuang Serious Server Member

    deleted
     
    Last edited: Mar 14, 2016
  11. jerrm

    jerrm Network Guru Member

    What version of Tomato is this?
     
  12. aksaraff

    aksaraff New Member Member

    Code:
    Tomato Firmware v1.28.1816
    Copyright (C) 2006-2010 Jonathan Zarate
    http://www.polarcloud.com/tomato/
    
    Built on Sun, 27 Jun 2010 20:10:52 -0700
     
  13. vincom

    vincom LI Guru Member

    from op:
    THIS SCRIPT IS INTENDED FOR A MODERN TOMATO BUILD, FROM 2013 OR LATER
     
  14. aksaraff

    aksaraff New Member Member

    I just discovered all of shibby's images! Now all I need to figure out is which one works best on my Linksys WRT54GS v4 router.
     
  15. jerrm

    jerrm Network Guru Member

    As @vincom points out, I can't really support a build that old. I have no way to test or develop.
     
  16. aksaraff

    aksaraff New Member Member

    The fault was mine - I was unaware of shibby's builds, instead, relying on polarcloud for my information. I upgraded to tomato-K26-1.28.RT-MIPSR1-132-Mini.zip and everything works perfectly! Thanks for all the help.

    The only outstanding issue is that of disk size but I don't believe there's anything you can do about that. Since the router has 4MB of flash memory, I can't allocate anything bigger than 64KB for the jffs partition so the entire install of adblock currently resides on ramdisk. I have entries to my Wan UP script to re-install in case of a power cycle so it isn't a blocker at the moment. I'll post the script later to get feedback.
     
  17. roberthuang

    roberthuang Serious Server Member

    ======Update on March 17th, 2016==================
    The Pixelserv is working good after I reloaded the router.
    ============================================

    ===========Original post shows below================
    Hi All,

    Please help me troubleshoot why the pixelserv is not running after I follow the OP's instruction to install the adblock. Please see the attached screenshot for the pixelserv status. Thanks in advance.

    Router: Netgear R6250
    Firmware: tomato-R6250-ARM--132-AIO-64K.zip
     

    Attached Files:

    Last edited: Mar 27, 2016
  18. jerrm

    jerrm Network Guru Member

    Post or pm output of "adblock.sh debug"
     
  19. AndreDVJ

    AndreDVJ LI Guru Member

    I had few problems with pixelserv not starting on my R7000. What I am about to tell logically makes no sense, but it seems to be resolved after I compiled pixelserv again.

    Maybe latest libraries from Tomatoware (1.2.1) helped.

    I am attaching binaries for ARM. Please try replacing pixelserv.tomatoware.performance.static, and see how it goes.
     

    Attached Files:

  20. roberthuang

    roberthuang Serious Server Member

    Thanks for your reply. I've attached the output of "adblock.sh debug".
     
  21. Tevatron

    Tevatron New Member Member

    Hi.

    Firmware is Tomato Firmware 1.28.0000 MIPSR2-132 K26AC USB AIO-64K
    Router is RT-N66U mips 600 mhz single core.

    With default LISTMODE=OPTIMIZE and with all sources enabled dnsmasq cpu usage fluctuates from 0% to 100% resulting in 1.31 Load Average in 1 min. After removing 6 megabytes and 3 megabytes sources dnsmasq cpu usage fluctuates from 0 to 30%.

    With LISTMODE=LEGACY cpu usage same.

    With LISTMODE=HOST and with all sources enabled dnsmasq dont use cpu more than in peak 10%. resulting in very low load average.

    Is it normal that LISTMODE=OPTIMIZE and LEGACY eats so much cpu?

    http://www.linksysinfo.org/index.php?threads/high-cpu-with-network-utilization-shibby-1-28.70766/

    And the ADBLOCK Script Kills the N16 in Seconds.
    -----------------------------------------------------------

    Try this:

    Disable:
    -Adblock
    -ALL LOGGING (Yes also the WebLogging <- 10-20% CPU LOAD
     
    Last edited: Mar 16, 2016
  22. jerrm

    jerrm Network Guru Member

    Unfortunately the debug didn't shed any light. If the re-compiled pixelserv doesn't help. Reboot with adblock disabled and post the output of:
    Code:
    echo "
    ifconfig
    ==============================================================="
    ifconfig
    echo "
    
    ===============================================================
    netstat
    ==============================================================="
    netstat -anl
    
    echo "
    
    ===============================================================
    ifconfig br0:test 192.168.1.252  up
    ==============================================================="
    
    ifconfig br0:test 192.168.1.252  up
    ifconfig
    
    echo "
    
    ===============================================================
    pixelserv 192.168.1.252
    ==============================================================="
    pixelserv 192.168.1.252
    netstat -anl
    
    
     
  23. roberthuang

    roberthuang Serious Server Member

    I rebooted the router and the pixelserv is working as expected after the reload.

    Thank you again for your help.
     
  24. IngoPan

    IngoPan Serious Server Member

    Whats the reason why i can´t see pixelserv in the ps task list? Also a killall pixelserv doesn´t work=no such task.
    Whats the take here?

    Thanks.

    @edit: Seems like i can´t assign a proper ip?
    pixelserv 192.168.2.252 -f
    pixelserv[2071]: pixelserv version: V35.HZ12WIP1 compiled: Jul 6 2015 19:20:09 options: 192.168.2.252 -f
    pixelserv[2071]: Abort: Cannot assign requested address - :192.168.2.252:80
     
    Last edited: Mar 18, 2016
  25. meazz1

    meazz1 LI Guru Member

    I have this running on my Asus RT-AC56U on Shibby fw.
    For weekly updates, do I need to manually run the ./adblock.sh or it's automatically runs the update at certain time of the week/day?
     
  26. jerrm

    jerrm Network Guru Member

    See "Scheduling Updates" in 1st post.
     
    meazz1 likes this.
  27. meazz1

    meazz1 LI Guru Member

    Can you explain a bit more , ie: adblock.sh cron, do I make a script to run daily or add that to the adblock.ini or adblock.sh file?
     
    Last edited: Mar 22, 2016
  28. jerrm

    jerrm Network Guru Member

    Start the script with the cron command line parameter and it schedules a nightly update.
     
    meazz1 likes this.
  29. meazz1

    meazz1 LI Guru Member

    Thaks, got everything figured out with your help.
     
  30. theoctavist

    theoctavist Reformed Router Member

    **ULTIMATELY** I would like to have one partition from which to run this script and one partition for entware. (curious... is there any reason I can not install /run this script from the same directory that contains entware?) there is not any specific reason for my interest in entware, other than a general desire to learn about it) thank you for your patience.


    EDIT, got it sorted. thank you for this script

    EDIT agh. no i did not. ok.. so when I issue a command via tomato gui and there is a restart, something weird happens. the "bit torrent" option in the tomato gui shows up, but the adblock (per your script) dissapears. and any attempts to run ad block fail. help?
     
    Last edited: Mar 23, 2016
  31. infekto

    infekto Reformed Router Member

    This is awesome, thanks for making it! If a consolidated hostfile is of any use this one is pretty good github [d0t] c0m/ StevenBlack/hosts
     
  32. theoctavist

    theoctavist Reformed Router Member

    frustration continues. dont know what is going on here. just re-formatted the USB drive using gparted (ext2)

    root@unknown:/tmp/home/root# cat /proc/mounts
    rootfs / rootfs rw 0 0
    /dev/root / squashfs ro 0 0
    proc /proc proc rw 0 0
    tmpfs /tmp tmpfs rw 0 0
    devfs /dev tmpfs rw,noatime 0 0
    sysfs /sys sysfs rw 0 0
    devpts /dev/pts devpts rw 0 0
    usbfs /proc/bus/usb usbfs rw 0 0
    /dev/sda1 /tmp/mnt/hubris ext2 rw,nodev,noatime 0 0
     
  33. Michael Malone

    Michael Malone Network Newbie Member

    How does this compare to everyone else?

    Linksys E1200 v2.0 32MB
    Tomato Firmware 1.28.0000 MIPSR2-133 K26 Max
    hosts: 31514
     
  34. voka

    voka Networkin' Nut Member

    You can still whitelist host in LEGACY/OPTIMIZE Mode by using something like this:


    the dnsmasq server option overrides the address option.
     
  35. JoeDirte

    JoeDirte Networkin' Nut Member

    Just wanted to say that I recently upgraded my firmware and decided to try this out on a clean install. I was using an edited version of the lean, mean... Anyway, I must say I am impressed that the install was so easy and I really like the web interface addition. Thanks a bunch!
     
  36. jsnepo

    jsnepo Serious Server Member

    I created a guest wifi (br1). Is there a way to apply this for it as well?
     
  37. joksik

    joksik New Member Member

    hello. can anybody tell me how to uninstall pixelserv and this ad block?i have install to directory /jffs/adblock but i don`t know what command write in console to uninstall it both. simple remove all folder??
     
  38. koitsu

    koitsu Network Guru Member

    rm -fr /jffs/adblock
     
    HunterZ likes this.
  39. HunterZ

    HunterZ Network Guru Member

    If nothing else is in /jffs/adblock then you can just type this command:
    rm -rf /jffs/adblock

    This will remove the adblock directory and anything in it.

    Then just remove any reference to it from your router GUI and reboot the router.
     
  40. joksik

    joksik New Member Member

    great guys! but maybe i does not need to remove adblock of course with your help. in dnsmasq custom i add:

    log-facility=/jffs/adblock/ablock.log

    now i have nice log file without syslog but after 3 days of using adblock this file have ~10mb :/ and i have only 15mb free left on jffs. so this log file is very big. what can i add to "dnsmasq custom" to limit log size? maybe log-async=5 will help reduce log size? or better add simple rm -fr /jffs/adblock/ablock.log to cron to remove log file automatically at midnight?
     
  41. koitsu

    koitsu Network Guru Member

    log-async has absolutely nothing to do with log rotation or "cleansing".

    dnsmasq does not have any kind of log rotation or log management capability -- nor should it.

    You cannot safely remove the log file in the fashion you allude to (cronjob doing nothing but rm /jffs/adblock/ablock.log). If you do this, the file will disappear from a directory listing, but dnsmasq will continue to hold open the file descriptor (hence inode) of the file, so the space will never be freed up. This is how UNIX filesystems work and is completely normal. The only way for the space to be freed is to make dnsmasq close the file handle/descriptor it has open on the log file.

    How to do that is discussed in the dnsmasq documentation, section "NOTES": you need to send the dnsmasq process a SIGUSR2 signal which will force it to close its log file and reopen it. In other words, your cronjob would need to be this:

    Code:
    rm /jffs/adblock/ablock.log && killall -s USR2 dnsmasq
    
     
    jerrm likes this.
  42. jerrm

    jerrm Network Guru Member

    See http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-8#post-250455
     
    HunterZ likes this.
  43. Xytrios

    Xytrios Guest

    ...
     
    Last edited by a moderator: Apr 14, 2018
  44. theoctavist

    theoctavist Reformed Router Member

    @Xytrios
    you need entware-ng and the SSL package.(openssl-util) I am using that very file. hosts: 33046
     
    Last edited: Apr 14, 2016
  45. jerrm

    jerrm Network Guru Member

    For ARM, it should work without entware.

    For MIPS, entware-ng or optware, the openssl-util or wget packages need to be installed. libopenssl is not enough, the openssl executable needs to be present.
     
  46. Xytrios

    Xytrios Guest

    ...
     
    Last edited by a moderator: Apr 14, 2018
  47. theoctavist

    theoctavist Reformed Router Member

  48. joksik

    joksik New Member Member

    thx koitsu i do like you say, everything is good.thx guys

    edit:
    can i add some another list from adblock?i must change only sources?
     
    Last edited: Apr 14, 2016
  49. theoctavist

    theoctavist Reformed Router Member

    yes, but JFFS partition is very small so be mindful
     
  50. Xytrios

    Xytrios Guest

    ...
     
    Last edited by a moderator: Apr 14, 2018
  51. jerrm

    jerrm Network Guru Member

    This is a MIPS only issue.

    The list you want is hosted on github, which only allows secured (https) connections. Tomato's busybox wget does not natively support https urls. It needs a helper application in openssl to do so.

    The openssl executable included in MIPS builds is very limited, and does not include the options needed by busybox wget to handle https. This is one of the tradeoffs of the MIPS code base needing to accommodate routers with extremely limited flash..

    The only workaround is to install either the full version of openssl or the full version wget from one of the available repos.

    The option in busybox wget to support https at all is only about a year old itself. Previously the only option was installing the full version of wget.

    There is nothing the script can do about this.

    Shibby ARM builds have the needed openssl options and https urls work just fine.
     
  52. reimer

    reimer Addicted to LI Member

    If you go to the Stevens hosts page
    https://github.com/StevenBlack/hosts

    He lists all the sources of his unified hosts file under the "Sources of hosts data unified here" heading

    So, as far as I can tell, you can just use those sources instead. They are all http connections
     
    visceralpsyche likes this.
  53. Spyros

    Spyros LI Guru Member

    Im using an E2000 on Shibby's Tomato Firmware 1.28.0000 MIPSR2-136 K26 Max (updated yesterday after one year :p)

    My jffs is 1.2MB, old small and tired so im using it as read only and thus avoiding frequent writes and file changes. Just put the required files in /jffs/adblock and use a simple script to create a folder, transfer, change permissions and execute in /tmp/adblock, i was using this even with the lean and mean adblocking method. Here is my wan up script:


    Code:
    ### Copy Adblock Script to /tmp   ###
    mkdir /tmp/adblock
    cp -r /jffs/adblock/* /tmp/adblock
    chmod +x /tmp/adblock/adblock.sh
    chmod +x /tmp/adblock/adblockweb.sh
    chmod +x /tmp/adblock/pixelserv
    ###   Enable AdBlock  ###
    /tmp/adblock/adblock.sh


     
    Jorge Benavides and khacduy1978 like this.
  54. James Charles

    James Charles New Member Member

    On an RT-N16 using the latest version of Shibby, I get "blocklist: down" -- any thoughts on what might potentially be causing this problem? I'm using JFFS.
     
  55. HunterZ

    HunterZ Network Guru Member

    Anything relevant in the router logs?
     
  56. Jorge Benavides

    Jorge Benavides Connected Client Member

    Hey Spyros, thank you sooooo much.
    That gave good breathing space to my WRT320 converted to E2000.

    Regards!
     
    Spyros likes this.
  57. Spyros

    Spyros LI Guru Member

    mine is also converted, almost forgotten it after so many years...long live wrt320n :)
     
    Jorge Benavides likes this.
  58. Jorge Benavides

    Jorge Benavides Connected Client Member

    Hey that's great. I love mine too, but currently thinking on getting an ASUS RT-AC68U or RT-AC66U so I can take advantage of the 5MHz band. Wife's phone works only in 2.4 so I'd need both :(.

    Have you tried StevenBlack's hosts file? My router simply dies when I try to use it, so basically I decided to use the six smaller lists and booted with LISTMODE=OPTIMIZE. Any recommendations?

    Thank you very much.
     
    Last edited: May 8, 2016
    Spyros likes this.
  59. Spyros

    Spyros LI Guru Member

    Ιm using 4 sources, the same with android's adaway app from xda

    it takes 93 seconds to compile and its fine
     
    Jorge Benavides likes this.
  60. Jorge Benavides

    Jorge Benavides Connected Client Member

    Mine are the same as your but added Cameleon Project's and MalwareDomainList's. Takes 126 sec so I think it's fine too. Currently gives me a list of 31500 blocked sites.
     
  61. HunterZ

    HunterZ Network Guru Member

    I'm using winhelp, yoyo, hosts-file, malwaredomainlist, and hostsfile.mine.nu, because they all allow adblock to check the timestamp of the file before downloading it. This allows me to run adblock as often as I want without wasting bandwidth, risking bans, or wasting time rebuilding the blocklist.

    I'm at 67728 blocked sites, with up to 7 of them being custom blacklist entries.
     
    Jorge Benavides likes this.
  62. Jorge Benavides

    Jorge Benavides Connected Client Member

    Sounds great, HunterZ.
    And what's the hardware you use to run it? How much time does the list take to compile?
    Regards!
     
  63. HunterZ

    HunterZ Network Guru Member

    Looks like 140 seconds on my RT-N66U.
     
    Jorge Benavides likes this.
  64. Jorge Benavides

    Jorge Benavides Connected Client Member

    Well, checking my rehearsals, my E2000 dies with the hosts-file, so I think I'll stay with my current configuration until I can get some more capable hardware :( but anyways thanks on the advise! Thumbs up to you.
     
  65. theoctavist

    theoctavist Reformed Router Member

    where is the hosts-file site? cant find it. google search of course doesnt help
     
  66. ΦDroid

    ΦDroid Serious Server Member

    http://hosts-file.net

    Sent from my ONE A2005 using Tapatalk
     
  67. Jorge Benavides

    Jorge Benavides Connected Client Member

    theoctavist, in the adblock config file, you should delete the hashtag for the second line shown below:

    ## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list)
    #SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt http://hosts-file.net/hphosts-partial.txt"

    Hope this helps.
     
    theoctavist likes this.
  68. HunterZ

    HunterZ Network Guru Member

    Here is my adblock.ini in case it helps: http://pastebin.com/XmKabx5E

    I moved my dnsmasq log to /var/log when I transitioned from CIFS to USB in order to avoid extra wear on my USB stick. I don't really care about persisting the log across reboots. I still run my log rotation script to rotate every 24 hours, so that it doesn't eat up all the RAM on the router.
     
  69. Frequenzy

    Frequenzy Networkin' Nut Member

    how do I prevent the script from creating the weblink since I don't use it.

    ADBLOCK[23783]: Creating web link /www/user/adblock.sh
    ADBLOCK[23783]: Web interface should be available at http://192.168.2.1/user/adblock.sh
     
  70. AndreDVJ

    AndreDVJ LI Guru Member

    If you really want to get rid of adblock weblink that bad, add variable weblink with a NULL value (weblink=""), then comment two lines headed with # write weblink to skip weblink checking, so it looks like this:
    Code:
    # write weblink
    if [ "$weblink" != "" ] &&  [ -x "$binprefix/$webscript" -o -x "$( which "$webscript" )" ]; then
        if ln -sf "$me" "$weblink" ; then
            local lanport=$(nvram get http_lanport)
            [ "$lanport" = 80 -o "$lanport" = "" ] && lanport="" || lanport=":$lanport"
            elog "Creating web link $weblink"
            elog "Web interface should be available at http://$(nvram get lan_ipaddr)$lanport/user/${weblink##*/}"
            echo "$weblink" >  $weblink.weblink
            addtomatolink
        else
            elog "ERROR - could not create web link $weblink"
        fi
    #else
    #    elog "ERROR - Web Script $webscript not found or not executable!"
    fi
     
  71. Tuurbo

    Tuurbo Reformed Router Member

    I would like to start by saying that I'm really a HUGH fan of your script jerrm!

    In my search to make this script working on Asuswrt-Merlin, I've stumbled upon a fork of pixelserv: pixelserv-tls. Any thoughts about this? It looks of having to option to serve a certificate with a SSL request?

    P.S. I've been using it for quite a while now, but unfortunately the (wireless) performance of my RT-AC66U on TomatoUSB isn't that great. I'm thinking of switching back to Asuswrt-Merlin, and will try to get you script working with it. Do you have experience with Asuswrt?

    EDIT: Back on TomatoUSB! In my case, changing the Transmit Power to 0 (hardware default) on my RT-AC66U, improves the wireless performance significantly!
     
    Last edited: May 20, 2016
  72. jerrm

    jerrm Network Guru Member

    No interest in pixelserv-tls. The browser issues it addresses are not a concern for me.

    As stated in the first post, the script is targeted only for Tomato. Merlin does not support dnsmasq.custom, but I think there is an analogue. No idea how the firewall rules will play with Merlin. I believe the admin httpd binds all addresses, so the admin server will need to run on an alternate port, or use iptables to redirect. Good luck.
     
  73. HunterZ

    HunterZ Network Guru Member

    @Tuurbo What wireless performance metrics are poor with TomatoUSB?
     
  74. Tuurbo

    Tuurbo Reformed Router Member

    Changing to pixels-tls is easy (especially if you're on Entware-ng). To keep maximum compatibility with your script I've only changed the symbolic link located at /opt/ect/pixelserv.

    If you're visiting a website which serves ads on HTTPS, you can clearly see the difference:

    without_cert.png with_cert.png

    Full disclosure; For this to work you require a Root CA cert and import this certificate into every client. This is probably for the more experienced users, but not impossible to integrate in your script. Maybe you could detect if there's a Entware environment (or a clean JFFS?) and set up the necessary tools (coreutils-mktemp, EasyRSA and pixelserv-tls).

    @HunterZ The problem of my wireless performance is a lack of knowledge by myself. Lucky I found the problem and I'm back on TomatoUSB.

    UPDATE: I received some questions about this, where I see now that my previous explanation wasn't clear. It's a best of both worlds; Without any changes to the client, this works exactly the same as the original. Only if you choose to install (your self signed) certificate, you experience no problems on HTTPS websites, like the example above.
     
    Last edited: May 23, 2016
  75. jerrm

    jerrm Network Guru Member

    I have absolutely no interest in this. The whole point of adblock on the router (for me) is that I don't need to muck with the clients. Otherwise just install a browser add in.
     
    JoeDirte likes this.
  76. Tuurbo

    Tuurbo Reformed Router Member

    You have a really good point there and you're absolutely right about that. Thanks voor listing and I'm looking forward to future updates :)
     
  77. Bird333

    Bird333 Network Guru Member

    Jerrm, what block lists are you using in your config?
     
  78. Justzee

    Justzee New Member Member

    Please I am a new user to tomato and all this code but i am very willing to learn. I want to add this adblock to my tomato version 1.28 by shabby. I am seeing all this add script and code. Where exactly do I enter this into my Tomato GUI. Can someone be kind enough to show me a step by step guide to do this?

    Please don’t be offended i am asking very basic questions i am very new to computer programming. please help
     
  79. vincom

    vincom LI Guru Member

    This is not the right forum/thread for you.
    Learn how to use google.
    Read as much as u can.
    There's a learning curve and u must be willing to learn.
    U can't go from caveman to astronaut overnight or by reading a few guides.
    U can't expect someone to post in this thread a tutorial on learning computer basics and/or networking .
    We were all newbs once and all the info is out there, the more u read the more you'll learn.
    Or just use a browser addon like adblock
     
    Tuurbo likes this.
  80. Mr9v9

    Mr9v9 Serious Server Member

    I love the work put into this script! I'm having one issue while testing the lists on my browsers I get the message for a "Bad SSL client authentication certificate" problem on some ads and pages.

    I also run DNSSEC and dnscrypt-proxy at the same time:
    Code:
    May 25 22:12:11 Router daemon.debug dnscrypt-proxy[1432]: resolver timeout (UDP)
    It's not a huge issue if it's normal behavior on startup but is it anything I need to worry about?
     

    Attached Files:

    Last edited: May 26, 2016
  81. Justzee

    Justzee New Member Member

    hello i just run '
    # For a custom location uncomment and edit PREFIX value
    # export PREFIX=/opt/bin
    wget -O - http://goo.gl/GfA7cQ | sh
    in Tools>system commands -----and this is what i got,

    Connecting to goo.gl (216.58.210.46:80)
    Connecting to tomato-adblock. weebly. com (199.34.228.54:80)
    adblock-install: PREFIX not set, looking for default folders
    mkdir: can't create directory '/opt/adblock': Read-only file system
    adblock-install: installing binaries and scripts to /jffs/adblock, config to /jffs/adblock/adblock.ini
    Connecting to tomato-adblock. weebly. com (199.34.228.53:80)
    adblock.sh
    adblockweb.sh
    adblock.changelog
    adblock.ini.default
    adblock.ini.readme
    pixelserv/arm/
    pixelserv/arm/LICENSE
    pixelserv/arm/README.md
    pixelserv/arm/pixelserv.tomatoware.performance.dynamic
    pixelserv/arm/pixelserv.tomatoware.performance.static
    pixelserv/arm/VERSION
    pixelserv/mips/
    pixelserv/mips/LICENSE
    pixelserv/mips/README.md
    pixelserv/mips/pixelserv.tomatoware.performance.dynamic
    pixelserv/mips/pixelserv.tomatoware.performance.static
    pixelserv/mips/VERSION
    pixelserv/mipsK24/
    pixelserv/mipsK24/LICENSE
    pixelserv/mipsK24/README.md
    pixelserv/mipsK24/pixelserv.mips.performance.dynamic
    pixelserv/mipsK24/pixelserv.mips.performance.static
    adblock-install: installing /jffs/adblock/adblock.changelog
    adblock-install: installing /jffs/adblock/adblock.ini.readme
    adblock-install: installing /jffs/adblock/adblock.ini.default
    adblock-install: installing /jffs/adblock/adblock.sh
    adblock-install: installing /jffs/adblock/adblockweb.sh
    adblock-install: installing /jffs/adblock/pixelserv.mips.performance.dynamic
    adblock-install: creating 'pixelserv' link for /jffs/adblock/pixelserv.mips.performance.dynamic
    adblock-install: installing default config file /jffs/adblock/adblock.ini


    can anyone guide me further please?
     
  82. koitsu

    koitsu Network Guru Member

    Looks pretty definitive to me. Provide output from mount?
     
  83. Justzee

    Justzee New Member Member

    Please how do I provide the output from mount?
     
  84. HunterZ

    HunterZ Network Guru Member

    If you keep looking, I think it moved on to try and succeed at a JFFS install.

    @Justzee you need to look at the RUNNING section of the first post in this thread, and note that your adblock is in /jffs/adblock rather than /opt/bin
     
  85. Tuurbo

    Tuurbo Reformed Router Member

  86. Mr9v9

    Mr9v9 Serious Server Member

    So is this the only way to hide HTTPS ads from showing up? Is there another way to make this easy so some users on my network aren't annoyed?
     

    Attached Files:

  87. HunterZ

    HunterZ Network Guru Member

    For HTTPS your options are the following:
    • Use stock pixelserv, which quickly closes HTTPS sessions with an error code. This displays an error in the browser but is fast and works on all LAN clients without any special client configuration.
    • Use pixelserv-tls or maybe stunnel and install a fake certificate on *every* LAN client that you want to fully hide the ads on. I'm not sure if there's a performance loss for clients that do not install the certificate. This may also cause problems if you want to disable blocking of an HTTPS site after browsers have associated the fake certificate with its domain name.

    Since half of the web browsers in my home are mobile devices, and my guests (extended family, etc.) are not technically-inclined enough to even know what a certificate is, I decided to just go with the first option.
     
    JoeDirte likes this.
  88. Tuurbo

    Tuurbo Reformed Router Member

    Can you give us some more info about the potential problems we can expect? (if we want to disable blocking an HTTPS site after browsers have associated the fake certificate).

    Schermafbeelding 2016-05-27 om 21.01.56.png Schermafbeelding 2016-05-27 om 21.02.26.png
    I've tested this on my Mac. Where I first visited the site where it uses my own generated cert. When I disable adblock (and flush the DNS cache) and visit the same site again I didn't experience any problems. Or is there a better way to test this?

    I'm running for about a week now with pixelserv-tls. While it's still early to draw conclusions, I'm very pleased with it. I agree that installing the cert on every device is annoying, that's why I've only installed it on my main machine. For example my wife's laptop, iPad en (Android) smartphone doesn't have a cert installed, where there's no noticeable performance loss.
     
  89. vincom

    vincom LI Guru Member

    no not really from the router level but at the device level just use the adblock app for browsers
     
  90. HunterZ

    HunterZ Network Guru Member

    Good to know. Which browsers have you tested with? They all seem to react differently. For example, I added HTTPS error code customization to the latest version of stock pixelserv because my mother in law was seeing weird behavior from Safari on a Macbook.

    Cool. What is the failure mode on the non-cert-installed devices? Some kind of invalid certificate error I'm assuming?
     
  91. SloBurn

    SloBurn New Member Member

  92. my_bey

    my_bey Serious Server Member

    My adblock stopped working properly. It was working flawlessly before.
    First my configuration:
    Tomato Firmware 1.28.0000 -136 K26ARM USB AIO-64K
    R7000
    Adblock release="2015-11-11"

    Yesterday, I realized I lost connection to internet and a reboot fixed the problem. Adblock did not run automatically after the reboot even though I have this in the in the Admin Init:
    /mnt/DATA/adblock/adblock.sh


    When i ran it manually from the system commands, I get this unusual page now at https://192.168.2.1/user/adblock.sh

    Empty white space under recently blocked hosts and recently resolved hosts instead of links populating under each.

    adblock status:
    blocklist: up
    iptables: up - 11 rules
    pixelserv: up
    logging: up
    hosts: 10217
    ttl: 0
    adblock actions:
    force
    start/update
    restart
    stop

    edit lists
    edit config
    time info:
    08:12:52 up 22:40, load average: 0.00, 0.04, 0.10

    pixelserv info:
    /mnt/DATA/adblock/pixelserv version: V35.HZ13 compiled: Nov 8 2015 23:33:28 options: 192.168.2.254
    2405 uts, 85 req, 437 avg, 813 rmx, 15 tav, 38 tmx, 0 err, 0 tmo, 0 cls, 0 nou, 0 pth, 43 nfe, 0 ufe, 0 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 16 ssl, 26 sta, 0 stt, 0 204, 0 rdr, 0 pst, 0 hed

    page will automatically refresh in 95 seconds
    recently blocked hosts:

    recently resolved hosts:

    In addition, my logfile is flooded with "daemon.info dnsmasq[27248]:" messages.

    Please let me know what is happening.

    Thanks in advance.
     
  93. HunterZ

    HunterZ Network Guru Member

    @my_bey sounds like maybe your dnsmasq didn't get configured to log properly for adblock.

    Also, you may want to have it run from WAN Up instead of init.
     
  94. my_bey

    my_bey Serious Server Member

    I moved the activation of adblock to WANUp instead of Init. After a reboot, Adblock started. :)
    Now, adblock status page is still blank.
    How do I ensure dnsmasq is configured correctly? I have this configuration script under Custom Configuration, I never had an issue before:

     
  95. my_bey

    my_bey Serious Server Member

    AdblockPage.JPG
    Is adblock broken?
    Above is the image I get.
    I think it is still working in the background because ads seems to be blocked but this page does not display the hosts blocked or allowed anymore. Is it related to dnsmasq?

    Also my dnsmasq.log is empty even though this is specified:

    dnsmasq_logqueries=1
    dnsmasq_custom="log-facility=/mnt/DATA/adblock/dnsmasq.log"
     
  96. HunterZ

    HunterZ Network Guru Member

    Yes, the script tries to parse the dnsmasq log to get that info.

    Have you checked that you can actually write to that file at that path?
     
  97. my_bey

    my_bey Serious Server Member

    From Windows, I am able to type text, save it close the file. My edits from Windows seems to be successfully saved. I also tried to see from WinSCP if I can write to it, yes, I can write to it any text.

    Pls check the screen capture below. Rights seem to be OK but what is "nobody" as owner while everything else is "root"?


    Ok, I decided to uninstall adblock and reinstall it. That fixed the issues.
    Thank you for your assistance. Not sure why it stopped working though!
     

    Attached Files:

    Last edited: Jun 12, 2016
  98. HunterZ

    HunterZ Network Guru Member

    If you're editing from Windows, make sure to edit with something like Notepad++ where you can set Unix line endings.
     
  99. my_bey

    my_bey Serious Server Member

    I used EditPlus from Windows. But now, I reinstalled it with default config, and I did not enable logging this time.

    6/14/2013: A few days later, I realized I lost the function again. Uninstalling and Reinstalling the adblock did not fix the problem.
     
    Last edited: Jun 15, 2016
  100. srouquette

    srouquette Network Guru Member

    I installed AdvancedTomato on my new R7000, but it seems conf-file doesn't work in dnsmasq.conf, and the blocklist isn't read.
    Is there a workaround?
    Is it related to AdvancedTomato or Shibby's build?

    edit: ok, using LISTMODE=HOST until it's fixed.
     
    Last edited: Jun 19, 2016
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice