Script: Clean, Lean and Mean Adblocking

Discussion in 'Tomato Firmware' started by haarp, Apr 23, 2013.

  1. ΦDroid

    ΦDroid Serious Server Member

    I think ads are hard coded into the YouTube app with IP addresses, so they cannot be blocked with hosts files. Personally, I use the YouTube adaway xposed module.

    Sent from my ONE A2005 using Tapatalk
     
    crusher9 likes this.
  2. QSxx

    QSxx Network Guru Member

    Any idea why this happens (nslookup error - see screenshot) - I tried reinstalling script (it runs on ext4 usb in /tmp/mnt/sda1/adblock/), checking for illegal characters in conf file; adblock script; logs... Can't seem to find what's wrong :(

    [​IMG]

    Also this happens alot on the recently resolved tab

    Code:
    Jan 30 14:27:53 10.1.1.100      api.miwifi.com
    Jan 30 14:27:51 10.1.2.120      mtalk4.google.com
    Jan 30 14:27:51 10.1.2.120      graph.facebook.com
    Jan 30 14:27:49 10.1.1.70       mtalk.google.com
    Jan 30 14:27:41 10.1.2.120      edge-mqtt.facebook.com
    Jan 30 14:27:39 10.1.1.133      weather.yahooapis.com
    Jan 30 14:27:35 10.1.1.70       dl-debug33.dropbox.com
    Jan 30 14:27:31 10.1.1.133      connectivitycheck.android.com
    Jan 30 14:27:28 10.1.1.70       clients4.google.com
    Jan 30 14:27:18 reading         dnsmasq[29516]:
    Jan 30 14:26:57 10.1.2.120      www.google.com
    Jan 30 14:26:56 10.1.1.70       dl-debug.dropbox.com
    Jan 30 14:26:53 10.1.2.110      www.cdn.viber.com
    Jan 30 14:26:53 10.1.2.110      aloha.viber.com
    Jan 30 14:26:44 10.1.1.177      shopvac.cyngn.com
    Jan 30 14:26:31 query[AAAA]     dnsmasq[29146]:
    Jan 30 14:26:24 10.1.2.120      blu406-m.hotmail.com
    Jan 30 14:26:15 10.1.1.70       notify10.dropbox.com
    Jan 30 14:26:12 10.1.2.120      dsn11.d.skype.net
    Jan 30 14:26:05 10.1.1.10       ssl.empirehost.me
    Jan 30 14:26:03 exiting         dnsmasq[28816]:
    Jan 30 14:25:59 query[AAAA]     dnsmasq[28034]:
    Jan 30 14:25:38 10.1.2.120      16416.0.1.6.20703.rst11.r.skype.net
    Jan 30 14:25:36 10.1.1.70       ssldev.oes.avira.com
    Jan 30 14:25:29 exiting         dnsmasq[27126]:
    Jan 30 14:25:20 10.1.2.120      16416.0.1.6.20703.rst9.r.skype.net
    Jan 30 14:25:10 10.1.2.120      16416.0.1.6.20703.rst4.r.skype.net
    Jan 30 14:24:43 exiting         dnsmasq[25491]:
    Jan 30 14:24:24 10.1.2.120      pipe.skype.com
    Jan 30 14:24:22 10.1.2.120      a.config.skype.com
    Jan 30 14:24:19 10.1.2.120      dsn2.d.skype.net
    Jan 30 14:24:19 10.1.2.120      conn.skype.com
    Jan 30 14:24:18 reading         dnsmasq[25271]:
    Jan 30 14:24:18 10.1.2.120      b.config.skype.com [+b]
    Jan 30 14:24:14 10.1.1.70       syndication.twitter.com
    Jan 30 14:24:05 10.1.1.177      mqtt-z.facebook.com
    Jan 30 14:23:42 exiting         dnsmasq[24925]:
    Jan 30 14:23:21 exiting         dnsmasq[24283]:
    Jan 30 14:23:03 10.1.2.110      elb-p-dms-dls-ew1-1486943014.eu-west-1.elb.amazonaws.com 
    Jan 30 14:23:00 exiting         dnsmasq[24148]:
    Jan 30 14:22:50 10.1.2.110      e5.whatsapp.net
    Jan 30 14:22:38 exiting         dnsmasq[23899]:
    Jan 30 14:22:18 10.1.2.110      ew1.dms-gld.bigdata.ssp.samsung.com
    Jan 30 14:22:18 exiting         dnsmasq[23571]: 
    notice dnsmasq error being treated as host
     
    Last edited: Jan 30, 2016
  3. WaLLy3K

    WaLLy3K Networkin' Nut Member

    I've seen that too, but can't seem to reproduce it reliably.
     
  4. jerrm

    jerrm Network Guru Member

    Something is matching the regex well enough to count as a hit, but obviously isn't. PM a tail a 1000 or so lines of syslog at the time of the error, and I'll take another look at it. The more samples the better.
     
  5. Michael Malone

    Michael Malone Network Newbie Member

    Is it possible to load jerrm's mod script automatically at startup on a WRT54GL v1.1. 4MB flash. I am running the latest shibby tomato mini and so everything would have to run from /tmp. I must be doing something wrong. NVRAM should be able to hold an install script to get everything downloaded and working in /tmp after a reboot?

    I was able to get ALL-U-NEED Ad Blocking v3.9e working by altering these two settings.

    ADD_CONF=”Y”
    USEHOSTS=”Y”

    Anyone had success?
     
    Last edited: Jan 31, 2016
  6. Michael Malone

    Michael Malone Network Newbie Member

    I got it working ....

    WAN Up
    [TomatoUSB] Admin: Scripts -> http://192.168.1.1/admin-scripts.asp

    # For a custom location uncomment and edit PREFIX value
    # export PREFIX=/opt/bin
    export PREFIX=/tmp/adblock
    wget -O - http://goo.gl/GfA7cQ | sh
    sleep 5
    /tmp/adblock/adblock.sh cron && echo Success!
     
    Last edited: Feb 3, 2016
  7. eelstrebor1

    eelstrebor1 Connected Client Member

    I get this:

    /tmp/mnt/sda1/adblock/pixelserv: line 1: syntax error: unexpected "("


    Since it appears to be a binary file, how am I suppose to fix it?
     
    Last edited: Jan 31, 2016
  8. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

    I have adblock running OK, but when I go to <router IP>/user/adblock.sh, the blocked and resolved hosts lists are empty. My dnsmasq log is perfectly fine.
    Code:
    root@Tomato:/tmp/home/root# nvram show | grep log_
    bt_log_path=/var/log
    log_events=acre,crond,sched
    log_file=1
    log_file_custom=1
    log_file_keep=1
    log_file_path=/tmp/mnt/data/logs/syslog.txt
    log_file_size=500
    log_ftp=0
    log_in=0
    log_limit=60
    log_mark=60
    log_out=0
    log_remote=0
    log_remoteip=
    log_remoteport=514
    log_wm=1
    log_wmdmax=2000
    log_wmip=
    log_wmsmax=2000
    log_wmtype=0
    
    and for my dnsmasq.log I have
    Code:
    log-facility = /tmp/mnt/data/logs/dnsmasq.log
    everything is being logged to an external USB

    Can anyone help me out?
     
    Last edited: Feb 6, 2016
  9. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

    I have a second question also: Since adblock is redirecting everything, then we have to use the router as the only DNS source i.e. we cannot add OpenDNS or Google as backup's otherwise the adblocking just gets bypassed.

    The problem is, in order to enable the vpn client, we have to have a DNS other than the router, otherwise it just blocks everything on my network.

    How do I get around running both adblock and a vpn client at the same time?
     
    Last edited: Feb 6, 2016
  10. WaLLy3K

    WaLLy3K Networkin' Nut Member

    They're two separate beasts, really. You should have already set your "Static DNS" under Basic > Networking, and you can make sure it's being used by selecting "Use Internal DNS" and "Intercept DNS port" under Advanced > DHCP/DNS.

    Now, assuming you're using OpenVPN and have set "Accept DNS configuration" to Exclusive/Strict, Adblock will not protect your browsing experience, as you're using the VPN's DNS servers instead of your routers.

    If you've set "Accept DNS configuration" to Disabled, you should be covered by Adblock. However, you will be open to DNS leaks (if that's a concern to you).
     
  11. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

    So If I understand you correctly, I can set static DNS servers (e.g. 208.67.222.222 for OpenDNS, 8.8.8.8 for Google) under my basic networking page provided I have checked my use internal DNS box under the advanced settings and also checked the intercept DNS port, so when the vpn is open (I assume it therefore has to be UDP on port 53), its forced to use the static DNS settings which are redirected through the vpn tunnel?

    Do I have it right?
     
  12. WaLLy3K

    WaLLy3K Networkin' Nut Member

    Yes I believe that's the case - as long as "Accept DNS configuration" is disabled under OpenVPN (Client) Advanced settings. You shouldn't need to change anything in regards to the UDP port though.
     
  13. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

    How does the whitelist work? I added a site in the whitelist, that comes up in the blocklist. The only way I can get it to work is to manually edit the blocklist, which I have to do every time it updates. There's a forum at www.thebugs.ws that I would like to unblock without having to constantly edit the blocklist before I visit, is there any way around this?
     
  14. jerrm

    jerrm Network Guru Member

  15. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

    I tried this and it doesn't seem to work
    It looks like the only DNS entry has to be that of the router. Removing all static DNS entries uses the router which is correct, but it looks like even with adblock enabled only (no vpn) there is a DNS leak since it has picked up the IP address from somewhere looking at the last 2 lines below.

    Oh and yes, I did close all my browsers and flushed the DNS cache before I ran the test
     
    Last edited: Feb 6, 2016
  16. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

  17. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

    Bumping this up again as I can't seem to get the recently blocked hosts: and recently resolved hosts: output to show up in the gui. Jerrm??
     
    Last edited: Feb 6, 2016
  18. jerrm

    jerrm Network Guru Member

    Remove the spaces in log-facility.
     
  19. Sean Rhodes

    Sean Rhodes Networkin' Nut Member

    Thanks Jerrm, that did it
     
  20. QSxx

    QSxx Network Guru Member

    Is there any way to keep dnslog slim if we use log-facility other than installing opt/entware and going down logrotate way? Or should i just keep dns stuff in my syslog and use it's native rotating?
     
  21. jerrm

    jerrm Network Guru Member

  22. phuklok1

    phuklok1 Network Guru Member

    This may be a silly question, and the answer is probably no, but is it possible to blacklist actual IP addresses from the blacklist file? Or is there an easy way to blacklist a bunch of outbound IP addresses, even ranges? is the only way through iptables in the firewall tab? This might be a nice future enhancement to the script allowing the consolidation of both named and direct address blocking if it cannot already do it. It seems like more and more companies are getting wise to blocking hostnames and are hard-coding IPs. Thanks.
     
  23. jerrm

    jerrm Network Guru Member

    Not in the scope of the adblock script, it relies on dns. @rs232's p2partisan is closer to what you ask.
     
  24. QSxx

    QSxx Network Guru Member

    @jerrm damn - i'm getting old and i'm getting blind - no idea how i missed that one... thanks buddy, it's just what i was looking for
     
  25. meazz1

    meazz1 LI Guru Member

    i am running this adblocking on my Asus RT-AC56U ARM router. Everything is running from an USB flash drive attached to the router (/tmp/mnt/sda1/adblock).
    I have a minor issue with the "dnsmasq.log". It growing.
    I searched this thread but did not find a suitable solution yet. This is what I have in my
    Dnsmasq custom configuration box.
    "log-queries
    log-facility=/tmp/mnt/sda1/adblock/dnsmasq.log"
     
  26. WaLLy3K

    WaLLy3K Networkin' Nut Member

    When you have "log-queries" enabled, that's exactly what it's going to do - log every single website that every single device within your network needs to contact.

    You have three options, depending on what you need logging for:
    • The simplest, is to remove "log-queries" and add it only when you're troubleshooting blocked websites.
    • If you want to keep logging constantly running but you're only in need of the last few days worth of results, you can set a custom schedule to run ">/tmp/mnt/sda1/adblock/dnsmasq.log" once a day/week, which will clear the logfile (It's a bit crude, but it works)
    • Finally, you can rotate dnsmasq logs as mentioned here to keep a longer, more comprehensive list of sites for whatever reason.
     
    Last edited: Feb 12, 2016
  27. Semson

    Semson Network Newbie Member

    that did the trick. thanks @Michael Malone
     
  28. meazz1

    meazz1 LI Guru Member

    If I use this script with correct prefix or path for my setup , how would i run it? Do I run it in the wanup or init box? do I have to manually run it or need to schedule it to run everday?
    # rotate dnsmasq logs
    DNSDIR=/cifs1/adblock
    DNSLOG=${DNSDIR}/dnsmasq.log
    DNSOLD=${DNSDIR}/dnsmasq.old
    echo "Rotating dnsmasq log ${DNSLOG} to ${DNSOLD}..."
    mv -f ${DNSLOG} ${DNSOLD}
    touch ${DNSLOG}
    chmod 777 ${DNSLOG}
    kill -USR2 `cat /var/run/dnsmasq.pid`"
     
  29. GaretJax

    GaretJax Reformed Router Member

    Hello all,

    I believe the adblock is running fine since this is what I get in my log file:

    Feb 12 10:39:17 unknown user.notice root: ADBLOCK: 516649 entries
    Feb 12 10:39:17 unknown user.notice root: ADBLOCK: sorting hosts...
    Feb 12 10:51:56 unknown user.notice root: ADBLOCK: hosts sorted.
    Feb 12 10:51:56 unknown user.notice root: ADBLOCK: 253027 entries
    Feb 12 10:52:05 unknown user.notice root: ADBLOCK: dnsmasq is running

    However, I am still seeing Ad Choice ads on various sites. The worst offender is Yahoo:

    ads0.png ads1.png

    Am I missing something?
     
  30. Michael Malone

    Michael Malone Network Newbie Member

    I don't know about Ad Choice, but I know you still have to combine ublock origin along with the adblock in the router. I use (uMatrix + uBlock) + Jerrm's Script in the router. uMatrix has a learning curve.
     
  31. WaLLy3K

    WaLLy3K Networkin' Nut Member

    You could look into more blocklists. Here's a few lists I've added:

    Just beware that the more lists you add, the longer it takes for lists to compile (If on default Optimize mode) and increases the potential of browsing slow-downs on older routers.

    Since using other peoples lists aren't going to be perfect, you can always add your own by right-clicking the ad in question and grab the URL.
     
  32. Tordenflesk

    Tordenflesk LI Guru Member

    I get this:
    Code:
    ADBLOCK[6134]: Download starting
    Connecting to raw.githubusercontent.com (23.235.43.133:443)
    wget: error getting response: Connection reset by peer
    ADBLOCK[6134]: Failed: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
    ADBLOCK[6134]: No source files found
    ADBLOCK[6134]: Exiting /tmp/mnt/USB/AdBlock/adblock.sh 3
    When trying to use "StevenBlack's Amalgamated hosts file"
     
  33. jerrm

    jerrm Network Guru Member

    HTTPS urls are not supported by busybox wget on MIPS due to openssl lacking s_client support on MIPS.

    Installing wget (or openssl) from entware/optware/tomatoware should get wget working, but the timestamp checks for the file will not work. No plans to implement https support at this time.
     
    koitsu likes this.
  34. Michael Malone

    Michael Malone Network Newbie Member

    I am running everything from /tmp although I have some space available in /jffs but don't have USB. Is it possible to get Wget with SSL working from /tmp? I must first install optware, but without USB I'm not sure if that's possible. I am only interested because of host blocklists on Github which require https to download.

    https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
     
    Last edited: Feb 13, 2016
  35. jerrm

    jerrm Network Guru Member

    You'll need to find a statically compiled wget with ssl support. I think @lancethepants has one at files.lancethepants.com.
     
  36. HunterZ

    HunterZ Network Guru Member

    Noticed today that windows7themes.net has an assholish policy of redirecting to http://windows7themes.net/stopadblockplus/ when I visit with ad blocking enabled. I don't think there's a good way for this ad blocking solution to provide workarounds for sites like that.
     
  37. WaLLy3K

    WaLLy3K Networkin' Nut Member

    I honestly can't believe GitHub doesn't support this. It means I need to mirror something like 10 lists on my web server and update it weekly if I don't want to waste bandwidth fetching the same content over and over again...
     
  38. jerrm

    jerrm Network Guru Member

    I'll likely get around to supporting s_client for https timestamp checks under ARM, but it's so far down on my list of priorities I wouldn't want to hazard a guess as to when.

    Unfortunately it wouldn't help for github downloads as they don't pass a Last-Modified header.
     
  39. koitsu

    koitsu Network Guru Member

    Chiming in, re: GitHub lacking Last-Modified headers: this plagues us in the FreeBSD project (specifically Ports) as well. However, we do have a methodology that works:

    Code:
    root@icarus:/usr/ports/sysutils/bsdhwmon # make fetch
    ===>  License BSD2CLAUSE accepted by the user
    ===>  Found saved configuration for bsdhwmon-20150429
    ===>   bsdhwmon-20151206 depends on file: /usr/local/sbin/pkg - found
    => koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
    => Attempting to fetch https://codeload.github.com/koitsu/bsdhwmon/tar.gz/9aec193?dummy=/koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz
    fetch: https://codeload.github.com/koitsu/bsdhwmon/tar.gz/9aec193?dummy=/koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz: size unknown
    fetch: https://codeload.github.com/koitsu/bsdhwmon/tar.gz/9aec193?dummy=/koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz: size of remote file is not known
    koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz             20 kB  258 kBps 00m00s
    ===> Fetching all distfiles required by bsdhwmon-20151206 for building
    root@icarus:/usr/ports/sysutils/bsdhwmon # make fetch
    ===>  License BSD2CLAUSE accepted by the user
    ===>  Found saved configuration for bsdhwmon-20150429
    ===>   bsdhwmon-20151206 depends on file: /usr/local/sbin/pkg - found
    ===> Fetching all distfiles required by bsdhwmon-20151206 for building
    root@icarus:/usr/ports/sysutils/bsdhwmon #
    
    You can see here that the initial fetch of
    https://codeload.github.com/koitsu/...=/koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz downloaded content (with fetch(1) (that's a FreeBSD wget-like utility)), but upon the 2nd time being asked to, didn't -- meaning, the locally cached copy of the tarball matched the remote end. So how'd we do it, given that there's no Last-Modified: header for that content -- and here's proof:

    Code:
    $ curl -v 'https://codeload.github.com/koitsu/bsdhwmon/tar.gz/9aec193?dummy=/koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz' 2>&1 | less
    ...
    > GET /koitsu/bsdhwmon/tar.gz/9aec193?dummy=/koitsu-bsdhwmon-20151206-9aec193_GH0.tar.gz HTTP/1.1
    > Host: codeload.github.com
    > User-Agent: curl/7.47.0
    > Accept: */*
    ...
    < HTTP/1.1 200 OK
    < Content-Length: 21149
    < Access-Control-Allow-Origin: https://render.githubusercontent.com
    < Content-Security-Policy: default-src 'none'
    < X-XSS-Protection: 1; mode=block
    < X-Frame-Options: deny
    < X-Content-Type-Options: nosniff
    < Strict-Transport-Security: max-age=31536000
    < Vary: Authorization,Accept-Encoding
    < ETag: "9aec19339040e771ad90953a4181452b342533ca"
    < Content-Type: application/x-gzip
    < Content-Disposition: attachment; filename=bsdhwmon-9aec193.tar.gz
    < Date: Mon, 15 Feb 2016 10:11:24 GMT
    < X-GitHub-Request-Id: 45B58ED5:12C9B:42F364:56C1A44C
    ...
    
    Note the Content-Length: header returned by GitHub servers. fetch(1) supports a flag called -S, which supports a size/value (in bytes) that matches against the Content-Length: header. But in this mode, instead of doing a GET request, it does a HEAD request first (identical to a GET but no actual content/payload is downloaded, only headers). If the size given to -S matches what's returned in the HEAD request, it doesn't need to download anything.

    Normally Content-Length: alone isn't sufficient -- someone could indeed release content that generates a tarball that ends up being, for example, 21149 bytes long, which would defeat the mechanism (erroneously not downloading content), right? Yup, but in this case it works because of the use of GitHub hash tags (note the ETag header -- this is a SHA1). So the next time someone updates the port to refer to a new release of the software, that value (in the Makefile) has to be changed, hence we have uniqueness combined with Content-Length: analysis.

    Busybox wget doesn't support HEAD requests (although it's "spider" mode with -s might do something like this internally, I simply haven't tested it) of this nature, but the non-Busybox wget does (through use of two flags combined: -S --spider) but it doesn't look like it natively supports something similar to fetch(1)'s -S flag. You could implement this capability in shell (comparison of the file size vs. Content-Length via HEAD), but only if wget -S --spider can actually return headers (e.g. wget -S --spider | awk '/^Content-Length: / { print $2 }').

    Food for thought.
     
    jerrm and visceralpsyche like this.
  40. jerrm

    jerrm Network Guru Member

    Thanks, I'll keep it in mind if I ever get to it.

    Busybox wget doesn't support the needed options. What the script does now is build a HEAD request and uses busybox netcat(nc) to send it to the server to grab the Last-Modified header. Theoretically, for https urls(not yet implemented), it would use openssl s_client instead of nc to process the HEAD request and have access to whatever response headers were needed, including ETag.
     
    Last edited: Feb 16, 2016
    visceralpsyche likes this.
  41. Michael Malone

    Michael Malone Network Newbie Member

    This is how I managed to get the remote url whitelist / blacklist working. Anyway they appear to be included properly and I don't have to force the update. The example below shows the position where I added the lines to adblock.sh

    adblock.sh

    } > "$tmpwhitelist"

    # Apprend Whitelist to Whitelist
    wget http://pastebin.com/raw.php?i=XXXXXXXX -O - >> "$tmpwhitelist"

    } > "$tmpblocklist"

    # Apprend Blacklist to Blocklist
    wget http://pastebin.com/raw.php?i=XXXXXXXX -O - >> "$tmpblocklist"
     
  42. Link2User

    Link2User Networkin' Nut Member

    I'm using adblock and p2ppartisn wondering how can i specify a custom dns to use based on the url accessed?
     
  43. crhiles

    crhiles Network Guru Member

    In DNSmasq Custom Configuration
    server=/url.com/DNS server
     
  44. IngoPan

    IngoPan Serious Server Member

    Hallo,

    I am a bit confused here. I was originally looking for a blocklist for germany, but then found out, that "adaway" on my tablet successfully blocks german ad networks with the default settings.
    I then visited www.movie-blog.org and noticed, displayad.layer-ad.org isn´t blocked (i would like to block the whole layer-ad.org domain). I edited the backlist file and it looks like this:

    Code:
    layer-ad.org
    What am i doing wrong please? Thanks in advance.

    @EDIT: a nslookup from my system says:

    Code:
    C:\WINDOWS\system32>nslookup layer-ad.org
    Server:  r7000
    Address:  192.168.2.1
    
    Name:    layer-ad.org
    Address:  192.168.2.254
    
     
    Last edited: Feb 27, 2016
  45. ksuuklan

    ksuuklan New Member Member

    Hi,

    Just installed this script, running Tomato Firmware 1.28.0000 MIPSR1-132 K26 Mini on:

    Model Buffalo WHR-HP-G54
    Chipset Broadcom BCM5352 chip rev 0 pkg 2
    CPU Freq 200MHz
    Flash Size 4MB

    But after some 5 minutes or so, load goes high and router crashes, can't log in into router via cli or gui, only manual pover off/on helps for next 5 minutes.

    Adblock loging isn't enabled. How to reduce high load? Maybe I should disable pixelserv and some sources, if so, how to automatically disable them after every router reboot as reboot downloads the script again with default options?
     
  46. Nathaniel Cowles

    Nathaniel Cowles Networkin' Nut Member

    Likewise I am getting the same result whitelisting wordswithfriends.zyngawithfriends.com, it continues to block it after Update.

    root@mainRouter:/tmp/home/root# nslookup wordswithfriends.zyngawithfriends.com
    Server: 127.0.0.1
    Address 1: 127.0.0.1 localhost

    Name: wordswithfriends.zyngawithfriends.com
    Address 1: 192.168.1.254

    Thanks.
     
  47. jerrm

    jerrm Network Guru Member

    A very underpowered router with only 16MB ram and no USB? You'll need to use only the smallest lists to have a prayer. Might get further if using a CIFS mount for script/list storage, but even then it would be doubtful.
     
  48. jerrm

    jerrm Network Guru Member

    http://www.linksysinfo.org/index.ph...and-mean-adblocking.68464/page-10#post-251222
     
  49. jerrm

    jerrm Network Guru Member

    dnsmasq is redirecting.

    Look at the logs and see which domains/hosts are actually being allowed or denied. Also make sure you are using optimize or legacy mode if you want the whole domain blocked.
     
  50. ksuuklan

    ksuuklan New Member Member

    What is the smallest list? And still, how to adjust thing ing config and keep them after reboot? One solution is to save modified script into my own webserver, but this seems kind of silly, so more options into script would be good idea, like:

    export PREFIX=/tmp/adblock
    wget -O - http://goo.gl/GfA7cQ | sh options=pixelserv=no, use sources=1-3, etc
    sleep 5
    /tmp/adblock/adblock.sh cron &&
     
  51. HunterZ

    HunterZ Network Guru Member

    You're supposed to put the config file where changes can be saved. This could be JFFS or USB on the router, or a CIFS mount of a share on the LAN.

    Re-downloading the script on every boot is also a bit fragile.
     
  52. jerrm

    jerrm Network Guru Member

    My version has never been targeted toward routers without some sort of persistent storage. Zero plans to add support for such a config. It is simple enough to create a minimal config file from Tomato's init script.
     
  53. rotorbudd

    rotorbudd Addicted to LI Member

    I've suddenly started having a problem with the adblock.sh file
    Everytime I try to run it, either on reboot or in the system commands page I get the following:
    /mnt/USB1/adblock/adblock.sh: config: line 27: syntax error: unexpected word (expecting ")")


    I've tried replacing the file with a freshly downloaded version, same problem
    .

    line 27 is the same in both files:

    stop() {
    It had been working for several months with absolutely no problems.
    Thanks for any help.
     
  54. rotorbudd

    rotorbudd Addicted to LI Member

    Never mind! I should read my own posts!
    I found the problem in the config file just like the error says.
    Somehow a strange character got added to line 17 in the config file
    Could having an old thumb drive as USB1 be a problem?
     
  55. jerrm

    jerrm Network Guru Member

    I hate the fact the config file is read in as executable source, but haarp's original used the method, for valid space saving reasons when nvram storage was still a concern.

    I haven't changed it to maintain compatibility since some use it as a method of injecting customizing code, but may cut the cord someday.
     
  56. Nathaniel Cowles

    Nathaniel Cowles Networkin' Nut Member

  57. ksuuklan

    ksuuklan New Member Member

    Then the whole script should be one file, like it was in Your first post and so I can put it into WAN Up window, as now I can't permanently add options.
     
  58. jerrm

    jerrm Network Guru Member

    Not sure what you're talking about. My version has ALWAYS required a config file. That will never change. It will not be tweaked for an NVRAM only approach.

    Creating the config file from init or wanup only requires a single echo statement.
     
  59. ambiance

    ambiance Networkin' Nut Member

    Has anyone had any issues since upgrading to 133? The script works, but I get nslookup: can't resolve '(null)' all over the place at the top.
     
  60. reimer

    reimer Addicted to LI Member

    I don't know if it's related but v133 changelog does state "Attention: You have to erase nvram after upgrade!!". Perhaps that is the issue.

    The script is working great for me after upgrading to 133
     
  61. ambiance

    ambiance Networkin' Nut Member

    Did it during upgrade and just now a thorough with the same result.
     
  62. ambiance

    ambiance Networkin' Nut Member

    I downgraded to 132 and had no problems, so I tried going back to 133 and nslookup returns. Ugh!
     
  63. Mercjoe

    Mercjoe Network Guru Member


    Just checked and I am having the same issue.
     
  64. my_bey

    my_bey Serious Server Member

    Same issue at my end with v133.
     
  65. ambiance

    ambiance Networkin' Nut Member

    Good to know
     
  66. Michael Malone

    Michael Malone Network Newbie Member

    I'm using 133, how can I check if there is a problem?
    Tomato Firmware 1.28.0000 MIPSR2-133 K26 Max

    Found out my remote blacklist, whitelist haven't been integrating properly. Still work fine from .ini but not what I wanted. Anyone have that working, I have pastebin Urls, how can I add them to the script?
     
  67. sp83tr

    sp83tr New Member Member

    I am running the 132 version optimized for NT-N66U and have also seen nslookup.

    For example, here is one that currently shows on my Adblock page:

    nslookup: can't resolve 'forwarded'

    which appears to be related to the following in the page's HTML source:

    span title="forwarded" class="line" data-hostname="dnsmasq[10267]:">Feb 28 22:13:28 forwarded dnsmasq[10267]: <span class="add" >[+b]</span></span>
     
  68. IngoPan

    IngoPan Serious Server Member

    Does someone know if this script works on Asus-WRT / Merlin / XVRT,too ?
     
  69. Weltherrscher

    Weltherrscher Reformed Router Member

    for the nslookup errors:
    are you using the standard busybox nslookup?
    I had these errors too, but they went gone after i installed bind-nslookup from optware.
     
  70. my_bey

    my_bey Serious Server Member

    Hi

    Can you provide step-by-step installation procedure of "bind-nslookup"? I installed v134, it still has nslookup errors.
    Thanks,
     
  71. jerrm

    jerrm Network Guru Member

  72. ambiance

    ambiance Networkin' Nut Member

    root@R7000:/tmp/home/root# nslookup www.google.com && echo NO ERROR || echo ERROR
    nslookup: can't resolve '(null)'

    Name: www.google.com
    Address 1: 2607:f8b0:400b:806::2004 yyz08s09-in-x04.1e100.net
    Address 2: 184.150.183.44
    Address 3: 184.150.183.55
    Address 4: 184.150.183.54
    Address 5: 184.150.183.29
    Address 6: 184.150.183.50
    Address 7: 184.150.183.20
    Address 8: 184.150.183.49
    Address 9: 184.150.183.59
    Address 10: 184.150.183.35
    Address 11: 184.150.183.30
    Address 12: 184.150.183.40
    Address 13: 184.150.183.39
    Address 14: 184.150.183.24
    Address 15: 184.150.183.25
    Address 16: 184.150.183.45
    Address 17: 184.150.183.34
    NO ERROR
     
  73. jerrm

    jerrm Network Guru Member

    So definitely a problem with 133/134. Is this a vanilla dns config, or is dnssec or dnscrypt enabled?
     
  74. ambiance

    ambiance Networkin' Nut Member

    Vanilla.
     
  75. AndreDVJ

    AndreDVJ LI Guru Member

    I have this problem when I compile Tomato with a "newer" toolchain for my R7000. (the one with support for pthread, etc)
    Code:
    root@R7000:/tmp/home/root# /usr/bin/nslookup www.google.com
    nslookup: can't resolve '(null)'
    
    Name:      www.google.com
    Address 1: 2607:f8b0:4008:80a::2004 mia07s34-in-x04.1e100.net
    Address 2: 216.58.192.68 mia07s34-in-f4.1e100.net
    
    Installing bind-nslookup from Entware solves this issue for me at the moment.
    Code:
    root@R7000:/tmp/home/root# /opt/bin/nslookup www.google.com
    Server:         127.0.0.1
    Address:        127.0.0.1#53
    
    Non-authoritative answer:
    Name:   www.google.com
    Address: 216.58.192.68
    
    I can't blame 133/134 for many reasons. I built stuff from 132 plus cherry-picking stuff from 133/134 and a different toolchain. I saw on the repo that Shibby tried to use that toolchain and reverted because of compiling issues (I figured out one for php, by install libmagic-dev), but 133/134 have so many kernel changes to support Multi-WAN that may have broken these as well. Maybe someone can chime in and let us know what exactly broke.

    In my case, using the normal toolchain will make nslookup work just fine.

    One option is to update busybox. I have tried upgrading busybox myself but no success so far. I need to figure out how the Entware folks did to compile this software.

    For now, install bind-nslookup from Entware.

    Sorry for hijacking this thread, but that's the currently what I am seeing at the moment. I did not want to bring toolchains into the conversation, but if anyone else is compiling firmwares on the wild, please let me know your findings.
     
    koitsu likes this.
  76. koitsu

    koitsu Network Guru Member

    Random comment in passing, re: nslookup (null) issue: at first glance this looks like an ABI compatibility problem. Tracking this down should be possible with binaries (and libraries) built with -g3 and/or -ggdb, and use of gdb + strace.
     
  77. jerrm

    jerrm Network Guru Member

    Whatever the underlying reason, someone running 134 needs to raise the issue with @shibby20. I don't plan to jump on the multi-wan firmware for at least another two or three releases.
     
  78. lancethepants

    lancethepants Network Guru Member

    Has shibby built the new versions with this "new" toolchain? Someone in his release thread mentioned the same nslookup issue there. I've seen that we got the idea for "new" toolchains from the RMerlin forum folks trying to use a newer uclibc version.

    My hope is that shibby does not use the "new" toolchain. There are a bunch of closed-sourced binaries with dependencies on the libraries they were built against. Simply ripping out the libc library with its companion libraries and replacing them with new versions is the most hacksaw job I can imagine. Not to mention that uclibc does not guarantee any ABI stability between releases, aka, everything should be recompiled, which is not possible and never will be.

    They experienced several breakages on the RMerlin side of things with closed source portions of the firmware. Then they want to patchelf the broken parts to go back and use the original libraries. I think we'll run into so many stupid little breakages exactly like this that will leave us guessing whether it was the new toolchain, or some new feature that broke things.

    You can get away with this in a glibc world, but uclibc was not designed for this, quite the opposite actually. uclibc could care less for backward-compatibility.
     
  79. AndreDVJ

    AndreDVJ LI Guru Member

    Just compiled Shibby's 134. No issues at all with nslookup.
    Code:
    root@R7000:/tmp/home/root# /usr/bin/nslookup www.yahoo.com
    Server:    127.0.0.1
    Address 1: 127.0.0.1 localhost
    
    Name:      www.yahoo.com
    Address 1: 2001:4998:58:c02::a9 ir1.fp.vip.bf1.yahoo.com
    Address 2: 200.152.175.146 r1.ycpi.vip.br1.yahoo.net
    root@R7000:/tmp/home/root# /usr/bin/nslookup www.google.com
    Server:    127.0.0.1
    Address 1: 127.0.0.1 localhost
    
    Name:      www.google.com
    Address 1: 2607:f8b0:4008:80a::2004 mia07s34-in-x04.1e100.net
    Address 2: 216.58.192.68 mia07s34-in-f4.1e100.net
    
    If someone wants my firmware for R7000, please let me know.
     
    my_bey likes this.
  80. my_bey

    my_bey Serious Server Member

    Yes, I am interested. I have R7000. Where can I download it from?
     
  81. Michael Malone

    Michael Malone Network Newbie Member

    I noticed my blocklist is not getting sorted alphabetically, any ideas on how to correct this?

    ex.

    # adblock blocklist, MODE=OPTIMIZE, IP=192.168.1.254, generated Tue Mar 8 13:55:54 CST 2016
    address=/theoads.com./192.168.1.254
    address=/127.0.0.1/192.168.1.254
    address=/195.70.38.70/192.168.1.254
    address=/TRIANGLESERVICESLTD.COM/192.168.1.254
    address=/js.cdn.ac/192.168.1.254
    address=/link.ac/192.168.1.254
    address=/content.ad/192.168.1.254
    address=/id.google.ad/192.168.1.254
    address=/files.native.ad/192.168.1.254
    address=/ads.7days.ae/192.168.1.254
    address=/ads.emirates.net.ae/192.168.1.254
    address=/angelphotos.dr.ag/192.168.1.254
    address=/c.supert.ag/192.168.1.254
    address=/end.ai/192.168.1.254
    address=/native.ai/192.168.1.254
    address=/waa.ai/192.168.1.254
    address=/smschat.alfabeta.al/192.168.1.254
    address=/get.al/192.168.1.254
    address=/casga.sogesca.al/192.168.1.254
     
    Last edited: Mar 9, 2016
    ambiance likes this.
  82. ambiance

    ambiance Networkin' Nut Member

    My blocklist is doing the same. Fortunately all the problems this new release is causing is cosmetic.
     
  83. shibby20

    shibby20 Network Guru Member

    yes, i used new toolchain to compile v133 and v134. I will re-compile ARM`s images using old toolchain.
     
  84. theoctavist

    theoctavist Reformed Router Member

    I am having major problems after each install of adblock and pixlserv. i get "success" messages, but immediately thereafter, i lose all internet connections. =

    15:00:02 unknown daemon.crit dnsmasq[1660]: FAILED to start up
     
  85. jerrm

    jerrm Network Guru Member

    Something is invalid in the dnsmasq config. Post the contents of the adblock config and GUI dnsmasq custom config,
     
  86. theoctavist

    theoctavist Reformed Router Member

    here is the adblock
     

    Attached Files:

  87. theoctavist

    theoctavist Reformed Router Member

    oh, im using JFFS, so i need to edit that CONFIG file for fewer entries, yes? could that be the error source?
     
  88. jerrm

    jerrm Network Guru Member

    Nothing in the posted config file should have an impact. What is in the GUI dnsmasq custom config text box? What version of Tomato?
     
  89. theoctavist

    theoctavist Reformed Router Member

    Tomato Firmware 1.28.0000 MIPSR2-3.1-132 K26 USB Max

    nothing in the custom config box as of yet, (the OP did not list that in the script instructions, and as I am new to the whole thing I was unaware as to what to put there). suggestions? I did see the "SED WRITE ERROR" earlier. (I reset the router because the script broke the connection)
     
  90. Michael Malone

    Michael Malone Network Newbie Member

    adblock.sh

    I actually fixed the blocklist sorting issue by adding " | sort" to the end of this line.

    sed -e :a -e 's/\([^#]*\)#\([^#]*\)/\2\.\1/;ta' -e "s/\(.*\)/address=\/\1\/$redirip/" | sort

    I also have both my remote URL whitelist and blacklist downloading properly and showing correctly in Adblock GUI, without the downloaded copies being removed (so I can examine them) but they aren't being integrated on boot. At least not my whitelist, to be sure. Domain appearing in host source blocklist remains blocked, but Force might work. I don't want to hit force to integrate them, my router is very slow. Can anyone fix that?

    # Apprend Whitelist @ ~ line 714
    wget http://pastebin.com/raw.php?i=XXXXXX -O - >> "$whitelist"

    # Apprend Blacklist @ ~ line 734
    wget http://pastebin.com/raw.php?i=XXXXXX -O - >> "$blacklist"

    # Removal of Sources to Save Space @ ~ line 765 using && rm -f command
    rm -f "$tmpblocklist" &>/dev/null && rm -f $prefix/source-* &> /dev/null
     
  91. jerrm

    jerrm Network Guru Member

    Sorting is not necessary. I didn't see any point in wasting the cycles(still don't). I can't remember the last time I looked at the generated blocklist.

    Unless your line numbers have shifted, you are appending the downloaded white/blacklist files after the on disk files have been processed. Your downloaded lines are completely ignored. Probably better to download the files before starting adblock.

    Deleting the source files effectively causes the equivalent of a "force" every time adblock is run.
     
  92. jerrm

    jerrm Network Guru Member

    Nothing needs to go in the dnsmasq custom config, but the most common reason for dnsmasq not starting up is something is wrong there. Are you sure the script folder is writable and there is enough space to accommodate the list creation?
     
    Last edited: Mar 10, 2016
  93. Michael Malone

    Michael Malone Network Newbie Member

    thank you for the reply jerrm

    I removed the downloading of the Whitelist and Blacklist from adblock.sh and have instead added them to adblock-install.sh. I have done this before, but I end up with exactly the same result. The files are correctly downloaded, but are not properly integrated; at least not the Whitelist, as the whitelisted domain remains in the generated blocklist and my first blacklist domain "t.reallifecam.com" always appears combined with adblock.is.loaded. Does the first line of the blocked domain list need to be a blank line, because I think the rest are ok?

    address=/t.reallifecam.comadblock.is.loaded/192.168.1.254

    adblock-install.sh

    # Apprend Whitelist @ ~ line 74
    wget http://pastebin.com/raw.php?i=XXXXXXXX -O - >> "$bin/whitelist" || abort "error downloading Whitelist"

    # Apprend Blacklist @ ~ line 75
    wget http://pastebin.com/raw.php?i=XXXXXXXX -O - >> "$bin/blacklist" || abort "error downloading Blacklist"
     
  94. jerrm

    jerrm Network Guru Member

    pastebin based downloads strip trailing line feeds. Make the last line of the pastebin file a comment "#"
     
    Michael Malone likes this.
  95. Michael Malone

    Michael Malone Network Newbie Member

    thank you, the "#" fixed the adblock.is.loaded issue with the blacklist. The first domain appears strangely, but seems to be working. I used finder.cox.net to test the remotely retrieved whitelist which still isn't working and the blocklist still shows that it's blocked. "address=/finder.cox.net/192.168.1.254"

    address=/t.reallifecam.com
    /192.168.1.254
     
    Last edited: Mar 10, 2016
  96. theoctavist

    theoctavist Reformed Router Member

    I think , due to jffs, there indeed is not, sir. thank you
     
  97. jerrm

    jerrm Network Guru Member

    Try setting RAMLIST="1" in the config.
     
  98. Michael Malone

    Michael Malone Network Newbie Member

    The remote whitelist doesn't get detected at all, no matter what I do.
    It only works from adblock.ini

    ### Whitelist sites from blocking ###
    ## (add hostnames inside the quotes, space-separated, without http://) ##
    WHITELIST=""
     
  99. WaLLy3K

    WaLLy3K Networkin' Nut Member

    From what I recall of the code, sorting was only necessary because the command was being used to remove duplicates. As long as something is handling the duplicates, the actual sort order makes no difference (Unless you have OCD and regularly access the file manually to see it!)
     
  100. jerrm

    jerrm Network Guru Member

    The list actually is sorted, jut not the way he wants.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice