Script: Adblock - not so lean


Thanks jerrm :) Any opinions on which is best?

If pixelserv returns an error does that mean scripts can detect this and display a warning or attempt to display something else?

BTW - love your adblock script :)
 
Another thing :)

I figured out the reason for the error when saving. I checked the console and saw:

line 330: not in a function

I checked the source of adblockweb.sh and at line 301 found:

local file=""

I changed that to:

file=""

and the error went away. Seems strange? Surely others would have received this?
 
AtTheAsylum said:
Thanks jerrm :) Any opinions on which is best?

If pixelserv returns an error does that mean scripts can detect this and display a warning or attempt to display something else?
Click to expand...

I have no interest in importing a ca cert into my clients, so I use pixelserv. For myself, the appeal of router based adblock is to not touch the client. I'd rather have the less than ideal output than muck with clients. To each his own.

The "error" returned is aborting the tls certificate negotiation, to my knowledge that part of the process in the bowels of the browser and not directly exposed to javascript (or other script engine) via normal page script, it's just a failed connection. Maybe via one of the browser add-in interfaces.
 
AtTheAsylum said:
Another thing :)

I figured out the reason for the error when saving. I checked the console and saw:

line 330: not in a function

I checked the source of adblockweb.sh and at line 301 found:

local file=""

I changed that to:

file=""

and the error went away. Seems strange? Surely others would have received this?
Click to expand...
This was discussed earlier and only impacts the most recent builds (shibby 138 similarly recent Toastman). The code in question started as a function and was moved inline. Earlier versions of busbox shell did not (but should have) thrown an error and it was never caught.

I really need to fix the posted script, but I doubt I will ever move beyond 132 with Shibby, so it hasn't been a on my personal radar.
 
Why does the adblock webpage tell me that the configuration has changed when I haven't made any changes?
 
Thank you for all your hard work, jerm. I've been a user of your awesome script since you first posted on the previous thread. The community appreciates you.

Just wanted to post my config/tweaks to the process. Made config/param changes to write everything to /tmp/adblock. It saves writes to the Router's flash and RAM is so much faster than JFFS. List (re)generation and even browsing latency is improved.

The following script installs entware (for https-capable wget), downloads the script (with my params), config, white/black lists, and pixelserv to /tmp/adblock and kicks off the process with a cron job.

I throw it in the Administration -> Scripts -> Firewall section.

Enjoy and thanks for making this easy to accomplish!

NOTE: I'm pretty sure the version I have on pastebin is 2015-11-11. Not 100%. And I recommend you do this on a router with enough RAM. >=128MB is safe. 64MB might be enough too if you modify to use less lists.
 

Attachments

  • install-jerm-adblock-ram-rootmbx.sh.txt
    2.5 KB · Views: 72
Busybox wget is HTTPS-capable (it actually runs "openssl s_client" as a middle-man for the encryption/decryption), with better support for servers that use or require TLS SNI since Busybox 1.25.0 (I know because I'm the patch author (further reference/details)). So as long as you're using a recent TomatoUSB firmware, you shouldn't need Entware-ng for wget, unless there's a GNU wget feature/flag you're relying on (and if so, could I know what it is could I could implement it in Busybox wget?)
 
koitsu said:
Busybox wget is HTTPS-capable (it actually runs "openssl s_client" as a middle-man for the encryption/decryption), with better support for servers that use or require TLS SNI since Busybox 1.25.0 (I know because I'm the patch author (further reference/details)). So as long as you're using a recent TomatoUSB firmware, you shouldn't need Entware-ng for wget, unless there's a GNU wget feature/flag you're relying on (and if so, could I know what it is could I could implement it in Busybox wget?)
Click to expand...

Wrote the script long ago, before that patch was applied. I'll probably cut that out and update jerm's at some point. Thanks for the info!

What's the word on the new built-in Adblocker in shibby's GUI or even AB Solution project (Merlin only) I just came across? Worth switching over to either?
 
Oops, sorry -- while dealing with unrelated stuff, I realised I got the version number wrong. It got introduced into official Busybox as of 1.26.0. Tomato MIPS uses 1.25.0 with my wget patch applied, so it works there (commit reference for Toastman, and I can confirm it does). ARM, on the other hand, does not have it (ARM uses Busybox 1.23.2 -- don't ask, I'm in a PM with Toastman about all that, there is chaos).
 
rootMBX said:
Wrote the script long ago, before that patch was applied. I'll probably cut that out and update jerm's at some point. Thanks for the info!

What's the word on the new built-in Adblocker in shibby's GUI or even AB Solution project (Merlin only) I just came across? Worth switching over to either?
Click to expand...
I used Shibby's ad block for a while, it does it's thing very well. The debugging is not as good as this UI, but I never had any big issues without a fancy debugger. When I had something not resolving, I simply assumed the adblocker was causing the name to not resolve, and added whatever dns name wasn't working to the whitelist. I never had any issues doing that. I can see a lot of situations that might really require a lot of debugging, but I never ran into a situation that needed it. It also doesn't have that fancy pixel server, but that wasn't any big thing.

The truth is, that I just now moved away from Shibby builds on the R7000 to Toastman due to too many bugs with basic functionality. I very much agree with Toastman's philosophy, and his focus on core router functionality shows in his build for at least the R7000. From my experience, I think that maybe Shibby might work work well on MIPS hardware while Toastman works better on ARM. I'm not entirely sure that is always the case, but I do think that the method of blocking ads is much less important than most of the other features. Shibby's version of ad blocking worked very well, was very simple, and problem free. But I'm definitely not going to use it simply for that feature.
 
I will say that when moving from Shibby V138 to Toastman V9008.6 for the R7000, I was a little bummed about the lack of adblocking in Toastmans's build. Toastman had too many basic features fixed that were broken in Shibby's build for the R7000; such as bugs in VLAN configurations, dyndns local WAN ip address resolution (I think also wanup triggering), and nvram issues with backup/restore functions. This adblocking feature was the only thing I really missed from the Shibby build. So I googled for scripts... and found this!

While this adblock system is a huge hack, I feel like it is an upgrade, not a downgrade from what I gave up in Shibby's build. Probably the larger the hack, the more difficult it is for the end user to maintain; but this was put together amazingly well. I'm more than happy with my Toastman setup and this adblock; I'm ecstatic.

Thank you jerrm for this amazing system. I've never seen anything like it.
 
Recently I've noticed a lot of garbled text at the top and it seems to happen while watching Netflix on my PS3. I'm using the latest version of your script on a flash drive on a R7000 using the latest version of Toastman.

Here's a sample:

recently resolved hosts:
�eb 11 10:34:26 query[A] dns}asu[31743]:
Veb 11 14:02:24 192.168.1.15 ichnaea.netflix.com
F�b 11�10:34:01 art-2.nflximg.n�t 192�168.1.15
F�b 11 1s:=3:22 query[A] d�soasq[14826]:
Fmb 11 10:34:1x DHCPACK(br0) dnsmasq-dhcp[31743]:
Fe� 11 17:01:10 qumry[A] dns}asq[19905]:
Fe� 11 97:12:08 from appboot.netflix.com
Fej 11 14:02:46 192.168.1n15Fmb tt�aka�.�flximg.com
Fej 11 14:00:35 192.178.1.15 a�t-2.nflximg.net
Fej 11 10:44:09 query[A] dlsmasq[31743]:
Fef 11 16:24:�7 query[A] dnsmasq[29390]:
Fed 11 16:34:59 192.168.1.4 54-149-26-14�-push.np.communication.playstation.net
1.3" class="line" data-hostname="ichnama.netglix/com">Fec 11 14:00:22 192.168>1.3 ichnama.netglix/com
Feb�11 17:01:27 api.steampow�red.com 192.168�1.10
Feb011 17:02:08 secure.netflix.com 192.168.1.5
Feb!11 1�:46:17 dnsmisq[31743]: apm-global.netflix.com query[A]
8.1.3Feb" class="line" data-hostname="api-global.netflix.co�">Feb 51 14:11:40 192.1>8.1.3Feb api-global.netflix.co�
Feb 1� 10:33:23 192.168.1.15 �rt-0.nflximg.net
Feb 1q 17:00:15 192.168.1.10 api.stecmpoweret.com
Feb 15 17:00:31 192.168.1.90 telk.google.com
Feb 11�16:21:55 appboot.netflkx.com 192.168.1.3
Feb 11`13:53:13 pollservmr.lastpass.com 192.16<.1.2
Feb 11 q0:47:18 192.368.1.1� api=global.nut&lkx.com

nslookup: can't resolve 'query[A]' BusyBox v1.23.2 (2017-01-21 01:37:51 ICT) multi-call binary. Usage: nslookup [HOST] [SERVER] Query the nameserver for the IP address of the given HOST optionally using a specified DNS server sh: syntax error: unexpected "(" nslookup: can't resolve 'qumry[A]' nslookup: can't resolve 'from' nslookup: can't resolve '192.168.1n15Fmb' nslookup: can't resolve '192.168' BusyBox v1.23.2 (2017-01-21 01:37:51 ICT) multi-call binary. Usage: nslookup [HOST] [SERVER] Query the nameserver for the IP address of the given HOST optionally using a specified DNS server BusyBox v1.23.2 (2017-01-21 01:37:51 ICT) multi-call binary. Usage: nslookup [HOST] [SERVER] Query the nameserver for the IP address of the given HOST optionally using a specified DNS server nslookup: can't resolve 'apm-global.netflix.com' nslookup: can't resolve '192.1' BusyBox v1.23.2 (2017-01-21 01:37:51 ICT) multi-call binary. Usage: nslookup [HOST] [SERVER] Query the nameserver for the IP address of the given HOST optionally using a specified DNS server BusyBox v1.23.2 (2017-01-21 01:37:51 ICT) multi-call binary. Usage: nslookup [HOST] [SERVER] Query the nameserver for the IP address of the given HOST optionally using a specified DNS server nslookup: can't resolve '192.368.1.1�'
 
@ambiance This looks like bad RAM, or kernel memory space that is being corrupted somehow. I can tell from the way it manifests, and how the corruption is happening. What isn't making sense is why there are things like actual HTML content in some of the logging lines. Tracking this down remotely is basically impossible.

This doesn't appear to be a pixelserv problem. I can tell because some of the logging lines (which are sent through syslog, thus end up handled by syslogd, then written to /var/log -- unless you changed the logging path or destination, or are using a remote syslog server?) are for dnsmasq, which most certainly are not pixelserv.

There's even evidence of the corruption being passed on to shell utilities like nslookup. I've underlined/bolded/italicised the character:

Code: 
Query the nameserver for the IP address of the given HOST optionally using a specified DNS server nslookup: can't resolve '192.368.1.1�'
This looks like a bit-level RAM error (ex. ASCII letter "1" is 0x31 (%00110001 in binary), while "3" is 0x33 (%00110011 in binary -- note the last bit)). Certainly it meant 192.168.1.1. Another example:

Code: 
Fe� 11 97:12:08 from appboot.netflix.com
The "b" in "Feb" is corrupted (can't tell what it becomes), but the timestamp should probably be 17:12:08; ASCII letter "1" is 0x31 (%00110001), while ASCII letter "9" is 0x39 (%00111001). So it's spread across multiple bits, i.e. isn't limited to just bit #3.

If only vendors used ECC RAM... :(

If you've overclocked this router in any way, stop immediately, though damage may have already been done/made permanent.
 
Last edited:
@koitsu It's never been overclocked and that is not good news if it's the hardware. I have IP, Bandwidth and dnsmasq being written to the flash drive the script is running on. Does the evidence suggest that the drive itself could be the problem? I've swapped it with fingers crossed.
 
ambiance said:
@koitsu It's never been overclocked and that is not good news if it's the hardware. I have IP, Bandwidth and dnsmasq being written to the flash drive the script is running on. Does the evidence suggest that the drive itself could be the problem? I've swapped it with fingers crossed.
Click to expand...
As I said: "I can tell because some of the logging lines (which are sent through syslog, thus end up handled by syslogd, then written to /var/log -- unless you changed the logging path or destination, or are using a remote syslog server?) are for dnsmasq, which most certainly are not pixelserv."

/var/log on Tomato is a directory that is a RAM-based filesystem (specifically, /var is a symlink to /tmp/var, and /tmp is tmpfs, which is pure RAM). The default logfile is /var/log/messages. syslogd (the daemon) is what writes to this file. Programs send logging lines to syslogd using the syslog() library call.

If you didn't change the logging destination, then the issue has nothing to do with a flash drive. If you're using a remote syslog server, then the issue has nothing to do with a flash drive.
 
I'm not using a server or flash drive for syslog and if I understand this correctly it's almost certainly not the drive as the programs being called are located in RAM which may have one, two, three (ha, ha, ha) bad sectors. How would I go about diagnosing the memory? If it's severely complicated or not possible via software I'm SoL.

I'd be much happier if this happened to my N66U and not the R7000. FWIW, there hasn't been any corruption since changing flash drives and I wasn't able to find any bad sectors on what I thought may be the culprit. I apologize for making you repeat yourself as my comprehension level on these matters requires me to be told at least twice or thrice depending on the day.

Thanks for your input
 
You must log in or register to reply here.

Back
Top