1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ALL-U-NEED Ad Blocking

Discussion in 'Tomato Firmware' started by srouquette, Nov 6, 2010.

  1. dkirk

    dkirk Network Guru Member

    windozer likes this.
  2. Low-WRT

    Low-WRT LI Guru Member

    Where do you add?

    Code:
    quiet-dhcp
    Thanks!
     
  3. Beast

    Beast Network Guru Member

  4. chrisanthropic

    chrisanthropic Networkin' Nut Member

  5. ppsun

    ppsun Networkin' Nut Member

    You should also mention that it's easier to set up pixelserv on a USB thumbdrive - if the router has a USB port and there's a spare thumbdrive lying around.

    Copy pixelserv to the root of the thumbdrive. And change the pixelserv directory path in the script to: PXL_EXE="/tmp/mnt/sda1/pixelserv".
     
  6. chrisanthropic

    chrisanthropic Networkin' Nut Member

    See, if I'd have known that... :p
    Thanks for clearing that up, I'll add it right now.
     
  7. AMurderOfCrows

    AMurderOfCrows Network Guru Member


    very nice tut, got me running quickly.

    overall, i like the idea of blocking ads for my entire network at the router level. I'd like to take this further though, and do the same blocklists that PeerBlock uses on top of adblock.

    has anyone looked into this possibility yet?

    also, i've noticed that the current instructions for doing this does not block ads on yahoo's main page, but the adblock plus plugin for firefox does. wondering what i need to do to fix this

    thanks!
     
  8. tido

    tido Networkin' Nut Member

    Hi everyone,

    Script is great, but do you have to reboot the router every time one edits the white list? How do you manually rerun the script without requiring a reboot?
     
  9. dkirk

    dkirk Network Guru Member

    I have a WAN connection via DHCP so I merely release the WAN connection and then renew it. This bounce of the WAN port triggers the script to run.
     
  10. Hackerivs

    Hackerivs Networkin' Nut Member

    or
     
  11. Kent_Diego

    Kent_Diego LI Guru Member

    I have noticed a white list issue where it seems the white list no longer functions after a certain amount of time and I have to re-start the wan service to get the white list to work again. I have version 3.9e and USEHOSTS="Y". Here is what I have:
    WHITE="ads.youtube.com feedburner.com revsci.net latimes.com tacoda.net imrworldwide.com tremormedia.com omniture.com intel.com cdn.gigya.com cdn.visiblemeasures.com fwmrm.net view.atdmt.com googlesyndication.com"

    I need to view an ad to use the LA Times crosword puzzle:
    http://games.latimes.com/games/daily-crossword/daily-crossword.aspx
    and every day I need to re-start WAN to see.

    Thanks,
    Kent
     
  12. ppsun

    ppsun Networkin' Nut Member

    I schedule a router reboot everyday at 4 am. That might be a good workaround for you.

    But may not be a good idea if you have something on the LAN that needs to be connected 24/7 though, like long downloads and such.
     
  13. Kent_Diego

    Kent_Diego LI Guru Member

    So how do you do that?

    EDIT: I figured it out. I will try a daily 5:00AM "service wan restart". Since the list update is at 4:00AM it should work but I am not certain it will. It seemed that I lost the white list after 5 hours today.
     
  14. Toink

    Toink Network Guru Member

    In my case the whitelist will not work using the latest version v3.9e no matter how much I tried exempting a couple of sites that I visit once in a while. That's why I reverted back to v3.9d
     
  15. Kent_Diego

    Kent_Diego LI Guru Member

    Hi Toink, the scheduled "service wan restart" did not help. Where can I get 3.9d? Perhaps you can post. The broken white list in 3.9e is getting annoying.

    Thanks,
    Kent

    EDIT: I found a 3.9d a few pages back. Everything looks good so far and I can see the LA Times crossword. I will check again to see if the whitelist is still working in a few hours.

    Kent
     
  16. calpchen

    calpchen Networkin' Nut Member

    Ad Blocking stopped working with last night's domains/hosts data update. Now dnsmasq fails to start up:

    Code:
    Jan  8 16:12:42 RT-C0C1C01C294B user.notice root: ADBLOCK: http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Jan  8 16:12:43 RT-C0C1C01C294B user.notice root: ADBLOCK: http://mirror1.malwaredomains.com/files/justdomains
    Jan  8 16:12:43 RT-C0C1C01C294B user.notice root: ADBLOCK: http://www.malwaredomainlist.com/hostslist/hosts.txt
    Jan  8 16:12:46 RT-C0C1C01C294B user.notice root: ADBLOCK: http://winhelp2002.mvps.org/hosts.txt
    Jan  8 16:12:46 RT-C0C1C01C294B user.notice root: ADBLOCK: 36120 entries
    Jan  8 16:12:46 RT-C0C1C01C294B user.notice root: ADBLOCK: sorting hosts...
    Jan  8 16:13:55 RT-C0C1C01C294B user.notice root: ADBLOCK: hosts sorted.
    Jan  8 16:13:55 RT-C0C1C01C294B user.notice root: ADBLOCK: 28715 entries
    Jan  8 16:13:56 RT-C0C1C01C294B daemon.info dnsmasq[1771]: exiting on receipt of SIGTERM
    Jan  8 16:13:57 RT-C0C1C01C294B daemon.crit dnsmasq[2335]: error at line 28715 of /tmp/gen
    Jan  8 16:13:57 RT-C0C1C01C294B daemon.crit dnsmasq[2335]: FAILED to start up
    Jan  8 16:13:57 RT-C0C1C01C294B user.notice root: ADBLOCK ERROR: restarting dnsmasq...
    Jan  8 16:13:57 RT-C0C1C01C294B daemon.info dnsmasq[2342]: started, version 2.59 cachesize 150
    Jan  8 16:13:57 RT-C0C1C01C294B daemon.info dnsmasq[2342]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n DHCP TFTP no-conntrack no-IDN
    Anybody have any ideas for fixing this?
     
  17. seraphsf

    seraphsf LI Guru Member

    I ran into the same problem. I narrowed it down to the winhelp2002.mvps.org/hosts.txt update that was causing the issue. You can disable it for now until they fix it.
     
  18. marshcroft

    marshcroft Networkin' Nut Member

  19. Low-WRT

    Low-WRT LI Guru Member

    Code:
    Jan  8 16:54:45 unknown user.notice root: ADBLOCK: http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Jan  8 16:54:46 unknown user.notice root: ADBLOCK: http://mirror1.malwaredomains.com/files/justdomains
    Jan  8 16:54:46 unknown user.notice root: ADBLOCK: http://www.malwaredomainlist.com/hostslist/hosts.txt
    Jan  8 16:54:49 unknown user.notice root: ADBLOCK: http://winhelp2002.mvps.org/hosts.txt
    Jan  8 16:54:49 unknown user.notice root: ADBLOCK: 36121 entries
    Jan  8 16:54:49 unknown user.notice root: ADBLOCK: sorting hosts...
    Jan  8 16:55:56 unknown user.notice root: ADBLOCK: hosts sorted.
    Jan  8 16:55:56 unknown user.notice root: ADBLOCK: 28717 entries
    Jan  8 16:55:56 unknown daemon.info dnsmasq[519]: exiting on receipt of SIGTERM
    Jan  8 16:55:57 unknown daemon.crit dnsmasq[1113]: error at line 28717 of /tmp/gen
    Jan  8 16:55:57 unknown daemon.crit dnsmasq[1113]: FAILED to start up
    Jan  8 16:55:57 unknown user.notice root: ADBLOCK ERROR: restarting dnsmasq...
    Jan  8 16:55:57 unknown daemon.info dnsmasq[1120]: started, version 2.59 cachesize 150
     
  20. marshcroft

    marshcroft Networkin' Nut Member

    Low you need to disable S4 for now
    You will see I have 1 2 3 5 8 9 10 running, 4 is out for now, 6 and 7 kills just about everything out there on the net so I disabled it but they are personal preference if you want them then put 6 and 7 back into the GETS line.

    OPTIMISE="Y"
    GETS="1 2 3 5 8 9 10"
    TRIM_BEGIN=3
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml" #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains" #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt" #97K
    S4="http://winhelp2002.mvps.org/hosts.txt" #620K
    S5="http://hosts-file.net/hphosts-partial.asp" #460K
    S6="http://hostsfile.mine.nu/Hosts" #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt" #3851K
    S8="http://www.abuse.ch/spyeyetracker/blocklist.php?download=domainblocklist"
    S9="http://www.abuse.ch/zeustracker/blocklist.php?download=domainblocklist"
    S10="https://easylist-downloads.adblockplus.org/easylist.txt"
     
  21. fubdap

    fubdap Addicted to LI Member

    @marshcroft
    So 6 and 7 will block everything? is that good or bad as far as adblocking is concern
     
  22. marshcroft

    marshcroft Networkin' Nut Member

    if you bring it up in as a webpage you can see what it blocks, megavideo is in there youtube etc, in a business environment I would see no issue blocking all of these sites, but for home use I personally use megavideo and youtube and therefore I have an issue with it, 8 and 9 however block allot of the spyware and malware links that would be found in most scripts on any of those sites, everyone has different requirements for blocks, and my own personal preference shows this above.
     
  23. fubdap

    fubdap Addicted to LI Member

    Thanks.
     
  24. JAC70

    JAC70 Networkin' Nut Member

    I've emailed Mike about the glitch. I think it's this line




    127.0.0.1
    newsoxy.us.intellitxt.com

    It looks fine in the HOSTS file, but when I import it into Excel it splits into two lines. Almost like an invisible hard return or something.
     
  25. ppsun

    ppsun Networkin' Nut Member

    Found this list today. Added it and works OK.

    http://someonewhocares.org/hosts/hosts

    About 280k file size. Last update 4th Jan. Bringing up a total of about 27k of sorted sites (minus mvps) as of today.
     
  26. Ed Wrap

    Ed Wrap Networkin' Nut Member

    You had it right before your edit - it was the "::1 localhost #[IPv6]" at the beginning causing issues, at least for me. I commented out only that line, stuck the modified hosts.txt on my dropbox, and it's back to working fine.
     
  27. JAC70

    JAC70 Networkin' Nut Member

    Huh. I asked him about that line first, and he said it had been there for months.
     
  28. Roimeister

    Roimeister LI Guru Member

    maybe the act of editing the file is somehow fixing the problem? maybe like a missing eof marker?
     
  29. Roimeister

    Roimeister LI Guru Member

    I'm getting an "error at line 1" for this file...
     
  30. ppsun

    ppsun Networkin' Nut Member

    OK on my end. Just did a "service wan restart" 2 mins ago. No errors.
    Using E3000 with shbby's latest bt-vpn.

    This is what I use:
    OPTIMISE="Y"
    GETS="1 2 3 4 5 6"
    TRIM_BEGIN=3
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"
    S2="http://mirror1.malwaredomains.com/files/justdomains"
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"
    S4="http://hosts-file.net/ad_servers.asp"
    S5="http://hosts-file.net/hphosts-partial.asp" #resets each month
    S6="http://someonewhocares.org/hosts/hosts"
     
  31. Ed Wrap

    Ed Wrap Networkin' Nut Member

    Yeah, un-commented out that line, and it still works... Weird!
     
  32. Roimeister

    Roimeister LI Guru Member

    don't know what they did, but the mvps file is working for me now...
     
  33. JAC70

    JAC70 Networkin' Nut Member

    Can't be, the link is down.
     
  34. Roimeister

    Roimeister LI Guru Member

    just dloaded it 30 mins ago, but you're right... it's unavailable now
     
  35. JAC70

    JAC70 Networkin' Nut Member

  36. QSxx

    QSxx LI Guru Member

    If i wanted to keep my blacklist/whitelist in form of txt file on my usb mount (/tmp/mnt/sda1/whitelist...) what would be exact settings in whitelist section?
     
  37. awestriker

    awestriker Networkin' Nut Member

  38. biatche

    biatche Network Guru Member

    I've just added the script to my WAN up.... but it doesn't seem to be working. how should i debug this? there a good site to test ad blocking?
     
  39. EekTheCat

    EekTheCat LI Guru Member

    MVPS Hosts seems to not working. Will try again in a bit.

    Update: Host file readable, just not by the WRT. They change something with the entries? Can't read log now. Anybody else having the same issue?
     
  40. fubdap

    fubdap Addicted to LI Member

    Please can anyone help explain how to ad whitelist to the adblock script. I am currently running the pixelserv from my usb stick. If I can run the whitelist script from there, that would be great.
     
  41. srouquette

    srouquette Network Guru Member

    I just replied to your PM, but I will post here how to do it aswell...

    You can add them in the script, for example:
    WHITE="intel.com
    google.com
    pouet.net"

    You can also use the remote whitelist:

    USEWHITELIST="R" # N/Y/R for remote
    WURL="http://example.com/my_whitelist.txt"

    And if it isn't good enough, I can try to update the script to support it...
     
  42. tido

    tido Networkin' Nut Member

    I've also pasted the script on page 1 of this thread into the WAN up section. But it is not working for me, I tested with http://adware-download.com/ and it heads directly to the URL. I`m running Tomato by shibby v 1.28. How would I troubleshoot this?

    Cheers,

    UPDATE:
    Okay just got it working, I've ran the "script_wanup.sh" from the CLI, and removed S3 "http://www.malwaredomainlist.com/hostslist/hosts.txt" as it was stalling. Is there a more thorough way of seeing the execution of the script. Just running "script_wanup.sh" from the CLI displays the files being downloaded?
     
  43. rs232

    rs232 Network Guru Member

    Is there any way I can store the whitelist in a cifs share?
    Also (just to understand) is the whitelist loaded once? or Queried every time my router does a lookup?

    The reason why I'm asking is to ease the whitelist management. Saving in a cifs share would make it easy to be edited by the clients. A smarter alternative would be to have a very simple web editing interface like ajax or html5 per say to add/modify/remove domains.

    I also wanted to ask: are the blocked domains logged?

    thanks
    rs232
     
  44. gijs73

    gijs73 LI Guru Member

    For me, the dnsmasq always ends up getting itself killed to what seems to be an out of memory issue (at least that's is what I believe given the logs.) I have an E3000 with 64mb RAM and I am using no where near that much (46.12mb free) . I've added two lists that were not stock, but the OOM happens before even using 25mb of RAM. Removing them does allow it to work again but I would like to keep both lists if possible. I keeping having to telnet in to restart dnsmasq but of course the adblocking is disabled. Re-running the adblock.sh just ends up killing the dns again. Is there a per process limit on how much memory a process is allowed to use? As I understand it, dnsmasq is low priority process so it is among the first to be killed when there is a OOM issue. Is there anyway to increase its priority? I don't believe there are actually any memory issues but I could be wrong.

    I am technologically proficient ( at least I like to think so :p ) so if someone can point me in the right direction I can try and figure it out myself.

    Relevant info:


    Code:
    Tomato Firmware v1.28.7496 MIPSR2-Toastman-RT K26 USB Ext
     
    __________________________________________________________
     
     
     
    OPTIMISE="Y"
     
    GETS="1 2 3 4 8 9"
     
    TRIM_BEGIN=3
     
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
     
    S2="http://mirror1.malwaredomains.com/files/justdomains"    #189K
     
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
     
    S4="http://winhelp2002.mvps.org/hosts.txt"              #620K
     
    S8="http://easylist-downloads.adblockplus.org/easylist.txt"
     
    S9="http://easylist-downloads.adblockplus.org/easyprivacy.txt"
     
    __________________________________________________________
     
     
     
    LOGS:
     
     
    user.notice root: ADBLOCK: http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
     
    user.notice root: ADBLOCK: http://mirror1.malwaredomains.com/files/justdomains
     
    user.notice root: ADBLOCK: http://www.malwaredomainlist.com/hostslist/hosts.txt
     
    user.notice root: ADBLOCK: http://winhelp2002.mvps.org/hosts.txt
     
    user.notice root: ADBLOCK: http://easylist-downloads.adblockplus.org/easylist.txt
     
    user.notice root: ADBLOCK: http://easylist-downloads.adblockplus.org/easyprivacy.txt
     
    user.notice root: ADBLOCK: 35010 entries
     
    user.notice root: ADBLOCK: sorting hosts...
     
    user.notice root: ADBLOCK: hosts sorted.
     
    user.notice root: ADBLOCK: 27801 entries
     
    daemon.info dnsmasq[589]: exiting on receipt of SIGTERM
     
    daemon.info dnsmasq[1263]: started, version 2.59 cachesize 8192
     
    daemon.info dnsmasq[1263]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n DHCP TFTP no-conntrack no-IDN
     
    daemon.info dnsmasq[1263]: asynchronous logging enabled, queue limit is 25 messages
     
    daemon.info dnsmasq-dhcp[1263]: DHCP, IP range 10.0.0.100 -- 10.0.0.149, lease time 1d
     
    user.notice root: ADBLOCK: dnsmasq is running
     
     
     
    



    If you need anything else let me know and I can provide it. I did change the cachesize and async loggings as you can tell from the logs, however the problem was occurring before and after those changes. Even though it shows dnsmasq is running, it either dies after or it is already dead.
     
  45. mstombs

    mstombs Network Guru Member

    If there is a duff line in the custom config dnsmasq crashes and is restarted by the adblock script without the adblock config. The Tomato OS also restarts dnsmasq without the adblock config if it dies (can be turned off Administration->debugging). I wonder if its just taking too long to load and dies after the initial check?
     
  46. gijs73

    gijs73 LI Guru Member

    What config could cause such a crash? If its regarding the DNS configs, I was having the crashing even prior to adding those configs ( I actually added them to help with the issue) I reverted those configs regardless.

    If you are talking about the easylists hosts, I have gotten it to work after countless restarts but another restart would just cause it to crash again. One error could be that the easylists are HTTPS and the shell did say how the url was not http or ftp but it still downloaded and parsed the files and I don't believe that should have this kind of effect because the script appears to finish running fine. I believe someone in this thread is also using them successfully.

    I retested to hopefully get more information from the logs by running adblock.sh . After it gives the last line stating that dnsmasq is running, I tested the net. Adblocking was working just fine but the DNS requests were being handled VERY slow (upwards of 10 seconds to resolve) After a couple of minutes, dnsmasq was once again killed and it was not restarted. Nothing new was on the log since the script was run, so there was no mention of why the process was killed.

    Is there any way to get more verbose logs or possibly even kernel logs so I can walk through whats happening?
     
  47. fubdap

    fubdap Addicted to LI Member

    @gijs73 - my advice is to cut back on your number of GETS. Start with the first 4, and gradually work your way back up.
     
  48. gijs73

    gijs73 LI Guru Member

    After more testing, it all came down to a line I had added in the blacklist. I don't know if its the blacklist functionality that doesn't work or if you cannot add a specific file to the listing (probably the latter.) I tried blocking a .gif file that was on a site that I wanted blocked without blocking the site or its resources as a whole. Removing that line resolved the issues I was having and I was able to use the additional GETS with no problem.
     
  49. ryzhov_al

    ryzhov_al Networkin' Nut Member

    A script from a first post have some disadvantages. May i offer an alternative?

    1) A script spoofs DNS-requests to blacklisted domains: it returns a router's a IP instead of real domain IP. But how good is dnsmasq there?

    Here is dnsmasq memory usage (heap only) without any blacklists:
    Code:
    $ cat /proc/`pidof dnsmasq`/smaps | awk 'NR%8==1,NR%8==2'
    ...
    00434000-00444000 rwxp 00434000 00:00 0          [heap]
    Size:                64 kB
    and here is a same with a one third of blacklists from a first post:
    Code:
    $ cat /proc/`pidof dnsmasq`/smaps | awk 'NR%8==1,NR%8==2'
    ...
    00434000-00cc8000 rwxp 00434000 00:00 0          [heap]
    Size:              8784 kB
    2) All scripts files is in a /tmp folder. Is its in RAM too?

    3) While using a local pixelserv you can't use a local web-server. And while using remote pixelserv your browser will wait an answer from it on any new DNS request.

    We can reduce a DNS answer delays by sending a NXDOMAIN reply immediately. All browsers caches "non-existent domain" replies and do not try to resolve blacklisted DNS names in future.

    I've compiled the adsuck - a small DNS server that spoofs blacklisted addresses and forwards all other queries. You may drop blacklisted HOSTS-files from a first post to /opt/etc/adsuck/ and use adsuck with a dnsmasq. An additional adsuck feature is a anti-spoofing white list - its a simple HOSTS-file with a online banking sites and other phishing valuable domains. Now it can be resolved locally and no upstream nameservers can spoof it.
     
  50. fubdap

    fubdap Addicted to LI Member

    Lately, every 4 AM, the whole script does not run. Here is an example of what I see on my log every 4 AM:

    Code:
     Feb 13 15:38:31 unknown daemon.info pixelserv[472]: 3669 requests, 523 errors, 0 bad, 1924 gif, 1222 txt replies 
    (this log is copied from forum but is similar to my log)

    Is this ok or is something wrong?
     
  51. Toink

    Toink Network Guru Member

    Hi, ryzhov_al!

    Do you have a working script that we can test in Tomato? The adsuck looks promising - but I don't have the know how on how I could implement it with Tomato script...

    Perhaps you can enlighten us? Many thanks!
     
  52. mstombs

    mstombs Network Guru Member

    That's not the error, its just the result of poking the pixelserv daemon with "kill -SIGUSR1 $(pidof pixelserv)" prompting it to report stats. There was a meaning for each of the counts...
     
  53. fubdap

    fubdap Addicted to LI Member

    Thanks for getting back to me. The point I am trying to make is that the script should run after the first line like the ones below.

    Code:
    Apr 7 04:00:02 Santos daemon.info pixelserv[1018]: 16471 requests, 370 errors, 12 bad, 9959 gif, 6130 txt replies
    Apr 7 04:00:23 Santos user.notice root: ADBLOCK: http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Apr 7 04:00:23 Santos user.notice root: ADBLOCK: http://mirror1.malwaredomains.com/files/justdomains
    Apr 7 04:00:24 Santos user.notice root: ADBLOCK: http://www.malwaredomainlist.com/hostslist/hosts.txt
    Apr 7 04:00:27 Santos user.notice root: ADBLOCK: http://winhelp2002.mvps.org/hosts.txt
    Apr 7 04:00:47 Santos user.notice root: ADBLOCK: http://hosts-file.net/hphosts-partial.asp
    Apr 7 04:00:48 Santos user.notice root: ADBLOCK: 36622 entries
    Apr 7 04:00:48 Santos user.notice root: ADBLOCK: sorting hosts...
    Apr 7 04:01:58 Santos user.notice root: ADBLOCK: hosts sorted.
    Apr 7 04:01:58 Santos user.notice root: ADBLOCK: 28623 entries
    But this is what I get:
    Code:
    Apr 16 04:00:01 Santos syslog.info root: -- MARK --
    Apr 16 04:00:01 Santos daemon.info pixelserv[1023]: 3041 requests, 8 errors, 18 bad, 1494 gif, 1521 txt replies
    Apr 16 05:00:01 Santos syslog.info root: -- MARK --
    Apr 16 06:00:01 Santos syslog.info root: -- MARK --
     
  54. dkirk

    dkirk Network Guru Member

  55. rs232

    rs232 Network Guru Member

    Is there any source you're aware of that can be used with this script to perform parental control?
     
  56. zorkmta

    zorkmta LI Guru Member


    Which menus do you put this scripts? on DSNMASQ or Scripts WAN? or other part
     
  57. ppsun

    ppsun Networkin' Nut Member

    Pasted into Admin -> Scripts -> WAN Up

    Been using a slightly reduced script (below) for several weeks now. Gives about 36k hosts after optimization.
    --------------
    OPTIMISE="Y"
    GETS="1 2 3 4 5"
    TRIM_BEGIN=2
    S1="http://mirror1.malwaredomains.com/files/justdomains"
    S2="http://www.malwaredomainlist.com/hostslist/hosts.txt"
    S3="http://someonewhocares.org/hosts/hosts"
    S4="http://winhelp2002.mvps.org/hosts.txt"
    S5="http://hosts-file.net/ad_servers.asp"
    --------------
    Note: This is only the changed portion of the script. The original script is taken from post no. 1
    Note2: I schedule a router reboot every night, or else pixelserver doesn't work after a few days.
     
  58. superdos

    superdos Networkin' Nut Member

    Hi,

    I use this script with pixelsrv on my WNR3500L V2 running Tomato Firmware v1.28.0497 MIPSR2-Toastman-RT-N K26 USB VPN.
    When have it activated some sites takes forever to load (for example svd.se, swedish newspaper).
    Is there some way to troubleshoot this kind of thing?
    I use 4 sources.

    Thanks!
     
  59. mstombs

    mstombs Network Guru Member

    Please confirm which version of pixelserv.c - V27 is the latest. (there's a log messages when it starts).


    My old router been up for a couple of months without rebooting - I do restart pixelserv in firewall script (and do not let it auto-update as per this thread...)

    @superdos

    Its possible that there are scripts or other content on blocked domains and you need to enable access by adding them to your whitelist to view the specific sites.
     
  60. ppsun

    ppsun Networkin' Nut Member

    @superdos
    No problem here. Loads up pages in less than 2 seconds for the site.
    Installed with shibby's 87V-AIO. And using OpenDNS (208.67.222.222:53, 208.67.222.220:53).

    Have you tried adding svd.se into the white list? That may bypass the problem.
     
  61. superdos

    superdos Networkin' Nut Member

    I use v27 of pixelserver and haven't tried to add svd.se on the whitelist.
    There's no issue if use Ablock in Chrome (with easylist http://adblockplus.mozdev.org/easylist/easylist.txt).
    What lists are you using?
    BTW can you use easylist with this adblock script?
     
  62. ppsun

    ppsun Networkin' Nut Member

    @superdos
    http://easylist-downloads.adblockplus.org/easylist.txt
    Not tried that list myself but should work. Don't trim.

    EDIT:
    Just tried adding it to my list. I don't think it works, as it only adds a few hundred hosts to the unoptimized host list.

    The list I currently use is at post #357. See if that helps you.
     
  63. superdos

    superdos Networkin' Nut Member

    Ok I've ruled out that it has something with the block lists to do.
    Even if I use one source (a blank txt file) some sites takes forever to load.
    I've tried without pixelserver but no change, so I guess it has something to do with the dnsmasq or some routing issue, maybe a loop is created?
    Any tips where I should start looking?
     
  64. ppsun

    ppsun Networkin' Nut Member

    A bit of a stab, but this is what I would try:

    1) Check if the problem exist on a different browser and/or on another computer. Or maybe Flash or Java might be holding up the page.
    2) Is the NVRAM almost full? If yes, I would cut down on scripts length.
    3) Go back to an older TomatoUSB firmware with full NVRAM clear?
    4) If all these fail, I would probably stop adblocking on the router, and run adblocking on the computer or browser instead.
     
  65. zapoqx

    zapoqx Networkin' Nut Member

    so interestingly enough, I am having a similar issue as superdos, but its in my Linksys E3000 router. Now here is the thing though...
    I think its something someone is doing and not telling me cause on the off chance that no system except mine is on, it works fine for almost the whole day (Pretty much something makes it stop working somewhere I guess). Now, before I used the newest version of Victek's 9011, it was working perfectly fine. I don't see pixelserv fail, but I can't see what version it is either come to think of it. The Ram is not getting full either. I've even tried it with just 2 hosts and it would still fail.
    I believe I am using the correct pixelserv file from the v27 zip.
    Just I would prefer not having to disable the adblocking on the router so I wouldn't have to deal with it in general on the systems, but if I have to, I guess I may need to disable it. Supposedly, I was recently told its been getting worse on someone elses machines. Now, one thing I do notice is those of us hardwired, it doesn't take long to load a page, but it does hang in some spots where I start seeing something like google analytics showing up as trying to load, but not getting there and instead of timing out, it just keeps sitting there, but the page is loaded. On the wireless devices though, they have the "Page taking forever to load." While I did try at one point the reset method of restarting wan service, doing so that way wouldn't restart everything properly. It got to the point that restarting the router every day was better than trying to do a scheduled relaunch of the wan service and pixelserv.
     
  66. JAC70

    JAC70 Networkin' Nut Member

    PixelServ fails for me quite often, when I save a config change and the router needs to restart a service. The only fix is to power cycle the router.
     
  67. mstombs

    mstombs Network Guru Member

    If a website stores more than just images on a blocked host/domain then the web page may not work. If the website runs a script that expect specific response from a blocked site the script may crash with the unexpected response from pixelserv. If you want to view the website you must remove the block - but its not always easy to know which lookup fails. Do not forget many websites give you content for free, they are funded by advertising, so not surprising there are regularly more websites that detect adblocking and fight back!

    I have seen pixelserv fail after config change (not the block, just the null image), it just sits there in memory not doing anything, I think the virtual interface that it is looking at just changes when the lan bridge is restarted. It's a bug, sorry! The workaround I use is to have something like "killall pixelserv && /jffs/pixelserv" (edit to match location and parameters for your use of pixelserv) in the firewall script which ensures it is killed off and restarted on such change.
     
  68. JAC70

    JAC70 Networkin' Nut Member

    Thanks for the suggestion. Sadly, when I tried this, my router froze. I just copied your command verbatim, sans quotes, in to the Firewall script section, is that correct?
     
  69. Inkrypted

    Inkrypted Serious Server Member

    I was unable to get any of the scripts provided here working but after much research and hair loss I now have a working solution that is effective throughtout the network and away from home if I use SSH. I would especially like to thank the individual who created and supplied pixelserv v2.7 and all the individuals here who supplied their knowledge and expertise. Thanks one and all.
     
  70. mstombs

    mstombs Network Guru Member

    I copied and pasted that command from a tomato gui, yes without quotes, so sorry. The router wasn't running the adblock script in this thread so maybe a conflict? Where do you store jffs locally?
     
  71. JAC70

    JAC70 Networkin' Nut Member

    My jffs folder is in the root of the router, if that's what you mean. It contains only the latest Pixelserv, with the read and execute flags enabled. I had to disable the write flag because I found it would shrink the file and stop working for some reason.
     
  72. waeking

    waeking Networkin' Nut Member

    I would really like to find one that works with this adblock like this one. It keeps crashing
    http://easylist-downloads.adblockplus.org/easylist.txt

    I have tried to load is optimized/unoptimized and trimmed/untrimmed and all other possible combinations.
    This running in chrome with adblock, blocks out some adds on sites that no other I have found here in this forum. I am using script 3.9e
     
  73. waeking

    waeking Networkin' Nut Member

    Just one more question. There is a WURL for a white list url. Can I create a BURL for a black list?
     
  74. vash2k6

    vash2k6 Serious Server Member

    Okay so this is my first post here but I'm so happy about what you guys are doing that I felt I should contribute some things I've done!

    So I got everything set up and running just fine. All was good but my pop didn't want to settle for grabbing pixelserv from another source. Actually, we were mostly curious as to where settings get saved on a device with a read-only filesystem. Obviously it went somewhere, it just took us a little bit to figure out it was in NVRAM. That was the holy grail. Suddenly it made sense as to why the script was using base64. The NVRAM stores settings information in plain text and each string in the NVRAM has a maximum length of 4096 bytes. So we gzipped the pixelserv binary and used openssl to encode it in base64:

    gzip < pixelserv.oldT > pixelserv.gz
    openssl enc -base64 < pixelserv.gz >pixelserv.gz.b64

    So we wrote up an init script and pasted the base64 text into it along with some commands to decode, unzip, and move it into the /tmp directory! Alas, the whole thing was larger than 4k. Much larger.

    Where things got crazy: NVRAM not only lets you view it's strings, often you can make new ones too. The only way we were going to get everything to fit was if we split it up. For starters, we cut the pixelserv base64 in half:

    split -l 42 pixelserv.gz.b64

    Then we made two new NVRAM entries:

    nvram set pixelserv_binary_1="$( cat pixelserv.gz.b64.1 )"
    nvram set pixelserv_binary_2="$( cat pixelserv.gz.b64.2 )"
    nvram commit

    And there you have pixelserv saved in local, non-volatile storage. Lastly, we modified the new init script to read as such:

    PXL="/tmp/pixelserv"

    b64="openssl enc -base64 -d"
    [[ "$(echo WQ==|$b64)" != "Y" ]] && b64="b64"

    b64(){
    awk 'BEGIN{b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"}
    {for(i=1;i<=length($0);i++){c=index(b64,substr($0,i,1));if(c--)
    for(b=0;b<6;b++){o=o*2+int(c/32);c=(c*2)%64;if(++obc==8){if(o)
    {printf"%c",o}else{system("echo -en \"\\0\"")}obc=o=0}}}}';}

    {
    nvram get pixelserv_binary_1
    nvram get pixelserv_binary_2
    } | $b64 |gunzip >>$PXL

    chmod 775 $PXL

    In addition to this, we found that it took some time for the router to unpack the binary so we added a 'sleep 120' command to the ALL-U-NEED script right before it invokes it's decompressed adblock script (i.e. right between "chmod 775 $ADB" and "$ADB") so nothing would go out of sync.

    It works great! I can back up the config, update, restore the config and not have to worry about shuffling things back over!

    Oh and no ads are nice too.

    A copy of the script and the files I made are in the attached zip file. Enjoy!
     

    Attached Files:

  75. mstombs

    mstombs Network Guru Member

    A dd-wrt user has previously put a version of pixelserv into nvram, it can be minimized in size by compiling with/without certain options, I think the above is a larger binary. Only worth bothering with if your router doesn't support jffs - but what about the blocklist - isn't that much larger than the pixelserv binary? I keep both on jffs.
     
  76. Chuck Eglinton

    Chuck Eglinton Serious Server Member

    I'm sorry (in advance) for the newbie question: Regarding Jac70's comment, "PixelServ fails for me quite often," same here. Installing the ALL-U-NEED script is easy. Pixelserv, however, not so easy.

    Can I just serve the 1x1 blank pixel from some external host? That is, if I have a 1x1 pixel on a public server somewhere, can I use either the "USEPIXELSERV=" parameters or the "NIP=" parameter (typically set to 0.0.0.0) to display the 1x1 pixel from some remote host? If so, how?
     
  77. Chuck Eglinton

    Chuck Eglinton Serious Server Member

    I think I may know the answer:
    1) In ALL-U-NEED, configure "NIP=" to the IP address of an external server, perhaps one I operate
    2) Create a 404 error page that displays a 1x1 blank pixel.
    http://www.htmlgoodies.com/beyond/r...1/How-Do-I-Create-a-Custom-404-Error-Page.htm

    Is this correct?

    I understand this method won't be as fast has hosting PIXELSERV on the router or on a local server, but it seems much easier to set up and more reliable.
     
  78. tutorialbs

    tutorialbs Serious Server Member

    Best-Of Current DDNSBL (Dynamic DNS Blacklists): http://www.sdsc.edu/~jeff/spam/cbc.html
    • Updated as recent as August 25th, 2012. Make sure the lists you find are (.txt) capable of being used by your script.
     
  79. srouquette

    srouquette Network Guru Member

    yes, you can do something like that.
    the easiest way would be to run a local apache server (xampp) and replace NIP by 127.0.0.1
     
  80. Dark_Shadow

    Dark_Shadow Addicted to LI Member

    By local do you mean on the router itself or an apache server on the PC. I have a PC dedicated to WEB, DNS, FTP server running apache for web. Could this be used?
     
  81. srouquette

    srouquette Network Guru Member

    yes you can use your dedicated server. replace NIP with your server's IP.
     
  82. Dark_Shadow

    Dark_Shadow Addicted to LI Member

    but how will it know what directory to look in?


    Sent from my iPod touch using Tapatalk
     
  83. JAC70

    JAC70 Networkin' Nut Member

    Is there any way to blacklist this without blocking the entire facebook domain? I'm trying to get rid of the stupid Facebook plugins. Thanks.

    http://www.facebook.com/widgets/
     
  84. jrichard326

    jrichard326 Serious Server Member

    I am using the script with pixelserv and it works fine, but lately one of the lists is blocking Songpop(game) on Facebook. I am using lists 1,4,5, and 10. I have started process of elimination and it might be list 5, however, I can't figure out which entry is the culprit. I know it's the script blocking it because it works fine via VPN service. Does anybody have any idea on this one? Which entry should I whitelist? Thanks. This thread has been very helpful to me. There are lots of good ideas here.
     
  85. chowyungfatso

    chowyungfatso Networkin' Nut Member

    Read through many, many guides. Script worked great pretty much right away. However, as many people have said, pixelserv not so much. I'm hoping someone can help me troubleshoot it.

    I have a main USB key partition mounted as /opt using the first part of this guide:

    http://tomatousb.org/tutorialptware-installation

    Code:
    echo "LABEL=Optware /opt ext2 defaults 1 1" >> /etc/fstab
    Other than a script to mount a swap partition and a third "Data" partition (i.e., partition named "Data"--in which I have stored a host file and a dhcp-hosts file for dnsmasq), I actually don't have anything else on the drive as I never went further with Optware (I don't use bittorrent, etc.). I created a bin directory under the opt directory, and put pixelserv in it. Thus, I can see pixelserv in /opt/bin after the router boots up and I've terminal'd (SSH) into it. I can also run pixelserv using the terminal and see it running as a process.

    However, the log says that pixelserv fails to start, and when I visit the pixelserv address (http://192.168.1.2), I don't get anything. This is the settings from the script:

    Code:
    USEPIXELSERV="Y"
    PXL_IP=192.168.1.2
    PXL_EXE="/opt/bin/pixelserv"
    PXL_URL="http://example.com/pixelserv"
    I copied what I think are relevant parts of my syslogd below.

    Can anyone throw me a bone as to why it may not be working? I've spent hours trying to figure things out, and I'm afraid I'm just at a complete loss.

    Thanks in advance.

    Code:
    Dec 31 16:00:47 myrouter syslog.info syslogd started: BusyBox v1.18.5
    Dec 31 16:00:47 myrouter user.notice kernel: klogd started: BusyBox v1.18.5 (2012-09-02 23:11:28 CEST)
    Dec 31 16:00:47 myrouter user.notice kernel: Linux version 2.6.22.19 (root@tomato) (gcc version 4.2.4) #6 Sun Sep 2 23:35:28 CEST 2012
    [...some presumably irrelevant stuff...]
    Dec 31 16:00:51 myrouter user.notice kernel: scsi 0:0:0:0: Direct-Access    Kingston DataTraveler 2.0 PMAP PQ: 0 ANSI: 0 CCS
    Dec 31 16:00:51 myrouter user.notice kernel: sd 0:0:0:0: [sda] 978944 512-byte hardware sectors (501 MB)
    Dec 31 16:00:51 myrouter user.notice kernel: sd 0:0:0:0: [sda] Write Protect is off
    Dec 31 16:00:51 myrouter user.debug kernel: sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
    Dec 31 16:00:51 myrouter user.err kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
    Dec 31 16:00:51 myrouter user.err kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
    Dec 31 16:00:51 myrouter user.info kernel:  sda: sda1 sda2 sda3
    Dec 31 16:00:51 myrouter user.notice kernel: sd 0:0:0:0: [sda] Attached SCSI removable disk
    Dec 31 16:00:51 myrouter user.notice kernel: scsi 1:0:0:0: Direct-Access    Multi    Flash Reader    1.00 PQ: 0 ANSI: 0
    Dec 31 16:00:51 myrouter user.notice kernel: sd 1:0:0:0: [sdb] Attached SCSI removable disk
    [...some presumably irrelevant stuff...]
    Dec 31 16:00:52 myrouter user.warn kernel: EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
    Dec 31 16:00:52 myrouter user.info hotplug[534]: USB ext2 fs at /dev/sda1 mounted on /opt
    [...some presumably irrelevant stuff...]
    Dec 31 16:00:59 myrouter daemon.err apcupsd[419]: apcupsd FATAL ERROR in linux-usb.c at line 609 Cannot find UPS device -- For a link to detailed USB trouble shooting information, please see <http://www.apcupsd.com/support.html>.
    Dec 31 16:00:59 myrouter daemon.err apcupsd[419]: apcupsd error shutdown completed
    [...some presumably irrelevant stuff...]
    Sep 21 18:29:26 myrouter user.warn kernel: HTB: quantum of class 10001 is big. Consider r2q change.
    Sep 21 18:29:26 myrouter user.warn kernel: HTB: quantum of class 10010 is big. Consider r2q change.
    [...some presumably irrelevant stuff...]
    Sep 21 18:30:17 myrouter user.notice root: ADBLOCK ERROR: cannot start pixelserv
    Sep 21 18:30:25 myrouter user.notice root: ADBLOCK: http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    
     
  86. mstombs

    mstombs Network Guru Member

    @chowyungfatso

    No message in log from pixelserv, so looks like script can't find the executable binary when it runs.

    What do you get when you enter

    Code:
    ls -laF /opt/bin/pixelserv
    I'm thinking its either a timing issue or linux access rights.
     
  87. chowyungfatso

    chowyungfatso Networkin' Nut Member

    I get this, which is a good thing, right?

    Code:
    root@myrouter:/tmp/home/root# ls -laF /opt/bin/pixelserv
    -rwxr-xr-x    1 root    root          7940 Sep  1  2010 /opt/bin/pixelserv*
    
    In the logs, the drive appears to be mounted before this script runs. Also, does pixelserv shut down by itself? I've run it manually, and I'll come back and check it, and it's not listed in the processes.
     
  88. ppsun

    ppsun Networkin' Nut Member

    My pixelserv process quits occasionally, so I recently added this into the scheduler (every hour):

    killall pixelserv
    /jffs/pixelserv

    So, in theory, I should have less than an hour non-pixelserved pages whenever pixelserv self-quits. But I haven't run it long enough yet to ascertain it's effectiveness (ie: pixelserv hasn't self-quit yet).

    You may want to give this a try?

    Note: I am using RT66U without optware.
     
  89. mstombs

    mstombs Network Guru Member

    Yes, it all looks fine - the * means it is executable etc. Only thing I notice is that its not the 'current' recommended V27
    I recommend you grab the latest pixelserv binary which reports :-


     
  90. chowyungfatso

    chowyungfatso Networkin' Nut Member

    Sorry guys, been busy. Got my new laptop, transitioning firms, etc., so haven't had a chance to really look at it.

    Thanks, really, for looking things over. Probably saved me another few hours. I'll try to upgrade to the new pixelserv first, and if that doesn't work, at least I'll be able to post reports. And if THAT doesn't work, I'll redo everything from scratch and just add it to jffs--I was really hoping to keep everything on the USB key. I'll keep that cron job idea in mind too.

    Haha, I did want to report that I previously "turned" on all 7 host files, and other than it taking a while too set up, the router still has plenty of memory:

    Code:
    Total / Free Memory    249.74 MB / 214.76 MB (85.99%)
    Total / Free Swap    60.93 MB / 60.93 MB (100.00%)
    Total / Free NVRAM:    64.00 KB / 30.57 KB (47.76%)
    So:

    Code:
    Sep 24 09:55:18 washburn user.notice root: ADBLOCK: 342820 entries
    Sep 24 09:55:18 washburn user.notice root: ADBLOCK: sorting hosts...
    ...
    Sep 24 10:03:20 washburn user.notice root: ADBLOCK: hosts sorted.
    Sep 24 10:03:20 washburn user.notice root: ADBLOCK: 141501 entries
    I'm surprised the list at:

    Code:
    http://support.it-mate.co.uk/downloads/hosts.txt
    has dropbox.com and tomshardware.com on it . It even had an entry for amazon.com on it (found out it was blocking aws.amazon.com, which is the cloud service used by Evernote to distribute its app--new laptop, so hitting a lot of sites to DL everything). I haven't looked at the sorted list, so not sure what else is in there, but I added entries for those domains in the whitelist for those sites, which seems to have superseded the host file (by design). I may then go ahead and add back in specific servers to block (e.g., amazon ad servers) in the blacklist.

    Anyhoo, back to the party...
     
  91. chowyungfatso

    chowyungfatso Networkin' Nut Member

    Looks like it worked with the new ("right") version:

    Code:
    Sep 24 20:14:23 washburn daemon.info pixelserv[1479]: /opt/bin/pixelserv V27 compiled: Dec 12 2010 21:49:05 from pixelserv27.c
    Sep 24 20:14:23 washburn daemon.notice pixelserv[1481]: Listening on br0 192.168.10.2:80
    [Then, I sent killall -USR pixelserv:]
    Sep 24 20:40:07 washburn daemon.info pixelserv[1481]: 19 requests, 9 errors, 0 bad, 10 gif, 0 txt replies
    
    Will set up a schedule to kill/restart it every so often so it will write something to the logs. Thanks guys. Next step: setting it up so it will grab the list from my Ubuntu box/NAS that will do the "heavy lifting" of sorting/processing the lists.]
     
  92. gijs73

    gijs73 LI Guru Member

    Unfortunately, I don't have "nice". Would someone please post or link to a copy of 3.9c. Thank you in advance.
     
  93. Toink

    Toink Network Guru Member

    Here you go: Click me
     
    pharma and gijs73 like this.
  94. zong

    zong Serious Server Member

    dear Toink,
    running up to now a simple script (http://www.linksysinfo.org/index.php?threads/auto-dl-hosts-file-and-install.21378/) and was scared to install pixelserv. Did I get this right, that in your script the "pixelserv" is on dropbox ?
    Is this your version , will it be downloaded and installed automatically- will it stay in dropbox permanently ?
    Can you pls make an example for the "customer black list". (e.g. example1.xyz.com , example2.xyz.com)
    BLACK=""
     
  95. pharma

    pharma Network Guru Member

    Thanks Toink!
     
  96. Toink

    Toink Network Guru Member

    Yes I put the pixelserve file in my dropbox account. If you also have a dropbox account, you can put yours and link to it. Just make sure you put the pixelserve file in your Public folder.

    I actually forgot to edit my own script before I uploaded it to share to this thread. Anyway, this is version 3.9c and already edited to its original state. Which I'm no longer using.

    I would suggest that you don't use the dropbox link in my previous linked file, coz I'm pretty sure I got a different pixelserve version in there - which may not be compatible to your router and the script may not work - I also cannot promise you that it will stay there permanently.

    Just upload one to your personal dropbox account, just to be sure. If you don't have a dropbox account just sign up for a free account HERE ;)

    If you want to place example1.xyz.com, example2.xyz.com in your blacklist, just insert it in there where:

    BLACK="example1.xyz.com, example2.xyz.com" - just make sure you got the links inside the quotation marks. :)

    Or you can create your own list,

    Code:
     127.0.0.1  example1.xyz.com
    127.0.0.1  example2.xyz.com 
    save it as text file then upload it to your Drop Boxe's "Public Folder". Copy the link and add it in the set of lists in the script example:

    S8 = "http://<your dropbox .txt link here"

    Just make sure you add "S8" to your "GETS" so it becomes "1 2 3 4 8" . And yes, you can add as many sites/ domains in there ... Just make sure you type 127.0.0.1 for each line, and then type the domain name. No need to include "www" just type in xyz.com so in the case of www.twitter.com for example, just type in 127.0.0.1 twitter.com

    HTH...
     
  97. Mr. Scary

    Mr. Scary Addicted to LI Member

    Is this script still supposed to work? I tried it and it blocked absolutely nothing.
     
  98. pharma

    pharma Network Guru Member

    Yep! Sounds like you don't have it setup properly ...
     
  99. pharma

    pharma Network Guru Member

    Actually, more ads than normal are appearing so possibly some of the Ad lists isn't functioning.

    EDIT: Rebooted and everything seems to work now ... :)
     
  100. Mr. Scary

    Mr. Scary Addicted to LI Member

    Well, the OP doesn't say anything about any setup, just to copy and paste the script and then some other stuff I assumed was optional.
     

Share This Page