1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Script: Clean, Lean and Mean Adblocking

Discussion in 'Tomato Firmware' started by haarp, Apr 23, 2013.

  1. koitsu

    koitsu Network Guru Member

    Code:
    : not found /cifs1/adblock/config: line 9:
    : not found /cifs1/adblock/config: line 27:
    : not found /cifs1/adblock/config: line 31:
    : not found /cifs1/adblock/config: line 35:
    
    This could also indicates a problem that could potentially affect the behaviour of the subsequent wget within the script. HTTP 400 / Bad Request usually indicates either a) incorrect HTTP client headers included, or b) incorrect/invalid HTTP GET/POST/etc. parameters passed to the server by the client.
     
  2. iommi

    iommi Network Newbie Member

    Thank you. I did try to eliminate those and can eliminate them if I delete all blank lines in the script.

    I am editing on Windows. There are no blank lines between the SOURCE settings but maybe the CR/LF is wrong for Tomato's Unix? I will try to convert the [CRLF] to [LF] and see what happens.

    Success! I used Notepad++ on windows and set the Edit>EOL Conversion setting to UNIX/OSX Format".

    Thank you very much for pointing me in the right direction! :)
     
    koitsu likes this.
  3. iommi

    iommi Network Newbie Member

  4. HunterZ

    HunterZ LI Guru Member

    If anyone cares, here's a set of block lists that report last modified time so that you can check for updates without being guaranteed that something will download and cause the blocklist to be regenerated every time:
    Code:
    ### Sources (uncomment desired blocklists) [must be compatible to the hosts file format!] ###
    ## MVPS HOSTS (~600k) [default]:
    SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
    ## pgl.yoyo.org (~70k) [default]:
    SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
    ## Hosts File Project (~3M!):
    SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    ## hpHosts ad/tracking servers (~400k):
    SOURCES="$SOURCES http://hosts-file.net/ad_servers.txt"
    ## MalwareDomainList.com (~40k):
    SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
    Don't worry about the other sources - this actually blocks even more stuff than I'd like, such that I've had to whitelist a dozen or so sites. I think mine.nu or one of the others may even block itself, preventing future updates unless you whitelist it.

    Edit: The sizes in the file comments are not accurate. My blocklist is only 1.29MB.
     
    Last edited: Jun 2, 2014
    pharma and iommi like this.
  5. timjordan

    timjordan Connected Client Member

    could someone post the latest script with the latest fixes. Right after the script was posted on the first page it seems someone found errors in it and a few posts down fixed the errors. I would assume there might be errors in posts after that with fixes and wanted to know if anyone had the latest script with errors fixed.
     
  6. D.Raven

    D.Raven Serious Server Member

  7. Mojonba

    Mojonba Network Guru Member

    For some reason this script is not loading completely in two of my routers as of yesterday. It gets stuck after downloading the lists. Below the log results:

    Code:
    root@unknown:/tmp/home/root# /jffs/adblock/adblock.sh
    ADBLOCK: Running as /jffs/adblock/adblock.sh
    ADBLOCK: Using config file /jffs/adblock/config
    ADBLOCK: Requested list mode is OPTIMIZE
    ADBLOCK: Config or script has changed - rebuilding list
    ADBLOCK: Download starting
    Connecting to winhelp2002.mvps.org (216.155.126.40:80)
    Connecting to www.malwaredomainlist.com (143.215.130.61:80)
    -  100% |*************************************************| 42023  0:00:00 ETA
    -  100% |*************************************************|  500k  0:00:00 ETA
    
     
  8. HunterZ

    HunterZ LI Guru Member

    Maybe try disabling all sources and then re-enabling them one at a time?
     
  9. Almaz

    Almaz Serious Server Member

    My hosts file located in my DropBox but it looks like they started forwarding http to https. Since I don't have wget-ssl on my tomato, it stopped working. I need to point hosts file to local file which is located in /cifs1/hosts. I can't find the settings to change it. Please advice.
     
    Last edited: Jul 1, 2014
  10. Spyros

    Spyros LI Guru Member

    pgl.yoyo.org was offline for some days, now it's back again
     
  11. HunterZ

    HunterZ LI Guru Member

    Has anyone come up with a slicker way to toggle ad blocking than SSHing in and running adblock.sh?

    I'd like to come up with something easy enough for my wife to use to help determine if the ad blocker is impacting her ability to browse various sites.

    I'm thinking of maybe setting up LIGHTTPD on LAN port 8080 with a simple interface that can be used to invoke adblock.sh with the desired parameters on the router.
     
  12. jerrm

    jerrm Network Guru Member

    LightHttpd is probably the best option, particulary if your build already includes it. If your build doesn't have lighthttd, you can use a cgi script with the native webserver, but she would need the router admin password to login.
     
    HunterZ likes this.
  13. HunterZ

    HunterZ LI Guru Member

    I've got Entware installed on a cifs mount, so lighttpd would be no big deal to install.

    Password-protected login would not be a dealbreaker, though, because she could just save the credentials in her browser.
     
  14. JoeDirte

    JoeDirte Serious Server Member

    If physical access in not a problem, you can simply map /<yourpath>/adblock.sh toggle to the SES/WPS/AOSS Button under Administration > Buttons/LED

    Another option is bookmark http://192.168.1.1/tools-shell.asp and run it from there. The command is retained until changed, so she would just have to authenticate and hit execute to toggle.
     
    HunterZ likes this.
  15. jerrm

    jerrm Network Guru Member

    HunterZ likes this.
  16. anthr4x

    anthr4x Reformed Router Member

    Does anyone knows how to unblock hulu?
    I've tried adding multiple lines to config file under whitelist with no success.
    Only way I menage to play hulu is by stopping adblock via command
    Any advice is greatly appreciated.
     
  17. HunterZ

    HunterZ LI Guru Member

    This is perfect! I was able to put it all in one script, by having it generate links that can be clicked to call the script with specific parameters:
    Code:
    #!/bin/sh
    REFRESHTIME=60
    case $QUERY_STRING in
      start|restart|stop) REFRESHTIME=5;;
      *)  REFRESHTIME=60;;
    esac
    
    echo '<!DOCTYPE html><html><head><meta http-equiv="refresh" content="'
    echo $REFRESHTIME
    echo '; URL=adblock.sh"></head><body>'
    
    case $QUERY_STRING in
      start)
      echo 'starting adblock...<p><pre>'
      /cifs1/adblock/adblock.sh
      echo '</pre>'
      ;;
      restart)
      echo 'restarting adblock...<p><pre>'
      /cifs1/adblock/adblock.sh restart
      echo '</pre>'
      ;;
      stop)
      echo 'stopping adblock...<p><pre>'
      /cifs1/adblock/adblock.sh stop
      echo '</pre>'
      ;;
      *)
      echo 'adblock status:<ul>'
      echo '<li>blocklist '
      if nslookup google-analytics.com | grep -q 192.168.1.254 ; then
      echo up
      else
      echo down
      fi
      echo '<li>iptables '
      echo `iptables -L | grep -c 192.168.1.254`/4
    
      echo '<li>pixelserv '
      if ps | grep -q pixelserv ; then
      echo up
      else
      echo down
      fi
      echo '</ul><br>'
    
      echo '<p>adblock actions:<ul>'
      echo '<li><a href="adblock.sh?start">start/update</a>'
      echo '<li><a href="adblock.sh?restart">restart</a>'
      echo '<li><a href="adblock.sh?stop">stop</a>'
      echo '</ul><p>'
      ;;
    esac
    
    echo '<p><hr>current time: '
    date
    echo '<br>page will automatically refresh after '
    echo $REFRESHTIME
    echo ' seconds'
    echo '<br>...or click <a href="adblock.sh">here</a> to refresh manually'
    echo '</body></html>'
    Note that the page auto-refreshes every 60 seconds in case you want to use it to monitor adblock status. Also, it may be over-customized for my setup, so anyone interested in using it may need to make some tweaks.

    edit: fixed HTML well-formedness (oops)
     
    Last edited: Jul 17, 2014
    jerrm and Tuurbo like this.
  18. jerrm

    jerrm Network Guru Member

    You went and got all fancy with it. I have some similar scripts as well.

    If I ever open my mod of the script back up, I might steal some of this. It wouldn't really honor the name of the thread any longer, but that has long been the case.

    It would be nice if we could get pixelserv stats somewhere besides syslog, maybe recognize special url that will return the stats as a text response.
     
    Last edited: Jul 18, 2014
  19. HunterZ

    HunterZ LI Guru Member

    Note that I called my UI script adblock.sh only because I wasn't feeling very creative. I suppose the functionality could be modded into the real adblock.sh though.
     
  20. jerrm

    jerrm Network Guru Member

    Yeah, if I did it would just be a link created in wwwext and if called via the link only execute the cgi stuff.
     
  21. Gustav Gans

    Gustav Gans Network Newbie Member

    I use original Tomato 1.28. Can I use this script?

    JFFS is called JFFS2 here. I enabled it.
    But in Tomato 1.28 user interface I see no "Tools - System" box where I could paste stuff.
     
  22. jerrm

    jerrm Network Guru Member

    I've never used any version that old. If there is no Tools->System then you will need to telnet/ssh into the box to create the files. It's also possible differences in busybox or dnsmasq could cause issues.
     
    Last edited: Jul 18, 2014
  23. Gustav Gans

    Gustav Gans Network Newbie Member

    I also see there is just 700 KB of free JFFS. That is really little, isn't it? I'm using an old WRT54G v3.1. Maybe those old devices should better be using that other "all u need" script?
     
  24. anthr4x

    anthr4x Reformed Router Member

    Does anyone knows how to fix the problem with hulu?
    I cant watch any movies if adblock its running... i tried adding several links to whitelist with no success..


    Please help!!!
     
  25. HunterZ

    HunterZ LI Guru Member

    Steps:
    1. Enable the log-queries option for dnsmasq (http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-4#post-233602). You will need to restart dnsmasq to make it pick up the option. You can accomplish this by restarting adblock.
    2. Try to play a video.
    3. View the router's system log to find the list of domain names that your computer tried to access that resolved to the adblock/pixelserv IP.
    4. Try whitelisting them one by one until videos play. You will need to force regeneration of the blocklist to pick up each whitelist change.
    5. Try removing all but the last added domain name to narrow the whitelist to contain just the needed entry/entries.
    6. Disable the log-queries dnsmasq option.
     
  26. anthr4x

    anthr4x Reformed Router Member

    Thank you ill give it a try.
     
  27. Gustav Gans

    Gustav Gans Network Newbie Member

    This is not easy when doing it for the first time. Do you have to install both the adblock script and pixelserv into the same folder? I'm asking because the pixelserv script defaults to "/adblock" while the updated adblock code uses "/bin", and the original adblock code also says "/adblock".

    Okay, nevermind, I put everything into the same directory, and it seems to work.
     
    Last edited: Jul 18, 2014
  28. JoeDirte

    JoeDirte Serious Server Member

    All I had to do was add hulu.com to the whitelist.
     
  29. anthr4x

    anthr4x Reformed Router Member

    I did adds hulu to whitelist but it didn't work.. but now thanks to the help provided it worked. The key of all was adblock.sh force.

    Again thanks for the help
     
  30. jerrm

    jerrm Network Guru Member

    Using Haarp's version or mine? Mine should have picked up the whitelist change. If it didn't then it is a bug I need to look at.
     
    Last edited: Jul 19, 2014
  31. anthr4x

    anthr4x Reformed Router Member

    Haarp's
     
  32. HunterZ

    HunterZ LI Guru Member

    Okay, now here's a *really* fancy version of my adblock browser control/status script. I had to learn some basic CSS and JavaScript, which was a good learning experience since I'm a C++ developer rather than a web one.

    New features:
    • General status page shows the last 100 blocked DNS queries performed by devices other than the router (assuming you have dnsmasq set to log to /cifs1/adblock/dnsmasq.log and to log queries).
    • Shows a countdown until the next page refresh.
    • Uses CSS frames to put info across the top of the window (but still looks fine if you disable the CSS style).
    • Shows 5 second "please wait" delay pages before performing actions, so that the user won't wonder why the result page is taking a while to load.

    Code:
    #!/bin/sh
    REFRESHTIME=60
    NEXTACTION=""
    case $QUERY_STRING in
      start)
      REFRESHTIME=5;
      NEXTACTION="?dostart";
      ;;
      restart)
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      stop)
      REFRESHTIME=5;
      NEXTACTION="?dostop";
      ;;
      *)
      REFRESHTIME=60;
      NEXTACTION="";
      ;;
    esac
    
    cat << EOF
    <!DOCTYPE html>
    <html>
    <head>
    <title>adblock status</title>
    <meta http-equiv="refresh" content="$REFRESHTIME; URL=adblock.sh$NEXTACTION">
    <style type="text/css">
    body {
      margin: 0;
      padding: 1px 1px 1px 1px;
      height: 100%;
      overflow-y: auto;
    }
    #status {
      display: block;
      top: 0px;
      left: 0px;
      padding: 1px 1px 1px 1px;
      width: 150px;
      height: 90px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #actions {
      display: block;
      top: 0px;
      left: 150px;
      padding: 1px 1px 1px 1px;
      width: 150px;
      height: 90px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #time {
      display: block;
      top: 0px;
      left: 300px;
      padding: 1px 1px 1px 1px;
      width: 100%;
      height: 90px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #blocks {
      margin: 90px 0px 0px 0px;
      padding: 1px 1px 1px 1px;
      display: block;
      padding: 0px;
    }
    </style>
    </head>
    <body>
    <script>
      var count=$REFRESHTIME;
      var counter=setInterval(timer, 1000);
      function timer()
      {
      count=count-1;
      document.getElementById("timer").innerHTML=count;
      if (count <= 0)
      {
      clearInterval(counter);
      return;
      }
      }
    </script>
    EOF
    
    echo '<div id="status">'
    echo '<b>adblock status:</b><br>'
    case $QUERY_STRING in
      start)
      echo 'starting/updating adblock...<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostart)
      echo 'start/update complete<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
      /cifs1/adblock/adblock.sh
      echo '</pre><p></div>'
      ;;
      restart)
      echo 'restarting adblock...<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dorestart)
      echo 'restart complete<p><pre></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
      /cifs1/adblock/adblock.sh restart
      echo '</pre><p></div>'
      ;;
      stop)
      echo 'stopping adblock...<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostop)
      echo 'stop completed<p><pre></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
      /cifs1/adblock/adblock.sh stop
      echo '</pre><p></div>'
      ;;
      *)
      echo 'blocklist '
      if nslookup google-analytics.com | grep -q 192.168.1.254 ; then
      echo up
      else
      echo down
      fi
      echo '<br>iptables '
      echo `iptables -L | grep -c 192.168.1.254`/4
    
      echo '<br>pixelserv '
      if ps | grep -q pixelserv ; then
      echo up
      else
      echo down
      fi
      echo '<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><a href="adblock.sh?start">start/update</a>'
      echo '<br><a href="adblock.sh?restart">restart</a>'
      echo '<br><a href="adblock.sh?stop">stop</a>'
      echo '<p></div>'
    
      echo '<div id="blocks">'
      echo 'last 100 blocked domain names:<br><pre>'
      grep -B1 192.168.1.254 /cifs1/adblock/dnsmasq.log | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | awk '{printf("%s %s %s) %-13s %s\n", $1,$2,$3,$8,$6)}' | tail -n 100
      echo '</pre><p></div>'
      ;;
    esac
    
    echo '<div id="time">'
    echo '<b>time info:</b><br>'
    echo 'router time:' `date`
    
    cat << EOF
    <br>page will automatically refresh in <span id="timer">$REFRESHTIME</span> seconds
    <br>...or click <a href="adblock.sh">here</a> to refresh manually
    </div>
    </body>
    </html>
    EOF
    And here's what it looks like:

    [​IMG]

    edit: tweaked to look better on tablets/smartphones
     
    Last edited: Jul 19, 2014
    Almaz and ShinichiYao like this.
  33. Gustav Gans

    Gustav Gans Network Newbie Member

    Since I disabled Pixelserv, I had noticeably faster page loads. My Router is an old WRT54G. Is it coincidence or does Pixelserv really decrease loading speed?

    Also I wonder if you might just turn Pixelserv off if you only use web browsers which don't show strings like "page not found" at every place where a banner should normally be. Correct me if I'm wrong, but as I understand the only purpose of Pixelserv is to prevent such visible error messages in your browsers, right? The purpose is not to gain any speed?
     
  34. HunterZ

    HunterZ LI Guru Member

    Pixelserv can be a speed gain over redirecting to a dead IP in cases where the browser would get hung up for a while while waiting for a response that will never come.

    When I had a WRT54G, I ran a pixelserv-like program on a separate Linux box on my LAN and had the ad blocker redirect to that.
     
  35. Gustav Gans

    Gustav Gans Network Newbie Member

    Sorry to bother you with my questions, but I need to know if it's dangerous to use JFFS for this script. I have to admit I just followed all those instructions but don't really know what's going on.
    I think the script starts, fetches all uncommented lists, gives them to a service that's called DNSMASQ and then exits. That all happens in the Router's RAM? Also I found some backup of the config script or something like that on the JFFS partition (if you need the name I'll have another look tomorrow). Can this behavior be disabled? I don't know if I'll ever need such a file, so maybe it's not worth the additional wear and tear.

    I already activated the option RAMLIST because I don't have enough space.
    Is there anything else left being written regularly to JFFS?

    Also I'd like to know if I have to use the Adblock script's "cron" or "update" options (if they are parameters at all) or if I can just put, in my case, "/JFFS/adblock/adblock.sh" into one of Tomato's custom Scheduler boxes without having to worry about multiple Adblock instances or RAM/speed decrease over time.

    Thank you
     
  36. HunterZ

    HunterZ LI Guru Member

    I don't know about the RAMLIST option, but normally one copy of each source list is stored, plus one copy of the final blocklist. There's no need to worry about multiple instances of anything because it mainly just provides configuration to the single instance of dnsmasq that is maintained by the router.
     
  37. Spyros

    Spyros LI Guru Member


    Ιm using this as WAN Up script, nothing gets written to JFFS
    Code:
    ### Copy Adblock Script to /tmp  ###
    cp -r /jffs/adblock/* /tmp/
    chmod +x /tmp/adblock.sh
    chmod +x /tmp/pixelserv
    ###  Enable AdBlock  ###
    /tmp/adblock.sh
     
    bluechalk likes this.
  38. Almaz

    Almaz Serious Server Member

    Deleted
     
    Last edited: Jul 25, 2014
  39. Almaz

    Almaz Serious Server Member


    You don't really need LightHttpd. Just put the script in /var/wwwext and start it with http://router/ext/script.sh Make sure to symlink adblock files/folder to /var/wwwext


    Update: after further testing, I just noticed, it's not working on my side. Everytime I click on the link it just runs adblock.sh but it doesn't run commands such as adblock.sh stop or adblock.sh restart. The path is correct for all the locations for adblock.sh in a script. What could be a problem?
     
    Last edited: Jul 25, 2014
  40. HunterZ

    HunterZ LI Guru Member

    If you're using my script, you'll have to fix some paths to be correct for your setup.
     
  41. Almaz

    Almaz Serious Server Member

    All the paths were changed in the first place. When I click "Stop" that's the message I get.

    ADBLOCK: Running as /www/ext/adblock.sh ADBLOCK: Using config file /www/ext/config ADBLOCK: Requested list mode is OPTIMIZE ADBLOCK: Blocklist active in OPTIMIZE mode ADBLOCK: List not old enough to update ADBLOCK: Exiting /www/ext/adblock.sh 0

    As you can see it's not stopping adblock and not forwarding the command "stop" after adblock.sh
     
  42. HunterZ

    HunterZ LI Guru Member

    What output do you get if you put echo in front of the lines that call adblock.sh?

    What happens if you run /www/ext/adblock.sh stop from an SSH prompt?
     
  43. Almaz

    Almaz Serious Server Member


    Adblock.sh and your script is in the same folder which is located in "/var/wwwext/adblock.sh" but when I run your script it shows "/www/ext/adblock.sh". ADBLOCK.sh by itself works fine.

    Even if I run the command, that's what I get

    /www/ext/adblock.sh stop
    ADBLOCK: Running as /www/ext/adblock.sh stop
    ADBLOCK: Using config file /www/ext/config
    ADBLOCK: Requested list mode is OPTIMIZE
    ADBLOCK: Processing 'stop' option, remaining options ignored
    ADBLOCK: Stopping
    ADBLOCK: Restarting dnsmasq
     
  44. HunterZ

    HunterZ LI Guru Member

    Maybe post or PM your customization of my script?
     
  45. Almaz

    Almaz Serious Server Member

     
    Last edited: Jul 28, 2014
  46. HunterZ

    HunterZ LI Guru Member

    I see the problem:

    In my setup, my web script is /tmp/var/wwwext/adblock.sh and my real adblock script is /cifs1/adblock.sh.

    If you want to have your real adblock.sh live in the same place as my script, you're going to need to rename all the meta and href HTML tags to point at whatever you called my script instead of at the real adblock.sh.
     
  47. Almaz

    Almaz Serious Server Member

    Great job for the script and catching my mistake. Just made a little improvement to your code to make it easier for everyone to use it and set it up. Added force command as well. Only the first 5 lines needs to be modified for your script to work. Also can be found on Github https://github.com/Almazick/AdBlock-Web-GUI

    Update:
    Added numbers of unique hosts
    Cleaned up dnsmasq log from syslog
    Now it shows time and router load average

    Script
    http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-9#post-250775
     
    Last edited: Sep 15, 2014
    HunterZ likes this.
  48. HunterZ

    HunterZ LI Guru Member

    It occurred to me that it would be a good idea to add that kind of configurability, but I was too lazy :p Looks good.
     
  49. Almaz

    Almaz Serious Server Member

    Updated your script, see the previous post.
     
  50. JakeS66

    JakeS66 Network Newbie Member

    I've been experiencing some lag while the adblock is enabled. I can really notice horrible load times with comcast.com and becu.org. In the case of becu, I enabled logging and I can see the following queries:

    Jul 28 07:36:23 router daemon.info dnsmasq[1706]: config fls.doubleclick.net is 192.168.1.254
    Jul 28 07:36:44 router daemon.info dnsmasq[1706]: config becu.122.2o7.net is 192.168.1.254

    If I add the two sites into my whitelist and perform a force, then the site loads without any lag or problems. I am able to connect to 192.168.1.254 via my web browser just fine (blank page, no html behind).

    Any thoughts or suggestions? Thanks!
     
  51. HunterZ

    HunterZ LI Guru Member

    Use pixelserv to get faster load times. If you are using pixelserv, verify that it's running by opening a test address in your browser that you haven't visited before, like http://192.168.1.254/test1234.png. If you don't get a response, then either pixelserv isn't running or your iptables rules are not in place.

    Note that mucking with your router's configuration UI has a tendency to wipe out your iptables rules, so you may need to run adblock.sh restart to put them back in place.
     
  52. JakeS66

    JakeS66 Network Newbie Member

    Thanks for the quick response! I ran a network capture and it would seem that these website are trying to access the 254 address via https and timing out. Is there a way to prevent this?

    Edit: I should report that standard http (80) is working fine for pixelserv.

    Thanks!
     
  53. HunterZ

    HunterZ LI Guru Member

    Yes, I believe that iptables rules for redirecting/dropping/rejecting HTTPS port 443 traffic were discussed earlier in the thread. I don't remember which solution I am using.
     
  54. JakeS66

    JakeS66 Network Newbie Member

    Okay, I'll scan through this thread again and see if I missed it. Thanks for your support!

    Edit: Found the solution within the thread! Works perfectly!

    iptables -t nat -A PREROUTING -i $BRIDGE -p tcp -d $redirip --dport 443 -j DNAT --to $redirip:80
     
    Last edited: Jul 28, 2014
  55. Almaz

    Almaz Serious Server Member

    Why not just update your adblock to the latest version? it should be already setup for ports 80 and 443
     
  56. JakeS66

    JakeS66 Network Newbie Member

    I already show as having version 4.5. I'll download it again just to perform a diff and see what has changed. thanks!
     
  57. Almaz

    Almaz Serious Server Member

  58. HunterZ

    HunterZ LI Guru Member

    This site really needs a wiki so that we can collaboratively maintain stuff when the original poster stops checking in.
     
    pharma likes this.
  59. Almaz

    Almaz Serious Server Member

    A simple solution just to start a new thread
     
  60. HunterZ

    HunterZ LI Guru Member

    ... Until that thread starter goes missing too. Not to mention people would then get confused at there being two threads on the same subject.

    Simple is not always ideal :-P
     
  61. Almaz

    Almaz Serious Server Member

    If there are no other solutions then it is only the solution for Jerrm to start a new thread. Add your idea and others ideas on a second post. The main question, if Jerrm wants to support his own script and deal with it.
     
  62. Almaz

    Almaz Serious Server Member

    Is there a similar script to setup the same thing on Debian box? I know I can do it manually but if there is a script ready, would be nice to use it. In regards to router to point to external pixelserve and dns, what would be the correct iptables?
     
  63. Almaz

    Almaz Serious Server Member

    Script works great!
     
    Last edited: Sep 11, 2014
  64. shipleythump

    shipleythump Network Newbie Member

    I used this:
    Code:
    PREFIX=/opt/adblock
    wget -O - http://pastebin.com/raw.php?i=xMiGfbyE | tr -d "\r" > $PREFIX/adblock.sh
    chmod +x $PREFIX/adblock.sh
    to create my adblock.sh (from page 4, I think it was, of this thread).

    I used the original config from OP, but change the prefix to /opt/adblock. I mount -o bind my usb drive to opt to make plenty of space for logs and this stuff. I have ran adblock.sh and it always starts my pixelserv (also in /opt/adblock) at 192.168.1.254, I want to run pixelserv from my router to get the 1x1 block instead of the not found error.

    I have read through this thread several times, and even got sidetracked on linked threads a couple of times. I cannot figure out how to get the pixelserv to run on 192.168.1.1. Also, as someone new to working with custom router firmware, this thread could use some help. It's a little confusing.

    I have it blocking some ads, but am unable to figure out how to track where the ads for, say, Hulu, are coming from to block them. So a good guide, or video, or tutorial would be great. I know the basics about using wireshark, but it's a little overwhelming when trying to find the packet that shows the URL the ad is loaded from.

    Appreciate the help. This will make my wife extremely happy if she can watch a ~30 minute show without 15 minutes of ads.

    <edit> wanted to add, this project is awesome, and you guys are saints for making it happen.
     
  65. HunterZ

    HunterZ LI Guru Member

    Generally you don't want to run pixelserv on your router's IP because it would mess with the router's web interface. Instead, adblock.sh sets up iptables rules to allow pixelserv to run on the router but listen on a different IP address (usually 192.168.1.254).

    Start reading from post #651 above for more info.
     
    HitheLightz likes this.
  66. Almaz

    Almaz Serious Server Member

    http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-7#post-248460
     
  67. JoeDirte

    JoeDirte Serious Server Member

  68. jerrm

    jerrm Network Guru Member

    I'm not against posting as a new thread. I consider this pretty much done though. If a bug came up or there was something the script should do within the current scope, I would likely fix it but "no promises."
     
  69. bluechalk

    bluechalk Networkin' Nut Member

    I needed to make a change to adblock.sh to get it to work with TomatoUSB on an Asus WL-520GU.

    I changed line 466 from:
    echo "conf-file=$blocklist" > "$CONF"
    to:
    echo "addn-hosts=$blocklist" > "$CONF"

    From my reading of the dnsmasq man page, I believe that using conf-file is a mistake.
     
  70. jerrm

    jerrm Network Guru Member

    Using conf-file is certainly not a mistake, and is usually required. Could be a bug if it's an unusual config though.

    Questions:
    1: What EXACT version of the script. Based on the line number, I assume it is mine.
    2: What EXACT version of Tomato.
    3: Post your config file.
     
  71. bluechalk

    bluechalk Networkin' Nut Member

    It is your script, from post #379. MyTomato about page says...

    Tomato Firmware v1.28.8754 ND USB Std

    USB support integration and GUI,
    Linux kernel 2.4.37.10 and Broadcom Wireless Driver 4.150.10.29 updates,
    support for additional router models and Wireless-N mode.
    Copyright (C) 2008-2010 Fedor Kozhevnikov and Ray Van Tassle
    http://www.tomatousb.org


    Based on Tomato Firmware v1.28
    Copyright (C) 2006-2010 Jonathan Zarate
    http://www.polarcloud.com/tomato/

    Built on Tue, 30 Nov 2010 18:43:04 -0500

    The config file is attached. I added a txt extension so that I could upload it here.
     

    Attached Files:

  72. bluechalk

    bluechalk Networkin' Nut Member

  73. jerrm

    jerrm Network Guru Member

    Your config runs fine on my routers. Unless more info comes forward to indicate otherwise, I'm attributing this as an incompatibility with the four year old version of dnsmasq and/or Tomato. I have no desire to patch/debug/fix for such an old version.

    If the addn-hosts hack works for you, great, but it doesn't make sense. It is definitely not appropriate for any recent version, throwing an error to syslog for every line in the blocklist and blocking nothing.
     
  74. bluechalk

    bluechalk Networkin' Nut Member

    Jerrm:

    I forgot to tell you that I'm using LISTMODE="HOST". I changed it by editing adblock.sh. I should have put that line in the config file. Perhaps you should add this as one of the Settings near the beginning of config as a reminder.

    That LISTMODE must be the problem. Its causes your script to build a host file instead of a configuration file.
     
    Last edited: Aug 5, 2014
  75. jerrm

    jerrm Network Guru Member

    Even with LISTMODE=HOST it still functions correctly on my unit.

    The issue is the older Tomato version apparently does not look in /etc/dnsmasq/hosts for additional hosts files. The folder does not exist, so the attempt to create the link to the hosts file fails.

    dnsmasq.custom should not be created at all when LISTMODE=HOST, but the shorthand conditional is falling through to the statement when the link creation fails.

    The bug is that dnsmasq.custom is created at all in this scenario.

    The next update, if ever posted will probably address this, but possibly just with an error message and exit. I don't have any desire to code for a version incompatible with what I am running.
     
  76. shipleythump

    shipleythump Network Newbie Member

    I know enough about networking to know that part, but I don't run on the default port

    I didn't realize this was what was occuring until you mention it, but it does make sense. I feel dumb for not having thought of that myself, but I still get an error when verifying adblocking at ad.doubleclick.net. I get a not found, instead of a blank page. So I figured there was something wrong with the pixelserv.

    <Edit>
    I guess I should clarify that nothing is being blocked on hulu. I can watch the videos fine, I'm trying to block the ads. Not unblock the videos.
     
    Last edited: Aug 5, 2014
  77. JoeDirte

    JoeDirte Serious Server Member

    Hulu worked for a while when I had all the ads blocked, but they made a change months ago where (for me) my Smart TV's hulu app would crash when hulu couldn't display their ads. So I had to add hulu.com to the white list. No ads on Netflix though, just have to sit through a couple on Hulu periodically. YMMV
     
  78. shipleythump

    shipleythump Network Newbie Member

    My wife and I are poor folk. We only have dumb TV's. she watches Hulu from her computer, and netflix is ad-free regardless (a feature Hulu would do well to recognize, paying customer = no ads). I have enabled all the different host lists in the initial config (not all at the same time, as that crashed my router) and even added a list of my own but nothing seems to block anything on Hulu. So I'm trying to figure a way (my initial question) to try and track exactly where the ads are coming from so i can manually block them in dnsmasq.

    <edit>I would like to clarify also, that she doesn't actually go to hulu, we have Comcast at home, and some of the shows she watches on Xfinity are hosted on Hulu. I have gone directly to Hulu in an attempt to block ads, but nothing is blocked from either hulu hosted Xfinity shows or Hulu directly. If it's not hosted with hulu, ads are blocked (and I still think that is completely awesome!)
     
  79. Nathaniel Cowles

    Nathaniel Cowles Reformed Router Member

    Nice! I got this working. Tell me please what do you have in Dnsmasq custom configuration? I had to search a little to get the log file and am using:
    log-queries
    log-async
    log-facility=/opt/adblock/dnsmasq.log

    ...but I don't know whether log-async is needed or good, though I was looking for it to keep the log file from growing indefinitely. Thanks much for your work!

    Also, it seems the contents of /www/user/ and /www/ext/ are not persistent across reboots, and the script is deleted, so how do you deal with that?

    My WAN Up script is not running /opt/adblock/adblock.sh at boot as it should. This may be because of my Static WAN IP...

    root@mainRouter:/tmp/etc/config# ls -l
    lrwxrwxrwx 1 root root 23 Aug 9 03:41 00.adblock.shut -> /opt/adblock/adblock.sh
    lrwxrwxrwx 1 root root 23 Aug 9 03:41 99.adblock.fire -> /opt/adblock/adblock.sh

    i see that I do not have a wanup or a start script. Should I?

    Thanks again.
     
    Last edited: Aug 9, 2014
  80. HunterZ

    HunterZ LI Guru Member

    I haven't bothered with log-async or log rotation yet, as I'm logging to a large cifs share.

    Regarding /www/user, I put my web script on a cifs share and my wanup script creates a symbolic link to it in /www/user.
     
  81. Nathaniel Cowles

    Nathaniel Cowles Reformed Router Member

    I will have to look further into rotating that log and if I figure it out I'll report to here.

    I have a USB drive for /opt. Will you kindly help me along a little more with how you are implementing the web and wanup scripts? Do I need to manually create a wanup script?
     
  82. HunterZ

    HunterZ LI Guru Member

    I can't say that what I did is the best way, but under Administration->Scripts->"WAN Up" in the router's web interface I have the following:
    Code:
    # wait for mount to complete
    while [ ! -e /cifs1/wanup.sh ]
    do
            logger 'Waiting for /cifs/wanup.sh to become available' -t WANUP
            sleep 1
    done
    
    /cifs1/wanup.sh
    
    And here is what the relevant parts of my /cifs1/wanup.sh look like:
    Code:
    #!/bin/sh
    # NOTE: This script should only be run on the primary gateway!
    logger "WANUP: Starting adblock." -t WANUP
    
    # start adblock
    /cifs1/adblock/adblock.sh | logger -t WANUP
    
    # create symbolic link to web interface script
    ln -s /cifs1/routers/intertron/adblock.sh /www/user/adblock.sh
    
    logger "WANUP: Done." -t WANUP
    
    So basically the router's WAN Up script waits for the OS to mount the external storage, then calls the externally-stored wanup.sh script. I did it this way because it makes it easier to set things back up after an NVRAM reset.
     
  83. HunterZ

    HunterZ LI Guru Member

    Oh, and for log rotation, I think you can have a daily scheduler script that renames the log file, then calls something like:
    kill -USR2 `cat /var/run/dnsmasq.pid` to signal dnsmasq to restart its logging facility.

    I should probably do something like this, as my log file is almost 150MB after running for 3 weeks.

    Edit: This experiment worked:
    Code:
    mv -f dnsmasq.log dnsmasq.old ; touch dnsmasq.log ; chmod 777 dnsmasq.log ;
    kill -USR2 `cat /var/run/dnsmasq.pid`

    Edit 2:

    Okay, got something going. First, I made this script at /cifs/routers/intertron/daily.sh:
    Code:
    #!/bin/sh
    echo "Running DAILY script..."
    
    # rotate dnsmasq logs
    DNSDIR=/cifs1/adblock
    DNSLOG=${DNSDIR}/dnsmasq.log
    DNSOLD=${DNSDIR}/dnsmasq.old
    echo "Rotating dnsmasq log ${DNSLOG} to ${DNSOLD}..."
    mv -f ${DNSLOG} ${DNSOLD}
    touch ${DNSLOG}
    chmod 777 ${DNSLOG}
    kill -USR2 `cat /var/run/dnsmasq.pid`
    
    # update adblock
    echo "Updating adblock..."
    /cifs1/adblock/adblock.sh
    
    echo "Done."
    
    Then I went to Administration->Scheduler in the router's web interface and set a custom script to be run once a day using the following command:
    Code:
    /cifs1/routers/intertron/daily.sh 2>1 | logger -t DAILY
     
    Last edited: Aug 9, 2014
  84. AndreDVJ

    AndreDVJ Addicted to LI Member

    I wanted to play with this Adblock control page for a while though I couldn't figure out how to make it work. I'm not very bright poking around with scripts/code (I'm not a bright programmer at all).

    One weekend off-work and I got this working. Some stuff wasn't working as I would like and started messing around with the code (mostly its CSS) and got this:

    [​IMG]

    I used mostly Almaz's code and the blocked domain names were messed up and I couldn't figure out what the bunch of parameters passed to grep were doing, so I used HunterZ's code and took out a parenthesis not needed, and got the desired results.

    Code:
    #!/bin/sh
    adblockpath="/tmp/mnt/storage/adblock/adblock.sh" # jerrm's ADBLOCK SCRIPT
    pixelservip="192.168.1.254" # PIXELSERV'S IP ADDRESS
    scriptname="/user/adblock.sh" # THIS SCRIPT
    dnsmasqlog="/tmp/mnt/storage/logs/dnsmasq.log" # WHERE dnsmasq STORES ITS LOGS
    ######################################################
    if grep -q 'echo $(wc -l < "$blocklist") > /tmp/adscount' $adblockpath
    then
    echo ""
    else
    sed '/elog "$(wc -l < "$blocklist") unique hosts to block"/ a \echo $(wc -l < "$blocklist") > /tmp/adscount' $adblockpath > /tmp/tmp090; mv /tmp/tmp090 $adblockpath
    chmod +x $adblockpath
    fi
    REFRESHTIME=60
    NEXTACTION=""
    case $QUERY_STRING in
      force)
      REFRESHTIME=5;
      NEXTACTION="?doforce";
      ;;
      start)
      REFRESHTIME=5;
      NEXTACTION="?dostart";
      ;;
      restart)
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      stop)
      REFRESHTIME=5;
      NEXTACTION="?dostop";
      ;;
      *)
      REFRESHTIME=60;
      NEXTACTION="";
      ;;
    esac
    cat << EOF
    <!DOCTYPE html>
    <html>
    <head>
    <title>adblock status</title>
    <meta http-equiv="refresh" content="$REFRESHTIME; URL=$scriptname$NEXTACTION">
    <style type="text/css">
    body {
      margin: 0;
      height: 100%;
      overflow-y: auto;
    }
    #status {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 0px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 150px;
      height: 100px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #actions {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 150px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 150px;
      height: 100px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #time {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 300px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 100%;
      height: 100px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #blocks {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      margin: 100px 0px 0px 0px;
      display: block;
      padding-top: 20px;
      padding-left: 10px;
    }
    </style>
    </head>
    <body>
    <script>
      var count=$REFRESHTIME;
      var counter=setInterval(timer, 1000);
      function timer()
      {
      count=count-1;
      document.getElementById("timer").innerHTML=count;
      if (count <= 0)
      {
      clearInterval(counter);
      return;
      }
      }
    </script>
    EOF
    echo '<div id="status">'
    echo '<b>Adblock status:</b><br><br>'
    case $QUERY_STRING in
      force)
      echo 'starting/updating adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      doforce)
      echo 'force complete<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      $adblockpath force
      echo '</pre><p></div>'
      ;;
      start)
      echo 'starting/updating adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostart)
      echo 'start/update complete<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      $adblockpath
      rm /var/wwwext/source*
      echo '</pre><p></div>'
      ;;
      restart)
      echo 'restarting adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dorestart)
      echo 'restart complete<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      $adblockpath restart
      echo '</pre><p></div>'
      ;;
      stop)
      echo 'stopping adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostop)
      echo 'stop completed<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      $adblockpath stop
      echo '</pre><p></div>'
      ;;
      *)
      echo 'blocklist '
      if nslookup ad-clix.com | grep -q $pixelservip ; then
      echo up
      else
      echo down
      fi
      echo '<br>iptables '
      echo `iptables -L | grep -c $pixelservip`/4
      echo '<br>pixelserv '
      if ps | grep -q pixelserv ; then
      echo up
      else
      echo down
      fi
      echo '<br>hosts'
      if [ -f /tmp/adscount ];
      then
      cat /tmp/adscount
      else
      echo "!!!Restart!!!"
      fi
      echo '<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><a href='$scriptname?force'>force</a>'
      echo '<br><a href='$scriptname?start'>start/update</a>'
      echo '<br><a href='$scriptname?restart'>restart</a>'
      echo '<br><a href='$scriptname?stop'>stop</a>'
      echo '<p></div>'
      echo '<div id="blocks">'
      echo '<br>Last 100 blocked domain names:<br><pre>'
      grep -B1 $pixelservip $dnsmasqlog | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | awk '{printf("%s %s %s %-13s %s\n", $1,$2,$3,$8,$6)}' | tail -n 100
      echo '</pre><p></div>'
      ;;
    esac
    echo '<div id="time">'
    echo '<b>Time info:</b><br><br>'
    echo `uptime`
    cat << EOF
    <br>Page will automatically refresh in <span id="timer">$REFRESHTIME</span> seconds
    <br><br>...or click <a href='$scriptname'>here</a> to refresh manually
    </div>
    </body>
    </html>
    
     
    Goggy and ShinichiYao like this.
  85. Goggy

    Goggy Network Guru Member

    Me wonders for what "iptables ?/4" stands for? i have 0/4, AndeDVJ has 1/4 ...
    AdBlock as such is working
    Thx!
     
  86. Almaz

    Almaz Serious Server Member

    You are probably running an old script or pixelserv . It's time to upgrade
     
  87. HunterZ

    HunterZ LI Guru Member

    These are the 4 iptables rules I have - not sure if I added or customized any of them myself or not:
    Code:
    ACCEPT  tcp  --  anywhere  192.168.1.254  tcp dpt:www
    ACCEPT  tcp  --  anywhere  192.168.1.254  tcp dpt:https
    REJECT  tcp  --  anywhere  192.168.1.254  reject-with tcp-reset
    REJECT  all  --  anywhere  192.168.1.254  reject-with icmp-host-prohibited
    These allow HTTP (port 80) and HTTPS (port 443) to go to pixelserv, and reject everything else with various protocol-specific means that hopefully cause a quick resolution.
     
  88. bluechalk

    bluechalk Networkin' Nut Member

    If you are writing your own whitelist file, be aware that the entries there are the UNIX extended regular expressions used by egrep, not simple strings. So if you put googleadservices.com in the whitelist, it will also whitelist sites like pagead2.googleadservices.com. If you want to only whitelist that one site, the line in the whitelist needs to be:
    ^googleadservices.com$

    By the way, I added this line to make it possible to link from ads at the top of Google searches.
     
  89. Goggy

    Goggy Network Guru Member

    Ok - imho these rules were necessary with older Pixelserv-Versions. I'm experiencing no delays without this iptables rules.
    Thx for pointing this out!
     
  90. HunterZ

    HunterZ LI Guru Member

    Yeah I probably haven't updated pixelserv for a while.
     
  91. Almaz

    Almaz Serious Server Member

    That's strange, using HunterZ grep code for blocked domain names were messed up with my router using Shibby E3000 firmware. I had to add another grep line to clean it up. I guess it depends on firmware how output looks. I was thinking maybe to add toggle option and last time update. If anyone interested let me know and I'll add to the script.
     
  92. HunterZ

    HunterZ LI Guru Member

    Almaz, can you post a sample of your raw dnsmasq log? I could try to come up with a grep command that tolerates both formats.
     
  93. Almaz

    Almaz Serious Server Member

    Raw from Router Log

    Code:
    Aug 10 20:11:58 Almaz daemon.info dnsmasq[12571]: config www.geoplugin.net is 192.168.3.254
    Aug 10 20:12:14 Almaz daemon.info dnsmasq[12571]: config mgid.com is 192.168.3.254
    Aug 10 20:12:17 Almaz daemon.info dnsmasq[12571]: config s7.addthis.com is 192.168.3.254
    Aug 10 20:12:17 Almaz daemon.info dnsmasq[12571]: config www.addthis.com is 192.168.3.254
    Aug 10 20:12:17 Almaz daemon.info dnsmasq[12571]: config ads.integral-marketing.com is 192.168.3.254
    After using my grep code
    Code:
    Aug 10 20:11:58 www.geoplugin.net
    Aug 10 20:12:14 mgid.com     
    Aug 10 20:12:17 s7.addthis.com
    Aug 10 20:12:17 www.addthis.com
    Aug 10 20:12:17 ads.integral-marketing.com 
     
  94. HunterZ

    HunterZ LI Guru Member

    Almaz, how did you configure dnsmasq to generate that logging? Are you using log-queries or something else?

    My outputs look like this:
    Code:
    Aug 10 07:20:43 dnsmasq[21431]: query[A] accounts.google.com from 192.168.1.111
    Aug 10 07:20:43 dnsmasq[21431]: forwarded accounts.google.com to 75.75.75.75
    Aug 10 07:20:43 dnsmasq[21431]: reply accounts.google.com is <CNAME>
    Aug 10 07:20:43 dnsmasq[21431]: reply accounts.l.google.com is 74.125.25.84
    Aug 10 07:20:43 dnsmasq[21431]: query[A] googleads.g.doubleclick.net from 192.168.1.111
    Aug 10 07:20:43 dnsmasq[21431]: config googleads.g.doubleclick.net is 192.168.1.254
    Aug 10 07:20:44 dnsmasq[21431]: query[A] www.y-cam.com from 192.168.1.150
    Aug 10 07:20:44 dnsmasq[21431]: cached www.y-cam.com is 198.136.48.130
    ...Which is much more useful, because I am able to capture which LAN IP performed the query.
     
  95. Almaz

    Almaz Serious Server Member


    I'm using the command "log-queries" in custom dnsmasq configuration. The main difference between yours and mine that I use router hostname "Almaz" in your case it is blank, which is under Basic -> Identification.

    Here is my full output.

    Code:
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: query[A] www.geoplugin.net from 192.168.3.2
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: config www.geoplugin.net is 192.168.3.254
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: query[A] www.amazon.com from 192.168.3.2
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: forwarded www.amazon.com to 208.67.222.222
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: reply adaway.sufficientlysecure.org is 91.250.99.24
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: query[A] adaway.sufficientlysecure.org from 192.168.3.2
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: cached adaway.sufficientlysecure.org is 91.250.99.24
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: reply www.amazon.com is 176.32.98.166
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: query[A] www.geoplugin.net from 192.168.3.2
    Aug 10 22:11:00 Almaz daemon.info dnsmasq[12571]: config www.geoplugin.net is 192.168.3.254
     
  96. HunterZ

    HunterZ LI Guru Member

    That we can fix - try something like this:
    Code:
    egrep -B1 "config .* is 192.168.1.254" /cifs1/adblock/dnsmasq.log | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | tail -n 100 | sed 's|^\(.*:..:..\) .*: quer|\1 |' | awk '{printf("%s %s %s) %-13s %s\n", $1,$2,$3,$7,$5)}'
    Edit: Optimization - run tail earlier so that only the last 100 lines are manipulated by sed/awk.
     
  97. Almaz

    Almaz Serious Server Member

    Looks like the same output as before.

    Code:
    /tmp/var/log/messages-Aug 10 23:03:04) 192.168.3.2   seg.sharethis.com
    /tmp/var/log/messages-Aug 10 23:03:04) 192.168.3.2   l.sharethis.com
    /tmp/var/log/messages-Aug 10 23:03:04) 192.168.3.2   b.scorecardresearch.com
    /tmp/var/log/messages-Aug 10 23:03:04) 192.168.3.2   ad4.liverail.com
    /tmp/var/log/messages-Aug 10 23:03:04) 192.168.3.2   ping.chartbeat.net
    /tmp/var/log/messages.0-Aug 10 22:58:39) 192.168.3.2   www.geoplugin.net
    /tmp/var/log/messages.0-Aug 10 22:58:39) 192.168.3.2   www.geoplugin.net
    /tmp/var/log/messages.0-Aug 10 22:58:48) 192.168.3.31  retracker.local
     
  98. HunterZ

    HunterZ LI Guru Member

    That's exactly how it's supposed to look. It's showing you which LAN IP performed the query, followed by the domain that was blocked.

    Edit: Refer to my screenshot here.
     
  99. Almaz

    Almaz Serious Server Member


    ohh I see, that's why I added an additional command to clean the path of log files and ")".
     
  100. HunterZ

    HunterZ LI Guru Member

    I don't see the log file name as part of my output. Not sure why that's happening for you, unless you're using a wildcard or specifying multiple filenames in your initial grep command.
     

Share This Page