1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Script: Clean, Lean and Mean Adblocking

Discussion in 'Tomato Firmware' started by haarp, Apr 23, 2013.

  1. jerrm

    jerrm Network Guru Member

    Adblock is a script that starts pixelserv and creates/manages blocklists for dnsmasq. Adbock is designed for tomato to do this without needing any GUI config changes.

    With your padavan firmware dnsmasq.custom is not used, the "stop" function in adblock.sh isn't doing anything other than stopping pixelserv, and restarting dnsmasq. Keeping pixelserv running is harmless. The dnsmasq restarts are just wasting CPU cycles.
     
  2. Goggy

    Goggy Network Guru Member

    Hi!

    I noticed that the first entry in my blocklist is the pixelserv-ip without a following hostname. I'm wondering if this is correct. Here is what i mean:

    # adblock blocklist, MODE=HOST, IP=192.168.5.254, generated Sun Sep 7 19:34:56 CEST 2014
    192.168.5.254
    192.168.5.254 ---.chine-li.info
    192.168.5.254 -ads.avast.dwnldfr.com
    192.168.5.254 -reports.com-57o.net
    192.168.5.254 0-29.com

    Thx!
     
  3. leandroong

    leandroong Addicted to LI Member

    deleted
     
    Last edited: Sep 8, 2014
  4. leandroong

    leandroong Addicted to LI Member

    CGI script with dnsmasq log rotate menu
     

    Attached Files:

  5. insane66613

    insane66613 Network Newbie Member

    I added update configuration and whitelist files from my network storage that I have mounted as CIFS1. Anytime I need a site whitelisted I pull up notepad++ and add the address to the bottom of my whitelist file.

    Adblock then grabs that file and updates accordingly with a restart etc. Thanks for all the contributions, I figured I should stop being completely lazy and share my tweaks.

    We can easily add a textbox to the cgi script and append its contents to the whitelist rather than this method, I'm just a newb when it comes to linux commands so I'm not quite sure the command sequence.

    Is there a way to avoid rebuilding and restarting dnsmasq for every whitelist update? I have an E4200 running Tomato Firmware 1.28.0000 MIPSR2-121 K26 USB AIO and pixelserv 34-2.

    FYI: this is my first post so if made some newb mistake with this post, let me know I'll fix it. not trying to break any rules or piss anyone off :), and I'm not sure where the attach button is so I'll just leave this code in a codebox

    Code:
    #!/bin/sh
    adblockpath="/opt/adblock/adblock.sh"
    pixelservip="192.168.0.254" # PIXELSERV'S IP ADDRESS
    scriptname="/user/adblockweb.sh" # THIS SCRIPT
    dnsmasqlog="/opt/adblock/custom/dnsmasq.log" # WHERE dnsmasq STORES ITS LOGS
    whitelist="/cifs1/whitelist.txt" # Primary whitelist file
    whiteactive="/opt/adblock/whitelist" # active whitelist config file
    activeconfig="/opt/adblock/config" # active config file
    primaryconfig="/cifs1/adblockconfig.txt" # primary config file
    ######################################################
    cd /opt/adblock/
    echo $(wc -l < "$blocklist") > /opt/adblock/adscount
    
    REFRESHTIME=60
    NEXTACTION=""
    case $QUERY_STRING in
      force)
      REFRESHTIME=5;
      NEXTACTION="?doforce";
      ;;
      start)
      REFRESHTIME=5;
      NEXTACTION="?dostart";
      ;;
      updateconfig)
      cp -f $primaryconfig $activeconfig;
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      updatewhite)
      cp -f $whitelist $whiteactive;
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      restart)
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      stop)
      REFRESHTIME=5;
      NEXTACTION="?dostop";
      ;;
      logrotate)
      REFRESHTIME=5;
      NEXTACTION="?dologrotate";
      ;;
      *)
      REFRESHTIME=60;
      NEXTACTION="";
      ;;
    esac
    cat << EOF
    <!DOCTYPE html>
    <html>
    <head>
    <title>Quantum Advert Defender</title>
    <meta http-equiv="refresh" content="$REFRESHTIME; URL=$scriptname$NEXTACTION">
    <style type="text/css">
    body {
      margin: 0;
      height: 100%;
      overflow-y: auto;
    }
    #status {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 0px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 150px;
      height: 100px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #actions {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 150px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 150px;
      height: 100px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #time {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 300px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 100%;
      height: 120px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #blocks {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      margin: 100px 0px 0px 0px;
      display: block;
      padding-top: 20px;
      padding-left: 10px;
    }
    </style>
    </head>
    <body>
    <script>
      var count=$REFRESHTIME;
      var counter=setInterval(timer, 1000);
      function timer()
      {
      count=count-1;
      document.getElementById("timer").innerHTML=count;
      if (count <= 0)
      {
      clearInterval(counter);
      return;
      }
      }
    </script>
    EOF
    echo '<div id="status">'
    echo '<b>Adblock status:</b><br><br>'
    case $QUERY_STRING in
      force)
      echo 'starting/updating adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      doforce)
      echo 'force complete<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      $adblockpath force
      echo '</pre><p></div>'
      ;;
      start)
      echo 'starting/updating adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostart)
      echo 'start/update complete<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh $adblockpath
      echo '</pre><p></div>'
      ;;
      restart)
      echo 'restarting adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dorestart)
      echo 'restart complete<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh $adblockpath restart
      echo '</pre><p></div>'
      ;;
      stop)
      echo 'stopping adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostop)
      echo 'stop completed<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh $adblockpath stop
      echo '</pre><p></div>'
      ;;
      logrotate)
      echo 'erasing log...<p></div>'
      echo 'div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
       ;;
      dologrotate)
      echo 'log erase complete<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh /opt/adblock/create-dnsmasqlog.sh
      echo '</pre><p></div>'
      ;;
      *)
      echo 'blocklist '
      if nslookup ad-clix.com | grep -q $pixelservip ; then
      echo up
      else
      echo down
      fi
      echo '<br>iptables '
      echo `iptables -L | grep -c $pixelservip`/4
      echo '<br>pixelserv '
      if ps | grep -q pixelserv ; then
      echo up
      else
      echo down
      fi
      echo '<br>hosts'
      if [ -f /opt/adblock/adscount ];
      then
      cat /opt/adblock/adscount
      else
      echo "!!!Restart!!!"
      fi
      echo '<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><a href='$scriptname?force'>force</a>'
      echo '<br><a href='$scriptname?start'>start/update</a>'
      echo '<br><a href='$scriptname?restart'>restart</a>'
      echo '<br><a href='$scriptname?stop'>stop</a>'
      echo '<br><a href='$scriptname?logrotate'>log erase</a>'
      echo '<p></div>'
      echo '<div id="blocks">'
    echo '<br><br>Last 40 blocked domain names:<br><pre>'
      grep -B1 $pixelservip $dnsmasqlog | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | awk '{printf("%s %s %s %-13s %s\n", $1,$2,$3,$8,$6)}' | tail -n 40 | sort -r
      echo '</pre><p></div>'
      ;;
    esac
    echo '<div id="time">'
    echo '<b>Time info:</b><br><br>'
    echo `uptime`
    cat << EOF
    <br>Page will automatically refresh in <span id="timer">5</span> seconds
    <br>...click <a href='$scriptname?updatewhite'>here</a> to update whitelist, or <a href='$scriptname?updateconfig'>here</a> to update config
    <br>...or click <a href='$scriptname'>here</a> to refresh manually
    </div>
    </body>
    </html>

    [​IMG]
     
    Last edited: Sep 9, 2014
  6. jerrm

    jerrm Network Guru Member

    Please post your config and any blacklist and/or whitelist files.
     
  7. Almaz

    Almaz Serious Server Member


    line with echo $(wc -l < "$blocklist") > /opt/adblock/adscount won't work by itself
     
    insane66613 likes this.
  8. jerrm

    jerrm Network Guru Member

    You can strip the whitelist items from the already generated blocklist. Haarp's script (or my LEGACY or OPTIMIZE modes) will require a dnsmasq restart. HOST mode only requires sending HUP to dnsmasq.
     
  9. HunterZ

    HunterZ LI Guru Member

    Not sure that the < should be there.
     
  10. Almaz

    Almaz Serious Server Member

    That's how I created the script to work. If anyone wishes to make it better, be my guest.

    Code:
    if grep -q 'echo $(wc -l < "$blocklist") > /tmp/adscount' $adblockpath
    sed '/elog "$(wc -l < "$blocklist") unique hosts to block"/ a \echo $(wc -l < "$blocklist") > /tmp/adscount' $adblockpath > /tmp/tmp090; mv /tmp/tmp090 $adblockpath
    chmod +x $adblockpath
    fi
     
    insane66613 likes this.
  11. HunterZ

    HunterZ LI Guru Member

    I need to look over what everyone has contributed and then hack on it, but I'm too busy with pixelserv at the moment.

    Edit: I should get it up on Github too.
     
  12. Almaz

    Almaz Serious Server Member


    I can make it easier for you.
    1) your script
    2) my script
    * Must add an option to use internal syslog or external dnsmasq log
    * if internal then
    egrep -B1 "config .* is $pixelservip" $dnsmasqlog | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | tail -n 100 | sed 's|^\(.*:..:..\) .*: quer|\1 |' | awk '{printf("%s %s %s) %-13s %s\n", $1,$2,$3,$7,$5)}' | sed -r 's:^/tmp/var/log/messages(.0)*-::' | sed 's/[)]//'

    If external then
    grep -B1 $pixelservip $dnsmasqlog | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | awk '{printf("%s %s %s %-13s %s\n", $1,$2,$3,$8,$6)}' | tail -n 100 | sort -r

    3) Another script by insane66613
     
  13. insane66613

    insane66613 Network Newbie Member

    I knew I shouldn't have changed those lines when I was trying to integrate minor updates from the community.

    I pulled that bug from leandroong's attached script (andreDVJ-lean2.sh.txt), so anyone else take note if you're downloading that script. at least until he can edit the attachment

    here's my current config file and whitelist, which should be apparent is customized to my current setup, but can be tweaked accordingly.
    Code:
    ### Settings ###
    PIXEL_IP="254"    ## 0: disable pixelserv
            ## 1-254: last octet of IP to run pixelserv on (default=254)
    PIXEL_OPTS=""    ## additional options for pixelserv
    BRIDGE="br0"    ## bridge interface for pixelserv (default=br0)
    RAMLIST="0"    ## 1: keep blocklist in RAM (e.g. for small JFFS) (default=0)
    CONF="/etc/dnsmasq.custom"    ## dnsmasq custom config (must be sourced by dnsmasq!)
                    ## confused? then leave this be!
    pixelbin=/opt/adblock/pixelserv
    wanup=/opt/adblock/custom/wanup.sh
    schedule=""
    prefix=/opt/adblock
    listtmp=/tmp
    
    ### Sources (uncomment desired blocklists) [must be compatible to the hosts file format!] ###
    ## MVPS HOSTS (~600k) [default]:
    SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
    ## pgl.yoyo.org (~70k) [default]:
    SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
    ## Hosts File Project (~3M!):
    SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    ## hpHosts ad/tracking servers (~400k):
    SOURCES="$SOURCES http://hosts-file.net/ad_servers.txt"
    ## MalwareDomainList.com (~40k):
    SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
    
    ### Blacklist additional sites ###
    ## (add hostnames inside the quotes, space-separated, without http://) ##
    BLACKLIST=""
    
    ### Whitelist sites from blocking ###
    ## (add hostnames inside the quotes, space-separated, without http://) ##
    WHITELIST=""
    
    ### Blacklist and Whitelist files (optional) ###
    ## create the files "blacklist" and "whitelist" with your hosts, one per line ##
    ## useful if you have many hosts in these (they generate faster aswell!) ##
    
    WHITELIST (I found the blacklists in my config to be overprotective, but I didn't mind too much as my whitelist updates were simple enough)
    Code:
    ad.doubleclick.net
    ssl.google-analytics.com
    ^w88.go.com$
    a1910.g.akamai.net
    vod.espn.go.com
    espnmediacdn.edgesuite.net
    ipv6.msftncsi.com.edgesuite.net
    a978.i6g1.akamai.net
    mighty-app.appspot.com
    clients1.google.com
    clients2.google.com
    clients3.google.com
    clients4.google.com
    clients5.google.com
    clients6.google.com
    clients7.google.com
    ^ad.doubleclick.net$
    ^m.espn.go.com$
    doubleclick.net
    go.com
    x10hosting.com
    id.google.com
    ssl.google-analytics.com
    odc.weather.com
    weather.com
    google-analytics.com
    ooyala.com
    msn.com
    gorillavid.in
    microsoft.com
    skype.com
    tomshardware.com
    api.skype.com
    crackle.com
    ebayrtm.com
    ea.com
    help.ea.com
    accounts.ea.com
    www91.intel.com
    passport.net
    netflix.com
    ign.com
    gamestop.com
    esomniture.com
    anandtech.com
    getclicky.com
    zopim.com
    paypal.com
    conviva.com
    hp.com
    yoast.com
    googletagmanager.com
    csi.gstatic.com
    espn.112.2o7.net
    geo.mozilla.org
    userscripts.org
    tiqcdn.com
    github.com
    I'm sure I'll add some more features to the CGI script later if no one else gets around to it. I originally had CGI script hosted on my Network drive, but I found updates and refreshing to be extremely slow, so I migrated all live scripts, etc to a cheap 2gb flash drive that I use as my optware drive.

    following that same thinking, I moved my live dnsmasq log to the usb drive as well, assuming that logging on an external host would be a burden to the router/script.
     
  14. leandroong

    leandroong Addicted to LI Member

    I have this in my copy
    Code:
    #!/bin/sh
    adblockpath="/media/optware/adblock/adblock.sh"
    pixelservip="10.0.1.254" # PIXELSERV'S IP ADDRESS
    scriptname="andreDVJ.sh" # THIS SCRIPT
    dnsmasqlog="/media/optware/adblock/dnsmasq.log"
    adscountpath="/media/optware/adblock/adscount"
    
    # rotate dnsmasq logs
    DNSDIR="/media/optware/adblock"
    DNSLOG=$DNSDIR/dnsmasq.log
    DNSOLD=${DNSDIR}/dnsmasq.old
    
    ######################################################
    cd /media/optware/adblock
    
    echo $(wc -l < "blocklist") > $adscountpath
    
    
     
  15. Almaz

    Almaz Serious Server Member



    That's how it's suppose to be http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-8#post-250430
     
  16. leandroong

    leandroong Addicted to LI Member

    Last edited: Sep 9, 2014
  17. Almaz

    Almaz Serious Server Member

    it shouldn't recognize $blocklist. What you have just won't work but it's already hard coded in your script and probably that's why it's working. It's suppose to only work once to make an injection into Jerrm's script and that's it. All I did just to make a shortcut for you guys so you don't have to mess with Jerrm's code by modifying his code manually.
     
  18. leandroong

    leandroong Addicted to LI Member

    @insane66613, wrote
    Is there a way to avoid rebuilding and restarting dnsmasq for every whitelist update? I have an e4200 running Tomato Firmware 1.28.0000 MIPSR2-121 K26 USB AIO and pixelserv 34-2.

    Code:
      updateconfig)
      cp -f $primaryconfig $activeconfig;
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      updatewhite)
      cp -f $whitelist $whiteactive;
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
    
    Not a bad idea, what is lacking is appending additions made on the list to the adblock "blocklist". I think, this can be done.
    What I recommend is to add new lists added on file whitelist and use that as basis for:
    1. appending to blocklist
    2. include that as well in config.
    3. initialize whitelist, empty contents
    That way, no need to perform adblock start, restart.
     
  19. insane66613

    insane66613 Network Newbie Member

    I went back and fixed my script, I evened up the top borders while I was at it.

    Code:
    #!/bin/sh
    adblockpath="/opt/adblock/adblock.sh"
    pixelservip="192.168.0.254" # PIXELSERV'S IP ADDRESS
    scriptname="/user/adblockweb.sh" # THIS SCRIPT
    dnsmasqlog="/tmp/mnt/DATA/dnsmasq.log" # WHERE dnsmasq STORES ITS LOGS
    whitelist="/cifs1/whitelist.txt" # Primary whitelist file
    whiteactive="/opt/adblock/whitelist" # active whitelist config file
    activeconfig="/opt/adblock/config" # active config file
    primaryconfig="/cifs1/adblockconfig.txt" # primary config file
    ######################################################
    if grep -q 'echo $(wc -l < "$blocklist") > /opt/adblock/adscount' $adblockpath
    then
    echo ""
    else
    sed '/elog "$(wc -l < "$blocklist") unique hosts to block"/ a \echo $(wc -l < "$blocklist") > /opt/adblock/adscount' $adblockpath > /opt/adblock/tmp090; mv /opt/adblock/tmp090 $adblockpath
    REFRESHTIME=60
    fi
    NEXTACTION=""
    case $QUERY_STRING in
      force)
      REFRESHTIME=5;
      NEXTACTION="?doforce";
      ;;
      start)
      REFRESHTIME=5;
      NEXTACTION="?dostart";
      ;;
      updateconfig)
      cp -f $primaryconfig $activeconfig;
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      updatewhite)
      cp -f $whitelist $whiteactive;
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      restart)
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      stop)
      REFRESHTIME=5;
      NEXTACTION="?dostop";
      ;;
      logrotate)
      REFRESHTIME=5;
      NEXTACTION="?dologrotate";
      ;;
      *)
      REFRESHTIME=60;
      NEXTACTION="";
      ;;
    esac
    cat << EOF
    <!DOCTYPE html>
    <html>
    <head>
    <title>Quantum Advert Defender</title>
    <meta http-equiv="refresh" content="$REFRESHTIME; URL=$scriptname$NEXTACTION">
    <style type="text/css">
    body {
      margin: 0;
      height: 100%;
      overflow-y: auto;
    }
    #status {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 0px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 150px;
      height: 103px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #actions {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 150px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 150px;
      height: 103px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #time {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      display: block;
      top: 0px;
      left: 300px;
      padding-top: 10px;
      padding-bottom: 10px;
      padding-right: 10px;
      padding-left: 10px;
      width: 100%;
      height: 103px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #blocks {
      font-family: "Verdana", Verdana, sans-serif;
      font-size: 13px;
      margin: 100px 0px 0px 0px;
      display: block;
      padding-top: 20px;
      padding-left: 10px;
    }
    </style>
    </head>
    <body>
    <script>
      var count=$REFRESHTIME;
      var counter=setInterval(timer, 1000);
      function timer()
      {
      count=count-1;
      document.getElementById("timer").innerHTML=count;
      if (count <= 0)
      {
      clearInterval(counter);
      return;
      }
      }
    </script>
    EOF
    echo '<div id="status">'
    echo '<b>Adblock status:</b><br><br>'
    case $QUERY_STRING in
      force)
      echo 'starting/updating adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      doforce)
      echo 'force complete<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      $adblockpath force
      echo '</pre><p></div>'
      ;;
      start)
      echo 'starting/updating adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostart)
      echo 'start/update complete<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh $adblockpath
      echo '</pre><p></div>'
      ;;
      restart)
      echo 'restarting adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dorestart)
      echo 'restart complete<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh $adblockpath restart
      echo '</pre><p></div>'
      ;;
      stop)
      echo 'stopping adblock...<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostop)
      echo 'stop completed<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh $adblockpath stop
      echo '</pre><p></div>'
      ;;
      logrotate)
      echo 'erasing log...<p></div>'
      echo 'div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
       ;;
      dologrotate)
      echo 'log erase complete<p><pre></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      echo '<div id="blocks"><pre>'
      sh /opt/adblock/create-dnsmasqlog.sh
      echo '</pre><p></div>'
      ;;
      *)
      echo 'blocklist '
      if nslookup ad-clix.com | grep -q $pixelservip ; then
      echo up
      else
      echo down
      fi
      echo '<br>iptables '
      echo `iptables -L | grep -c $pixelservip`/4
      echo '<br>pixelserv '
      if ps | grep -q pixelserv ; then
      echo up
      else
      echo down
      fi
      if [ -f /tmp/adscount ];
      then
      cat /opt/adblock/adscount
      else
      echo "!!!Restart!!!"
      fi
      echo '<p></div>'
      echo '<div id="actions">'
      echo '<b>Adblock actions:</b><br>'
      echo '<br><a href='$scriptname?force'>force</a>'
      echo '<br><a href='$scriptname?start'>start/update</a>'
      echo '<br><a href='$scriptname?restart'>restart</a>'
      echo '<br><a href='$scriptname?stop'>stop</a>'
      echo '<br><a href='$scriptname?logrotate'>log erase</a>'
      echo '<p></div>'
      echo '<div id="blocks">'
    echo '<br><br>Last 40 blocked domain names:<br><pre>'
      grep -B1 $pixelservip $dnsmasqlog | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | awk '{printf("%s %s %s %-13s %s\n", $1,$2,$3,$8,$6)}' | tail -n 40 | sort -r
      echo '</pre><p></div>'
      ;;
    esac
    echo '<div id="time">'
    echo '<b>Time info:</b><br><br>'
    echo `uptime`
    cat << EOF
    <br>Page will automatically refresh in <span id="timer">5</span> seconds
    <br>...click <a href='$scriptname?updatewhite'>here</a> to update whitelist, or <a href='$scriptname?updateconfig'>here</a> to update config
    <br>...or click <a href='$scriptname'>here</a> to refresh manually
    </div>
    </body>
    </html>
     
  20. Almaz

    Almaz Serious Server Member


    Appending idea is great but it might not work because the list must be formatted properly for dnsmasq. Sure you can do that manually but that defeats the purpose of automation. Jerrm's optimize option was made for it which will trigger with force script or restart I believe so. I usually use force method.
     
  21. Beast

    Beast Network Guru Member

    Hi all

    I think I am using jerms mod of Harp 4.5 adblock script. In the script Default setting section note the header

    ###############################
    # Default values - can be changed in config file.# <----- can be change in config file
    ##############################

    # additional options for pixelserv
    PIXEL_OPTS=""

    Pixelserv options are also defined in the config file as PIXEL_OPTS=""

    So if you add the redirect switch -r to the script variable, will the config file overide this with it's =""

    A bit confusing to me, the script seems to say that the config file has final say as to the pixel_opts values.
    So should I comment out the pixel_opts in the config file, if I choose to define the options for pixelserv in the
    script????

    I hope this makes sense.
     
  22. HunterZ

    HunterZ LI Guru Member

    If I remember correctly, the script itself sets it to empty by default, *then* reads in the config setting as an optional override. The final result gets used by the script when launching pixelserv.

    Edit: The idea is that you're never supposed to need to modify the adblock script itself, so that updates to it can be easily installed by overwriting. Only the config file is supposed to need tweaking to suit your needs.
     
  23. Beast

    Beast Network Guru Member

    ( Ok thanks, that clears it up for me. Thanks HZ )


    Just wondering, how many of us are using which version of the original ( Clean, Lean and Mean Adblocking ) script?

    I [ think ] I am using Jerrm's,,,, Maybe Jerrm could relist his with a version ID?

    I hope I am not the only one that finds this thread hard to follow. lol
     
    Last edited: Sep 9, 2014
  24. jerrm

    jerrm Network Guru Member

    Correct. Pretty much everything in the config other than the SOURCES value has a useable (usually erring on the side of haarp compatibility) default value.

    I don't think it is reasonable to change the PIXEL_OPTS default in the published script due to backward issues with earlies pixelserv versions.

    I think there could be a discussion on what pixelserv's default should be.

    I will likely change the default for FWBRIDGE to include the loopback interface.

    I'm considering an update that will support the web interface as a relatively easy drop in add-on, but I will not be adding the interface into the script itself.
     
  25. HunterZ

    HunterZ LI Guru Member

    I believe pixelserv tries to run with reasonable settings if provided with only an IP address. I'm not sure that there are any other default settings that could be passed that would be compatible with all the different versions that people are using at this point.

    Regarding the web interface, I'm not sure what the adblock script could do to make it an official add-on, other than maybe to symlink it into the web server folder. I would recommend having it create the symlink regardless of the state of the ad-blocking, as the web interface is meant to also report when the ad-blocking is disabled.
     
  26. jerrm

    jerrm Network Guru Member

    My main goal would be to eliminate the need for all the hard coded paths. Having to edit scripts is evil. It shouldn't take long, I'll try and post this evening.
     
  27. jerrm

    jerrm Network Guru Member

    The discussion would be if pixelserv should default to "-r" behavior. I've haven't seen any real downside to it, and it makes dealing with searches much more friendly.
     
  28. HunterZ

    HunterZ LI Guru Member

    I see. Splitting out into a config file is a good idea. It may be best if it's a separate one from adblock's though.

    I don't currently enable it by default because it was an experimental feature from the LI forum community's perspective. When HZ10 is closer to being ready, it may be good to discuss in the pixelserv thread whether people would like it to be enabled by default.
     
  29. jerrm

    jerrm Network Guru Member

    My goal would be no config file needed for most of the functionality.


    Understand (and agree) on it not being the default now, but worth discussing and seeing where the consensus is.
     
  30. leandroong

    leandroong Addicted to LI Member

    Which version to follow?
    The adblock that I'm using is located from top of page 1 by haarp v4.5.
    or jerm adblock, seems to be the latest
    note: This is the reason why almaz script not working for me too.

    edit2: I think, if jerm version is the latest, it should also be posted in top of 1st page to avoid confusion.
     
  31. HunterZ

    HunterZ LI Guru Member

    This is why I was saying a few pages back that a wiki would be nice to keep people pointed at the latest version.

    Alternatively, it would help if someone could maintain a Github repo. I may do this once pixelserv development settles down, but if someone wants to do it first I wouldn't be opposed.
     
  32. jerrm

    jerrm Network Guru Member

    It's Haarp's thread. I can't change his post (and probably wouldn't if I could).
     
  33. vincom

    vincom LI Guru Member

    u could always start ur own, i have harps and ur post saved in my favs, i start w/his then finish up w/yours whenever i have to redo my adblock
     
  34. Beast

    Beast Network Guru Member

    It maybe because I'm Old and Slow..!

    I know I have been confused of late. Hard time following who is developing what script to work with which other script... and parameters where they need to be in which file, config or script.

    I have opted to use Jerrm's adblock.sh and config file dated 2014-01-05. Complemented with pixelserv from Huter Z version HZ8, untill he comes out with HZ10.

    And I would like to try the script (script to easily toggle adblock on/off and query some stats) I think it's done
    by almaz. But I am not sure for which version of adblock it's meant for.

    I assume its not for Jerrm since Jerrm made mention of it and said he mite make into a drop in addon to his adblock script.
     
  35. Almaz

    Almaz Serious Server Member

    Jerrm should start his own thread. I did say that before and I'm saying this again. His script is the latest and the best. Here is a question for Jerrm. Will you start your own thread?
    My script for GUI were made for Jerrm's script. I don't know if it'll work for harp's script. (all the credit goes to HunterZ who started the script)

    Maybe GitHub would the best.
     
  36. Almaz

    Almaz Serious Server Member

    Last edited: Sep 15, 2014
  37. marc_dc

    marc_dc Network Newbie Member

    Hi, I've got an Asus RT-AC68U with merlin wrt firmware. I'm new to these kinds of scripts. Whose adblock script should I use for my router? I assume I should go with darkknight93's script and instructions, post 525 in this forum, from January 22 of this year, since it's the only post that mentions asus-merlin firmware by name. Correct? Or is it possible to use something more recent?

    Thanks!
     
  38. vincom

    vincom LI Guru Member

    afaik this will not work w/merlins, there is a way to adblock w/merlins as its posted in the smallnet builder forum but using pixelserv and the script from this thread works better
     
  39. Drats

    Drats Network Newbie Member

    I have an RT-N16 and a RT-N66 both using Merlins latest and a modified version of Haarps script running on each router. I am still testing it out, but being not at home has made the going slow. I am also using hunterz's version HZ9 of pixelserver running with it.

    I started with darkknight93's posting with many trials and errors to get where I am. It is certainly not production ready, but seems to be working, so it is possible to use the info in this thread to get it working.

    I won't be able to give this ample time until I return home in October.

    Ray
     
  40. marc_dc

    marc_dc Network Newbie Member

    Vincom, the adblock solution posted in the Smallnet builder forum won't work with an RT-AC68U, as it requires the use of entware, which isn't supported by that router. But thanks for the tip!
     
  41. vincom

    vincom LI Guru Member

    k, i got the ac66u, i tried the user e2301010 adblock soltion from smallnet and it wasnt stable for me but maybe i fracked something up with the configuration
     
    Last edited: Sep 11, 2014
  42. leandroong

    leandroong Addicted to LI Member

    After installing adblock, my opendns test fails, is this normal?
     
  43. HunterZ

    HunterZ LI Guru Member

    It's possible that the "fake" malware/phishing sites that OpenDNS uses for its test are in one of the adblock lists. You could always whitelist them if you want.
     
  44. marc_dc

    marc_dc Network Newbie Member

    Vincom: Now that I did a more thorough search of the Smallnet forum, I found a solution that I think will work, here: http://forums.smallnetbuilder.com/showthread.php?t=18895&page=3, post number 21. I see the e2301010 solution, too, but if you say it didn't work, then I won't try it.
     
  45. leandroong

    leandroong Addicted to LI Member

    Thanks. It turn out missing configuration on dnsmasq setting too.Needed to add the ff also.
    Code:
    no-resolv
    server=127.0.0.1#2053
     
  46. vincom

    vincom LI Guru Member

    it sort of did, it works then craps out but idk why
     
  47. Goggy

    Goggy Network Guru Member

  48. Nathaniel Cowles

    Nathaniel Cowles Reformed Router Member

    I tried again to get the web script going, and I was successful. The script was not working when I had my dnsmasq log dir as /mnt/usb/var/log, now I installed Optware and I'm using /opt/adblock/logs. I did have to swap in HunterZs egrep statement, thanks for that. I have two IP cams that try to access this site every 11-12 seconds. They're nuts! I put it in my blacklist and now it is clogging up my last 100 blocked domain names. How can I ignore these entries?

    Sep 12 11:25:57) 192.168.1.74 www.gipcam.com
    Sep 12 11:26:02) 192.168.1.75 www.gipcam.com
    Sep 12 11:26:08) 192.168.1.74 www.gipcam.com
    Sep 12 11:26:13) 192.168.1.75 www.gipcam.com
    Sep 12 11:26:19) 192.168.1.74 www.gipcam.com

     

    Attached Files:

  49. HunterZ

    HunterZ LI Guru Member

    Nathaniel:

    Try changing:
    grep -v 'from 127.0.0.1'

    to:
    egrep -v 'from (127.0.0.1|192.168.1.74|192.168.1.75)'

    This will exclude all IP cam requests from the results, assuming that you always assign them those IP addresses.

    Alternatively, you could change it to:
    grep -v 'from 127.0.0.1' | grep -v 'www.gipcam.com'

    This will exclude www.gipcam.com from the results.
     
  50. Nathaniel Cowles

    Nathaniel Cowles Reformed Router Member

    Perfect! You are so resourceful. I went the second option.
     
  51. Nathaniel Cowles

    Nathaniel Cowles Reformed Router Member

    A new mystery now...

    My GUI WAN Up Script, /opt/adblock/WANup.sh, stopped running at startup, seemingly from installing Optware, even though it runs fine from the command line. Why might this be? I made the change trying to get the adblock web script running right. It is solely for running adblock.sh and linking the web script:

    #!/bin/sh
    # NOTE: This script should only be run on the primary gateway!
    logger "adblockWANup: Starting adblock." -t adblockWANup

    # start adblock
    /opt/adblock/adblock.sh | logger -t adblockWANup

    # create symbolic link to web interface script
    ln -s /opt/adblock/webControlAdblock.sh /www/user/webControlAdblock.sh

    logger "adblockWANup: Done." -t adblockWANup
     
  52. jerrm

    jerrm Network Guru Member

    Finally got a some time this evening open adblock back up, touch up a few things, and played with the web gui for the first time.

    One thing that became apparent looking at the logs is the number of repeated queries for the same address. Not surprising, dnsmasq's ttl for hosts file and config file addresses is 0.

    Can anyone think of any horrible consequences of setting a short ttl of 5 or 10 seconds in dnsmasq for these queries (dnsmasq directive "local-ttl")? Even a short ttl, would drastically cut down on dnsmasq queries.
     
  53. HunterZ

    HunterZ LI Guru Member

    Does ttl setting direct the client to cache the value for the specified amount of time?

    It could be an inconvenience when temporarily disabling adblock to reach a blocked site, but most OSes provide mechanisms for flushing the client-side DNS cache.

    I would think that a ttl of at least a couple of minutes may be good.
     
  54. jerrm

    jerrm Network Guru Member

    Yes, ttl would tell the client not to ask again for however long is set. Dnsmasq defaults to "0" for locally stored addresses from hosts and config files. Even a few seconds would probably cut the requests in half on based on my logs.
     
  55. leandroong

    leandroong Addicted to LI Member

    how to do that?
    Code:
    C:\Users\lean>ping yahoo.com
    
    Pinging yahoo.com [98.138.253.109] with 32 bytes of data:
    Reply from 98.138.253.109: bytes=32 time=215ms TTL=50
    Reply from 98.138.253.109: bytes=32 time=214ms TTL=50
    Reply from 98.138.253.109: bytes=32 time=217ms TTL=50
    Reply from 98.138.253.109: bytes=32 time=214ms TTL=50
    
    Ping statistics for 98.138.253.109:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 214ms, Maximum = 217ms, Average = 215ms
    found answer:
    edit /etc/sysctl.conf
    net.ipv4.ip_default_ttl =<0-255>

    edit3: I don't have that file on my padavan FW. How about increasing dnsmasq caching, from 1000 to 1500?
     
    Last edited: Sep 15, 2014
  56. jerrm

    jerrm Network Guru Member

    NO.

    dnsmasq dns ttl has nothing to do with ping or sysctl.
     
  57. leandroong

    leandroong Addicted to LI Member

    I too interested on removing duplicate queries
    edit2: I have this on router "run before router initialized"
    ..
    # neigh ipv4
    echo 256 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
    echo 1024 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
    echo 2048 > /proc/sys/net/ipv4/neigh/default/gc_thresh3
    ...
    Arer you talking about this?
     
  58. Almaz

    Almaz Serious Server Member

    Updated AdBlock Web GUI v1.51 with an option to enable and disable logging. Also it can be found on https://github.com/Almazick/AdBlock-Web-GUI Save the script as ads.sh

    Code:
    #!/bin/sh
    # AdBlock Web GUI by Almaz
    # Version: 1.51
    # Using Tomato firmware just put all the files in /var/wwwext/
    # You can access GUI by openning in browser http://routerIP/ext/ads.sh
    ###############################################################################
    adblockpath="/var/wwwext/adblock.sh"    # location of adblock.sh by Jerrm
    pixelservip="192.168.1.254"             # Pixelserv IP address  
    scriptname="ads.sh"                     # this script name
    dnsmasqlog="/tmp/var/log/messages*"     # dnsmasq log location, by default it's using syslog
    tmpfolder="/tmp"                        # location of your temp folder
    dnsmasq_external_log="n"                # for external dnsmasq log then enter "y" otherwise "n" for default syslog
    dnsmasqconf="/etc/dnsmasq.custom"       # location for dnsmasq.custom
    ###############################################################################
    if ! grep -q 'adscount' $adblockpath
    then
    sed '/elog "$(wc -l < "$blocklist") unique hosts to block"/ a \echo $(wc -l < "$blocklist") > dummyname5/adscount' $adblockpath > $tmpfolder/tmp090; mv $tmpfolder/tmp090 $adblockpath
    sed -i "s|dummyname5|"$tmpfolder"|g" $adblockpath
    chmod +x $adblockpath
    fi
    
    REFRESHTIME=60
    NEXTACTION=""
    case $QUERY_STRING in
      force)
      REFRESHTIME=5;
      NEXTACTION="?doforce";
      ;;
      dnsmasqtoggle)
      REFRESHTIME=5;
      NEXTACTION="?dodnsmasqtoggle";
      ;;
      start)
      REFRESHTIME=5;
      NEXTACTION="?dostart";
      ;;
      restart)
      REFRESHTIME=5;
      NEXTACTION="?dorestart";
      ;;
      stop)
      REFRESHTIME=5;
      NEXTACTION="?dostop";
      ;;
      *)
      REFRESHTIME=60;
      NEXTACTION="";
      ;;
    esac
    
    cat << EOF
    <!DOCTYPE html>
    <html>
    <head>
    <title>adblock status</title>
    <meta http-equiv="refresh" content="$REFRESHTIME; URL=$scriptname$NEXTACTION">
    <style type="text/css">
    body {
      margin: 0;
      padding: 1px 1px 1px 1px;
      height: 100%;
      overflow-y: auto;
    }
    #status {
      display: block;
      top: 0px;
      left: 0px;
      padding: 1px 1px 1px 1px;
      width: 150px;
      height: 115px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #actions {
      display: block;
      top: 0px;
      left: 150px;
      padding: 1px 1px 1px 1px;
      width: 150px;
      height: 115px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #time {
      display: block;
      top: 0px;
      left: 300px;
      padding: 1px 1px 1px 1px;
      width: 100%;
      height: 115px;
      position: fixed;
      background-color: #ffffff;
      border: 1px solid #888;
    }
    #blocks {
      float:left;
      width:50%;  
      overflow:auto;
      margin:auto;
      margin-top: 120px;
      display: block;
    }
    #blocks2 {
      float: right;
      width:50%;
      overflow:auto;
      margin-top: 120px;
      display: block;
    }
    </style>
    </head>
    <body>
    <script>
      var count=$REFRESHTIME;
      var counter=setInterval(timer, 1000);
      function timer()
      {
      count=count-1;
      document.getElementById("timer").innerHTML=count;
      if (count <= 0)
      {
      clearInterval(counter);
      return;
      }
      }
    </script>
    EOF
      echo '<div id="blocks2">'
      echo 'Not blocked last domain names:<br><pre>'
      egrep -B1 "reply .* is .*" $dnsmasqlog | grep -v 'forwarded' | grep -v 'NODATA-IPv6' | sort -r | sed '/ reply /!d;s//&\n/;s/.*\n//;:a;/ is /bb;$!{n;ba};:b;s//\n&/;P;D' | awk '!a[$0]++' | tail -n 200
      echo '<p></div>'
    echo '<div id="status">'
    echo '<b>adblock status:</b><br>'
    case $QUERY_STRING in
      force)
      echo 'starting/updating adblock...<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dodnsmasqtoggle)
      echo 'DNSMASQ LOG<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
        if grep -Fxq 'log-queries' $dnsmasqconf
        then
        touch $tmpfolder/dnsmasqtmp
        sed 's/\<log-queries\>//g' $dnsmasqconf > "$tmpfolder/dnsmasqtmp"
        mv $tmpfolder/dnsmasqtmp $dnsmasqconf
        echo ""
        echo "DNSMASQ LOG DISABLED"
        else
        echo 'log-queries' >> $dnsmasqconf
        echo ""
        echo "DNSMASQ LOG ENABLED"
        fi
      service dnsmasq restart
      echo '</pre><p></div>'
      ;;
      doforce)
      echo 'force complete<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
      $adblockpath force
      echo '</pre><p></div>'
      ;;
      start)
      echo 'starting/updating adblock...<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostart)
      echo 'start/update complete<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
      $adblockpath
      echo '</pre><p></div>'
      ;;
      restart)
      echo 'restarting adblock...<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dorestart)
      echo 'restart complete<p><pre></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
      $adblockpath restart
      echo '</pre><p></div>'
      ;;
      stop)
      echo 'stopping adblock...<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
      ;;
      dostop)
      echo 'stop completed<p><pre></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><i>please wait...</i>'
      echo '<p></div>'
    
      echo '<div id="blocks"><pre>'
      $adblockpath stop
      echo '</pre><p></div>'
      ;;
      *)
      echo 'blocklist '
      if nslookup ad-clix.com | grep -q $pixelservip ; then
      echo up
      else
      echo down
      fi
      echo '<br>iptables '
      echo `iptables -L | grep -c $pixelservip`
    
      echo '<br>pixelserv '
      if ps | grep -q pixelserv ; then
      echo up
      else
      echo down
      fi
      echo '<br>hosts'
      if [ -f $tmpfolder/adscount ];
      then
      cat $tmpfolder/adscount
      else
      echo "!!!Restart!!!"
      fi
    
      echo '<p></div>'
    
      echo '<div id="actions">'
      echo '<b>adblock actions:</b>'
      echo '<br><a href='$scriptname?force'>force</a>'
      echo '<br><a href='$scriptname?start'>start/update</a>'
      echo '<br><a href='$scriptname?restart'>restart</a>'
      echo '<br><a href='$scriptname?stop'>stop</a>'
      echo '<p></div>'
    
      echo '<div id="blocks">'
      echo 'Last blocked domain names:<br><pre>'
      if [ $dnsmasq_external_log = "n" ]
        then
            egrep -B1 "config .* is $pixelservip" $dnsmasqlog | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | tail -n 100 | sed 's|^\(.*:..:..\) .*: quer|\1 |' | awk '{printf("%s %s %s) %-13s %s\n", $1,$2,$3,$7,$5)}' | sed -r 's:^/tmp/var/log/messages(.0)*-::' | sed 's/[)]//' | sort -r
        else
            grep -B1 $pixelservip $dnsmasqlog | egrep 'query.* from ' | grep -v 'from 127.0.0.1' | awk '{printf("%s %s %s %-13s %s\n", $1,$2,$3,$8,$6)}' | tail -n 100 | sort -r
      fi
      echo '</pre><p></div>'
      ;;
    esac
    
    echo '<div id="time">'
    echo '<b>time info:</b><br>'
    echo `uptime`
    
    cat << EOF
    <br>page will automatically refresh in <span id="timer">$REFRESHTIME</span> seconds
    <br>...or click <a href='$scriptname'>here</a> to refresh manually
    EOF
    if [ -f $dnsmasqconf ];
    then
    if grep -Fxq 'log-queries' $dnsmasqconf
    then
    echo '<br>dnsmasq log enabled click to <a href='$scriptname?dnsmasqtoggle'>DISABLE</a>'
    else
    echo '<br>dnsmasq log disabled click to <a href='$scriptname?dnsmasqtoggle'>ENABLE</a>'
    fi
    fi
    cat << EOF
    </div>
    </body>
    </html>
    EOF
    EDIT: Updated to a new version 1.51

    [​IMG]
     
    Last edited: Sep 18, 2014
  59. jerrm

    jerrm Network Guru Member

    No. Neighbor tables have nothing to with DNS.

    Re-read the original message for the dnsmasq directive.
     
    koitsu likes this.
  60. HunterZ

    HunterZ LI Guru Member

    jerrm, any idea why the dnsmasq man page states that local-ttl=0 is the best setting in almost all circumstances?
     
  61. leandroong

    leandroong Addicted to LI Member

    Guys, I just discover that padavan FW has adblock built-in. Just tested it with "ad.doublick.net" url and immediately my youtube main page open. I think, this adblock should also be incorporated in tomato router FW in the future.
    ATM my blocklist is fixed. Updating will be done manually.
     
  62. jerrm

    jerrm Network Guru Member

    I would assume because dnsmaq is usually only serving local pc addresses, and immediate updates can be advantageous. With the number of queries adblock generates, I think on balance it would be better to have at least a short ttl, so a single page isn't generating multiple queries.

    I don't have ANY non-dnsmasq dns servers set to 0 for a LAN domain. I generally use 30 minutes, and if I know significant changes are coming may lower it to 5 minutes or so. Never had an issue.

    For these purposes, I'd probably go with a minute or less. There would be a significant reduction in queries, and any inconvenience from cached DNS would be minimal.
     
  63. leandroong

    leandroong Addicted to LI Member

    Interesting, the URL I enter in firewall is simply added to iptables:
    Code:
    Chain urllist (1 references)
    target     prot opt source               destination
    REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           webstr: url ad.doubleclick.net  reject-with tcp-reset
    
    
     
  64. HunterZ

    HunterZ LI Guru Member

    As I mentioned in the pixelserv thread, your firmware may not work well with pixelserv's method of dead-ending SSL (HTTPS) connection requests.

    You may instead want to try an iptables rule that rejects all port 443 connections on the pixelserv IP with a tcp-reset.
     
  65. phuklok1

    phuklok1 Network Guru Member


    totally agree, it would be great to finally add this to tomato's core functionality as a new menu item. I have a script working on a lowly wrt-54g 2.0, so it shouldn't be a problem with HW limitations.
     
  66. HunterZ

    HunterZ LI Guru Member

    Tomato can only store those kinds of settings in NVRAM, which is quite limited on most supported devices. You'd never be able to put tens of thousands of URLs in NVRAM.

    Does padavan force you to use JFFS or something?
     
  67. leandroong

    leandroong Addicted to LI Member

    I agree, very limited filter URL only. No Jffs, i'm using external usb as storage for ads.
     
  68. jerrm

    jerrm Network Guru Member

    You say this is "adblock built-in" but this is an iptables rule with a single url. What does it do with an entire list of hundreds/thousands of urls? A rule per url would be a nightmare.
     
  69. mstombs

    mstombs Network Guru Member

    And think of the overhead of parsing everything that goes through the router, which won't work for https anyway!
     
  70. leandroong

    leandroong Addicted to LI Member

    I admit, wrong declaration. Here is the screen shot of the router menu
     

    Attached Files:

  71. mstombs

    mstombs Network Guru Member

    This shouldn't get forgotten - you can check what your windows7 PC gets for dns TTL using

    Code:
    C:\Users\>nslookup -type=A -debug [test site]
    I can confirm adding "local-ttl=600" to tomato web gui custom config does work, and I think well worth doing. Only downside is time to unblock if you want to turn off adblocking temporarily, but there are commands to flush the windows cache I am sure!

    Look for these entries in the output of the above command before/after making the change

     
  72. koitsu

    koitsu Network Guru Member

    ipconfig /flushdns is the command you're looking for to clear the local resolver cache on a Windows client.
     
    HunterZ likes this.
  73. jerrm

    jerrm Network Guru Member

    Yeah, I think the trade off is worth it.

    I have a test version of adblock controlling local-ttl, log-queries, and log-facility via the adblock config file, but I think I'm against releasing it with such a feature.

    The biggest issue is that dnsmasq won't start up if multiple entries for some of the directives exist. The adblock code is reasonably smart and shouldn't let that happen during adblock's startup, but I have no control over what the user might do in the GUI while adblock is already enabled.
     
  74. Almaz

    Almaz Serious Server Member

  75. leandroong

    leandroong Addicted to LI Member

    Thanks Almaz, nice column addition (Not blocked listing).
    I just installed unbound acting as DNS server. DNS query is from dnsmasq->unbound->dnscrypt-proxy. Don't know yet beneficial effect.
     
  76. Almaz

    Almaz Serious Server Member

    You're welcome. I thought, it would be easier to figure it out how to block sites instead of looking at a complicated logs. Now it just shows a clean screen with just not blocked urls. In regards dnscrypt for home use, I think it's useless. It's really useful for public internet. By the way I updated the AdBlock Web GUI to v1.51. Fixed CSS.
     
  77. jerrm

    jerrm Network Guru Member

    Updated Adblock - with Web Status Support

    This is beta. Not as tested as it should be. Report back. I will update main post if feedback is positive.

    Two elements to this update - adblock and web ui. The web ui is a second script that can be updated or replaced without editing the main adblock script.

    The result should be a drop in replacement for my script or an unmodified Haarp. Place both adblock.sh and adblockweb.sh in whatever folder your current adblock.sh is in and the web ui should work without further configuration.

    Adblock Changes:
    Support Web UI
    Creates link at /www/user/adblock.sh (http://routerip/user/adblock.sh)
    Exports adblock settings to web script
    Runs specified web script
    Link and script can be specified in adblock's config file​
    Add loopback interface access to $redirip. Allows HunterZ's pixelserv status to work from router.
    Truncate $CONF instead of deleting it.
    Allow dnsmasq logging to be enabled from adblock config
    Add record count footer to blocklist file

    Web UI:
    Adapted (stolen) from HunterZ, Almaz. et al forum web script
    Support Adblock host and legacy/optimize modes
    Detect log file location
    Support my version of firewall rules
    Add pixelserv status
    Clean up host report grep/sed statements a little
    Move most of the code out of the html
    Only intended to be run via the main adblock script

    See the script comments for more details.

    Disclaimer - this is intended for modern Tomato. Use on any other platform at your own risk.

    EDIT: UPDATED 10/26 @ 14:00 EST. Minor Bug Fixes/Added Config Editor.

    See Post 379 for updated version. http://www.linksysinfo.org/index.ph...-and-mean-adblocking.68464/page-4#post-237877
     
    Last edited: Oct 26, 2014
    chrisanthropic, pharma and Goggy like this.
  78. HunterZ

    HunterZ LI Guru Member

    Thanks! It looks like the config file I have is probably considered to be in a legacy format. How do I go about getting a fresh config file in the native format of this new version?
     
  79. Tolocdn

    Tolocdn Network Newbie Member

    Jerrm (et al) thanks for this! Just getting back into Linux in general again so please bear/bare? with me. I've installed pixelserv and your latest scripts here and they seem to be working fine (no ads anywhere!) - but - umm how do I launch/view the webgui portion? :bashful: I have everything stored (until I get a USB key for my R7000) on my NAS/MediaServer (running Ubuntu) via cifs1/sharedfolder.
     
  80. jerrm

    jerrm Network Guru Member

    Haarps' config format is fine. I probably need to create a new sample config, but if you look at the script there is a block that defines the defaults for everything meant to be tunable.

    The only required config option should be the SOURCES line(s). The defaults for everything else should be OK in most instances.
     
  81. jerrm

    jerrm Network Guru Member

    It should be as easy as opening up a browser and entering http://you.rro.ute.rip/user/adblock.sh in the address bar, and then entering the router admin login when prompted.

    But as I said this wasn't tested as thoroughly as I normally would before submitting to the wild. If it doesn't work, report back and we will figure it out.
     
  82. Tolocdn

    Tolocdn Network Newbie Member

    When you say "user" would that be "root"? I tried http://routerip/root/adblock.sh and http://routerip/cifs1/folder/adblock.sh and get the error:

    /tmp/.wxIIQEIb: line 5: ./root/adblock.sh: not found

    and

    /tmp/.wx0EJXsR: line 5: ./cifs1/routershare/adblock.sh: not found
     
  83. HunterZ

    HunterZ LI Guru Member

    jerrm likes this.
  84. Tolocdn

    Tolocdn Network Newbie Member

  85. HunterZ

    HunterZ LI Guru Member

    Setting up the new script now. Some quick comments:
    • Tons of new options. Didn't know what they did, so I looked through the script and added descriptions to the error output:
    Code:
      "debug")
         ;;
       *)
         elog "'$p' not understood! - no action taken"
         elog "Supported actions:"
         elog "<none>  : start/update adblock"
         elog "clean  : clean script generated files"
         elog "fire  : update firewall rules"
         elog "stop  : disable adblock and automatic updates"
         elog "toggle  : toggle adblock but not automatic updates"
         elog "cron  : enable automatic updates"
         elog "force  : force update of all sources"
         elog "restart : stop and restart adblock"
         elog "update  : force update check regardless of list age"
         elog "debug  : log debug info (environment)"
      
         pexit 1
         ;;
    esac
    done
    • 'toggle' doesn't delete the cron job.
    • It seems weird that the cron stuff is done inline with the command line handler code, and that it's a command line option rather than a config one.
    • It seems to do a lot of stuff before processing the command line, including creating the web script link.
    • It may be a good idea to make start an official action. Some of the code feels a bit magical because of the implied action.
    • With log-queries=1, it logs to /var/log/messages, which is highly undesirable because it's very spammy.
    • Edit: Blocklist generation takes a lot longer than it used to. I guess I wasn't using OPTIMIZE mode before?

    Web script comments:
    • What did you guys do to my javascript countdown timer for the refresh time? It's gone! :(
    • pixelserv info always says "retrieving status..." even though pixelserv is running (?)
    • After 60 seconds, it tried to redirect to "http://192.168.1.1/user/adblock.sh'webscript" which results in a 404 error.
    • All of the action links go to "http://192.168.1.1/user/adblock.sh'" which just generate a 404 error.
    • iptables status shows a whopping 9 rules. Is that right? I may have some old rules in effect - I will reboot my router when people aren't using the WAN.
    • hosts count is 1 even though my blocklist is 62618 lines and seems to be working.
     
  86. HunterZ

    HunterZ LI Guru Member

    Also got a weird error on a restart - I guess it wiped the blocklist and then tripped over itself:
    Code:
    root@intertron:/cifs1/adblock# ./adblock.sh restart
    ADBLOCK: Running as /cifs1/adblock/adblock.sh restart
    ADBLOCK: Using config file /cifs1/adblock/config
    ADBLOCK: Requested list mode is OPTIMIZE
    ADBLOCK: Logging previously enabled
    ADBLOCK: Logging to syslog
    ADBLOCK: Creating web link /www/user/adblock.sh
    ADBLOCK: Stopping
    ADBLOCK: Restarting dnsmasq
    
    Done.
    ADBLOCK: Config or script has changed - rebuilding list
    ADBLOCK: Download starting
    ADBLOCK: Unchanged: http://hosts-file.net/ad_servers.txt (Last-Modified: Sat, 20 Sep 2014 13:46:57 GMT)
    ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Wed, 17 Sep 2014 10:13:02 GMT)
    ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Thu, 11 Sep 2014 08:04:45 GMT)
    ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Wed, 20 Aug 2014 18:15:38 GMT)
    ADBLOCK: Unchanged: http://hostsfile.mine.nu/Hosts (Last-Modified: Wed, 09 May 2012 16:09:21 GMT)
    ADBLOCK: Filters unchanged
    ADBLOCK: Blocklist does not exist
    ADBLOCK: Generating /cifs1/adblock/blocklist - OPTIMIZE mode
    ./adblock.sh: line 1209: can't open /opt/etc/adblock/blocklist: no such file
    ADBLOCK: Blocklist generated - 106 seconds
    logger: invalid option -- 1
    BusyBox v1.21.1 (2014-05-27 00:13:34 ICT) multi-call binary.
    
    Usage: logger [OPTIONS] [MESSAGE]
    
    Write MESSAGE (or stdin) to syslog
    
      -s  Log to stderr as well as the system log
      -t TAG  Log using the specified tag (defaults to user name)
      -p PRIO Priority (numeric or facility.level pair)
    
    ADBLOCK: -1 unique hosts to block
    ADBLOCK: Setting up 192.168.1.254 on br0:adblk
    ADBLOCK: Setting up pixelserv on 192.168.1.254
    pixelserv[23250]: /cifs1/adblock/pixelserv version: V35.HZ10 compiled: Sep 14 2014 21:48:54
    ADBLOCK: Writing File /etc/dnsmasq.custom
    ADBLOCK: Restarting dnsmasq
    ..........
    Done.
    ADBLOCK: Exiting /cifs1/adblock/adblock.sh 0
     
  87. jerrm

    jerrm Network Guru Member

    • toggle - I never really liked that, only kept it around for compatibility. Will think about it.
    • cron - cmd line vs config. Command line option is a standard for our internal scripts for various reasons. Doubt I will remove it, but may consider adding as a config option.
    • web link creation - could be moved, but the script doesn't actually do anything else, only read/initialize environment.
    • start - Actually agree, but (so far) have erred on the side of compatibility.
    • log-queries - if the user doesn't want syslog, then the will need to specify log-facility in dnsmasq configuration. Multiple log-facility directives will cause dnsmasq startup to hang and there is no 100% safe way to prevent that from happening if attempting to manage from the script.
    • generation time - OPTIMIZE was not in harrp's. It is slower generation, but has significant memory savings. Host is probably the best speed/memory compromise.


    • timer - countdown should be there. Not sure if it's your's or not. I picked up a version from the forum and proceeded on with it.
    • pixelserv - what do you get with http://192.168.1.1/user/adblock.sh?dopixstat from the browser?
    • 404 errors - looks like the shell is expanding the variables? Not sure why that would be.
    • iptables - that is normal, they are all under one chain.
    • host count - was the blocklist regenerated yet? Host count is pulling from the list footer.
     
  88. jerrm

    jerrm Network Guru Member

    I let a hard coded path sneak in. I will update shortly.
     
  89. HunterZ

    HunterZ LI Guru Member

    Does anyone still use it?

    Does running with the cron option also do the normal start action logic?

    I probably won't use it either way, as I use Tomato's scheduler screen to call hourly and daily scripts on cifs1, and I have an entry in the daily script that calls adblock.

    It's a bit goofy though that it ends up doing it on stop actions and such as a result. EDIT: On second thought, that makes perfect sense, as you want to know when adblock is down (duh).

    Doesn't the same concern apply to log-queries?

    Also, if the user specifies a log-facility option in the adblock config, then they should know better than to also have one in the router's config.

    I'm drastically under-utilizing my router's RAM, but I suppose an optimized list results in faster query responses too.

    It's probably related to the 404 error. I think it's causing parse errors in the javascript.

    I get a normal raw pixelserv output:
    Code:
    /cifs1/adblock/pixelserv version: V35.HZ10 compiled: Sep 14 2014 21:48:54
    25 uts, 1 req, 0 avg, 0 rmx, 0 err, 0 tmo, 0 cls, 0 nou, 0 pth, 0 nfe, 0 ufe, 0 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 0 ssl, 1 sta, 0 stt, 0 rdr
    Something is going terribly wrong. Here's a Firefox source view of the beginning of the output. Note the meta tag and the end of the pixstat() function:
    Code:
    <!DOCTYPE html>
    <html>
    <head>
    <title>adblock status</title>
    <meta http-equiv="refresh" content="60; URL=adblock.sh'
    webscript">
    
    <script>
       var count=60;
       var counter=setInterval(timer, 1000);
    
       function timer()
       {
         count=count-1;
         document.getElementById("timer").innerHTML=count;
         if (count <= 0)
         {
           clearInterval(counter);
           return;
         }
       }
    
       function pixstat()
       {
         if (window.XMLHttpRequest)
         {
           xhttp=new XMLHttpRequest();
         }
         xhttp.onreadystatechange=function()
         {
           if (xhttp.readyState==4 && xhttp.status==200)
           {
             document.getElementById("pixstat").innerHTML=xhttp.responseText;
           } else if (xhttp.readyState==4) {
             document.getElementById("pixstat").innerHTML='ERROR: Could not query status<br>';
           }
         }
         xhttp.open("GET","adblock.sh'
    webscript?dopixstat",true);
         xhttp.send();
       }
    
       window.onload = pixstat;
    
    </script>
    
    Yes, but there's a problem - here's the footer:
    Code:
    # -1 records
     
  90. jerrm

    jerrm Network Guru Member

    Last edited: Sep 27, 2014
  91. dasfast

    dasfast Network Guru Member

    Jerrm, your updated script has made the web gui functional on my end!
     
  92. HunterZ

    HunterZ LI Guru Member

    That is working much better all around, thanks.

    hosts count, countdown timer, pixelserv stats are all working now.

    I see that it also figures out that I'm logging queries to an alternate file with my existing dnsmasq config, so that's good.


    What's the magic that makes the pixelserv status show up *after* the page loads? We should use that for everything on the page that requires an external call, so that the page itself loads up as quickly as possible. It takes a really long time to load now that it has that second column of query info.

    Edit: I see - some web wizardry there to make the script call itself to do a sub-task.
     
  93. Almaz

    Almaz Serious Server Member

    With a new update for some reason blocks and blocks2 not visible at all. Here is the source page from Chrome

    Code:
    <div id="blocks">
        <span title="/var/log/messages">
         </span>
        <pre>
        /pre>
      </div>
      <div id="blocks2">
          <br>
        <pre>
        </pre>
      </div>
    </body>
    </html>
     
  94. jerrm

    jerrm Network Guru Member

    Some javascript to grab the pixelserv output in the background and update the page. Look at the pixstat javascript function.

    This was the easiest way to get the data in a non-blocking way. If I grab the status in-line with the main shell script, then if wget or pixelserv hangs it delays the entire page.

    I have the script "proxy" pixelserv in case there are issues with access to $redirip. In my case I was actually working through an ssh tunnel part of the time.

    There are other ways of accomplishing the same thing, I'm sure more "current" web devs could give pointers, I just used the methods I knew.

    And yes, the technique could be used to load the log reports, eliminate the round trips for start/dostart, etc, but honestly, this goes as far as I personally wish to take it. As an old boss used to say, programs are never "completed," you just choose to stop working on them.

    It's a separate script though, so anyone is free to expand on it.
     
  95. jerrm

    jerrm Network Guru Member

    Multiple log-queries lines won't prevent dnsmasq from starting. Multiple log-facility lines will.

    I actually had code to set log-facility in dnsmasq.custom if there were no other entries for it. The problem is if the user then subsequently adds log-facility in the GUI or elsewhere, dsnmasq will hang.

    I agree a user should know better, but if they don't they have an unusable router. I went with the premise that if they were knowledgeable enough to care, they should be able to enter their own setting in the GUI.
     
  96. jerrm

    jerrm Network Guru Member

    The question is why aren't the reports and headers in the output. That should only happen in on the "start/restart/force/stop" pages or the script thinks logging is disabled.
     
    Last edited: Sep 20, 2014
  97. Almaz

    Almaz Serious Server Member

    You are absolutely right. I forgot with my script I could toggle logging. I prefer by default logging to be disabled so it won't spam syslog and only enabled when needed. Would be an idea to add the feature which I had in my script.
     
  98. HunterZ

    HunterZ LI Guru Member

    Jerrm's new adblock release can detect if you are logging queries to a separate file from syslog via the dnsmasq log-queries + log-facility options. I posted a few days ago how to set up log rotation as well. This is the best of both worlds because you get logging that doesn't spam the syslog, plus it takes less time for the web script to extract data from when it's just dnsmasq data in the file.
     
  99. Tolocdn

    Tolocdn Network Newbie Member

    I get one error now with the new script jerrm. On start up:

    ADBLOCK: Setting up pixelserv on 192.168.1.254
    /cifs2/pixelserv: line 1: syntax error: unexpected "("

    On the left column: pixelserv: up
    On the right column: pixelserv info:
    ERROR: No response from pixelserv.

    But everything seems to run?!
     
  100. pharma

    pharma Network Guru Member

    No issues here with your new adblock script.
    Thanks Jerrm!
     

Share This Page