1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ALL-U-NEED Ad Blocking by Yaqui

Discussion in 'Tomato Firmware' started by yaqui, Jul 9, 2008.

  1. IOY2009

    IOY2009 Addicted to LI Member

    Still nothing.

    Previously I used this script and autoupdate works perfectly. I don't have enough knowledge to compare them.

    Code:
    #!/bin/sh
    ## Adblock script [Version 2.1 | 08 July 2008 | 3778 bytes]
    ##
    ## Created by Adrian Jon Kriel: root-AT-extremecooling-DOT-org
    ##
    ## tomato WAN Up script
    ##
    ## 0 = disable
    ## 1 = enable
    ## (1) = default value
    ## optimising of dnsmasq, (1)
    eval OPTDNSMASQ="1"
    ## automatic updating, (1)
    eval AUTOUPDATE="1"
    ## MVPS HOSTS ~18,500 lines, 680 Kbyte, (1)
    eval MVPSSOURCE="1"
    ## pgl.yoyo.org ~2,200 lines, 68 Kbyte, (1)
    eval PGLSOURCE="0"
    ## hosts-file.net ~53,000 lines, 1.5 Mbyte, (0)
    eval HSFSOURCE="0"
    ## Hosts File Project ~102,000 lines, 3.0 Mbyte ***6+mb free memory***, (0)
    eval HFPSOURCE="0"
    ##
    ## varibles
    ## location of temp file, (/tmp/hosts)
    eval GENFILE="/tmp/hosts"
    ## redirect ip, (0.0.0.0)
    eval REDIRECTIP="0.0.0.0"
    ## sources
    eval MVPSOURCEFILE="http://www.mvps.org/winhelp2002/hosts.txt"
    eval PGLSOURCEFILE="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts"
    eval HSFSOURCEFILE="http://www.it-mate.co.uk/downloads/hosts.txt"
    eval HFPSOURCEFILE="http://hostsfile.mine.nu/Hosts"
    
    if ping -c 1 yahoo.com ; then
    
    eval GOTSOURCE="0"
    echo "" > $GENFILE
    ## download 
    if [ "$MVPSSOURCE" = "1" ]  ; then
    if wget $MVPSOURCEFILE -O - >> $GENFILE ; then
    logger ADBLOCK Downloaded $MVPSOURCEFILE
    eval GOTSOURCE="1"
    else
    logger ADBLOCK Failed $MVPSOURCEFILE
    fi
    fi
    if [ "$PGLSOURCE" = "1" ]  ; then
    if wget $PGLSOURCEFILE -O - >> $GENFILE ; then
    logger ADBLOCK Load $PGLSOURCEFILE
    eval GOTSOURCE="1"
    else
    logger ADBLOCK Fail $PGLSOURCEFILE
    fi
    fi
    if [ "$HSFSOURCE" = "1" ]  ; then
    if wget $HSFSOURCEFILE -O - >> $GENFILE ; then
    logger ADBLOCK load $HSFSOURCEFILE
    eval GOTSOURCE="1"
    else
    logger ADBLOCK Fail $HSFSOURCEFILE
    fi
    fi
    if [ "$HFPSOURCE" = "1" ]  ; then
    if wget $HFPSOURCEFILE -O - >> $GENFILE ; then
    logger ADBLOCK Load $HFPSOURCEFILE
    eval GOTSOURCE="1"
    else
    logger ADBLOCK Fail $HFPSOURCEFILE
    fi
    fi
    
    if [ "$GOTSOURCE" = "1" ]; then
    logger ADBLOCK Got Source Files
    #FREE MEMORY!
    service dnsmasq stop
    killall -9 dnsmasq
    logger ADBLOCK Ignor Fail Safe
    ##strip source file
    sed -i -e '/^[0-9A-Za-z]/!d' $GENFILE
    sed -i -e '/%/d' $GENFILE
    sed -i -e 's/[[:cntrl:][:blank:]]//g' $GENFILE
    sed -i -e 's/^[ \t]*//;s/[ \t]*$//' $GENFILE
    
    ## dnsmasq, sanitize, optimised
    sed -i -e 's/[[:space:]]*\[.*$//'  $GENFILE
    sed -i -e 's/[[:space:]]*\].*$//'  $GENFILE
    sed -i -e '/[[:space:]]*#.*$/ s/[[:space:]]*#.*$//'  $GENFILE		
    sed -i -e '/^$/d' $GENFILE
    sed -i -e '/127.0.0.1/ s/127.0.0.1//'  $GENFILE		
    sed -i -e '/^www[0-9]./ s/^www[0-9].//'  $GENFILE		
    sed -i -e '/^www./ s/^www.//' $GENFILE
    ## remove duplicates (resource friendly)	
    cat $GENFILE | sort -u > $GENFILE.new
    mv $GENFILE.new $GENFILE
    ## format
    sed -i -e 's|$|/'$REDIRECTIP'|' $GENFILE
    sed -i -e 's|^|address=/|' $GENFILE
    ## load values from dnsmasq config
    cat /etc/dnsmasq.conf >> $GENFILE
    ## optimise dnsmasq
    if [ "$OPTDNSMASQ" = "1" ] ; then
    cat >> $GENFILE <<EOF
    cache-size=2048
    log-async=5
    EOF
    fi
    
    ## remove/whitelist websites
    ## removes 3 websites (aa.com, bb.com, cc.com) 
    ## remove the # and edit the website urls.
    sed -i -e '/aa.com/d' $GENFILE
    sed -i -e '/bb.com/d' $GENFILE
    sed -i -e '/cc.com/d' $GENFILE
    
    ## apply blacklist
    dnsmasq --conf-file=$GENFILE
    
    ## failsafe added
    dnsmasq
    logger ADBLOCK Ignor Fail Safe
    
    ## dev info
    logger ADBLOCK Unique Hosts Blocked $(awk 'END { print NR }' $GENFILE)
    else
    logger ADBLOCK Error Not Downloaded
    fi
    else
    logger ADBLOCK Error No Internet
    fi
    ## remove the generated files
    rm $GENFILE*
    ## automatic update
    if [ "$AUTOUPDATE" = "1" ] ; then
    ## script exists
    if [ -x /tmp/script_wanup.sh ] ; then
    cru a UpdateAdlist "5 3 * * 3 /tmp/script_wanup.sh >/dev/null 2>&1"
    fi
    fi
    ## the end
     
  2. mta99

    mta99 Guest

    Hi,

    I installed this script this afternoon (DNS Poison), great work gents but it's not quite as good as the Easylist powered addons for Firefox and Chrome. Can I replace one of the lists in the script with Easylist or does it require special formatting?
     
  3. bluey

    bluey Guest

    How do I exactly update the Database by hand? Just execute /tmp/ADBLOCK.sh ?
     
  4. srouquette

    srouquette Network Guru Member

    @bluey: yes, that's what the cron do.
     
  5. DAWOOD6283

    DAWOOD6283 Network Guru Member

    I have a WL520GU and I'm running Tomato with VPN and USB. I know the router has only 4 meg of memory. Is anyone running this script with that setup or would I be pushing it on memory.
     
  6. vanhh

    vanhh Network Guru Member

    hi guys,
    I'm using Yaqui DNS poisoning version 2.92. I'm trying to whitelist - my.msn.com and it didn't work. All the other sites in the whitelist worked as expected. What do I have to do to fix it?

    Thanks,
     
  7. kennyj

    kennyj Addicted to LI Member

    Is there any way to enable logging, either on Yaqui's scripts or on groosh's new hotness, so I can see which sites have been redirected to 127.0.0.1 recently? I'm having trouble figuring out how some sites are breaking, like collegehumor.com, and so far my only solution has been to disable the script as needed because there's a non-obvious culprit and simply whitelisting the main sites in question doesn't always work.
     
  8. eangulus

    eangulus Network Guru Member

    I have the code in and working perfectly excet one issue.

    I decided to make my own hosts file copying the format of one of the other ones.

    I have tried to put some entries into it and none of it is being bocked. can anyone explain why?

    Here is my hosts file: http://eangulus.homeip.net/hosts.txt
     
  9. kerbdog

    kerbdog Addicted to LI Member

    I recently updated to the new Kernel 2.6 version of Tomato - finally wireless N!
    Using 'Tomato Firmware v1.27.9047 MIPSR2-beta15 K26 USB vpn3.6'

    I was trying to update my ad blocking scripts and I found this thread and the info on 'DNS Cache Poisoning' so decided to give it a go. Noticed that in the logs it didn't say how many domains/addresses are to be blocked;
    Decided to dig around and found that this line wasn't working for some reason
    Think it was something to do with the cut command, couldn't work it out so decided to use the old nasty way :)
    Now reports fine.
    Thought I would share :)
     
  10. rhester72

    rhester72 Network Guru Member

    Your solution is the correct one, the cut solution makes a (silly) assumption on the number of spaces in the string (which will vary based on the number returned).

    Rodney
     
  11. kerbdog

    kerbdog Addicted to LI Member

    Ah great :) good to know I know something!
     
  12. windozer

    windozer Networkin' Nut Member

    I though I'd check it out. How may I disable host file # 1 and 3? in Groosh's edit?
     
  13. groosh

    groosh Addicted to LI Member

    at the top of the script, under ## Editable Variables

    There is a variable GETS="2 3"

    This instructs the script to only grab data from the second and the third hosts files.

    If you wanted to include all of the hosts files you would do:

    GETS="1 2 3 4 5 6"

    If you wanted to disable 1 and 3 you would do:

    GETS=" 2 4 5 6"
     
  14. windozer

    windozer Networkin' Nut Member

    thanks groosh

    btw I've saved the script here because it expired on paste.bin
     
  15. groosh

    groosh Addicted to LI Member

    I corrected a small typo in LBLK, 'sleep15' to 'sleep 15'. Also updated the size of each hosts, just informational. I only pull in 1 2 and 3, but f you need to reduce the amount of data that is pulled in just change the GETS line accordingly, the third (3) site has a really clean hosts file.

    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## Dnsmasq Ed. v2.92
    ##
    ## Code optimizations Oct 9 2009
    ## idea by ng12345 / implemented by srouquette
    ##
    ## Some bit overides taken from xcooling July 8 2008 2.1 build
    ##
    ## slightly modified Nov 6 2009 - Groosh
    ## autoupdate on, intel.com added to whitelist, pdl.yoyo.org url chng, added a few other lists, creation ADBLOCK.sh removed, single cln, added more logging, chng function flow
    sleep 20
    
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    GETS="1 2 3"
    S1="http://www.mvps.org/winhelp2002/hosts.txt"                           #602165B
    S2="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=hosts"   #72231B
    S3="http://someonewhocares.org/hosts/hosts"                              #229401B
    S4="http://hostsfile.mine.nu/Hosts"                                     #2641915B
    S5="http://support.it-mate.co.uk/downloads/hosts.txt"                   #3851717B
    S6="http://hosts-file.net/hphosts-partial.asp"                           #283284B
    # Enter sites in format below
    WHITE="intel.com
    webex.com
    yahoo.com"
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    
    ## FUNCTIONS
    CLR ()
    {
    rm -f $GEN
    touch $GEN
    logger ADBLOCK Clearing TEMP File
    }
    
    FMEM ()
    {
    service dnsmasq stop
    killall -9 dnsmasq
    logger ADBLOCK Unloading Dnsmaq From Memory
    }
    
    DS ()
    {
    ##(Inline grep for 127.0.0.1 & Delete DOS Format Returns)
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | grep 127.0.0.1 | tr -d "\r" >> $GEN ; then
    logger ADBLOCK Retrieved $url
    else
    logger ADBLOCK ERROR Retriving $url
    fi
    done
    }
    
    CLN ()
    {
    sed -i -e 's/[[:cntrl:][:blank:]]//g' $GEN
    sed -i -e '/\#.*$/ s/\#.*$//' $GEN
    sed -i -e '/\[.*\]/ s/\[.*\]//' $GEN
    sed -i -e '/^$/d' $GEN
    sed -i -e '/127\.0\.0\.1/ s/127\.0\.0\.1//' $GEN
    sed -i -e '/^www[0-9]*\./ s/^www[0-9]*\.//' $GEN
    sed -i -e '/^[0-9]*www[0-9]*\./ s/^[0-9]*www[0-9]*\.//' $GEN
    sed -i -e '/^www\./ s/^www\.//' $GEN
    sed -i -e '/</d' $GEN
    sed -i -e 's/^[ \t]*//;s/[ \t]*$//' $GEN
    cat $GEN | sort -u > $TMP
    mv $TMP $GEN
    rm -f $TMP
    logger ADBLOCK Blacklist Scrubbed
    }
    
    FDNSM ()
    {
    sed -i -e 's|$|/'$NIP'|' $GEN
    sed -i -e 's|^|address=/|' $GEN
    logger ADBLOCK Blacklist Alignment
    }
    
    LCFG ()
    {
    cat /etc/dnsmasq.conf >> $GEN
    logger ADBLOCK Dnsmasq Config Applied
    }
    
    LWHT ()
    {
    for site in $WHITE
    do
    sed -i -e "/$site/d" $GEN
    done
    logger ADBLOCK Whitelist Applied
    }
    
    LBLK ()
    {
    dnsmasq --conf-file=$GEN
    logger ADBLOCK Launching Dnsmasq With Blacklist
    sleep 15
    FS
    }
    
    FS ()
    {
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then
    logger ADBLOCK Dnsmasq with Blacklist RUNNING
    else
    logger ADBLOCK Dnsmasq NOT Running starting Dnsmasq without Blacklist
    service dnsmasq stop
    killall -9 dnsmasq
    dnsmasq
    fi
    }
    
    TST ()
    {
    if sed -n -e '/^address=\/ad\..*\..*\/0\.0\.0\.0$/p' $GEN ; then
    TOT=$(expr $(awk 'END { print NR }' $GEN) - 15)
    logger ADBLOCK Blacklist Format PASS
    logger ADBLOCK Blacklist Contains $TOT Entries
    LBLK
    else
    logger ADBLOCK Blacklist Format ERROR
    FS
    fi
    }
    
    ## Run Functions
    CLR
    FMEM
    DS
    CLN
    FDNSM
    LCFG
    LWHT
    TST ## Launches LBLK and/OR FS
    CLR
    
    
     
  16. vibe666

    vibe666 Network Guru Member

    hey, just wanted to give my thumbs up to Yaqui for the work on this, it's saved me a lot of Ad viewing and bandwidth. :)

    i had an idea though to help improve it and i was wondering what you thought about it.

    i read an article about adding malware blocking to adblock plus and i thought maybe it was something that could be added to the ad-block script for tomato.

    here's the article if you're interested: http://adblockplus.org/blog/blocking-malicious-sites-with-adblock-plus

    thanks :)
     
  17. jochen

    jochen LI Guru Member

    Cronjob not working

    As mentionened above the cron job is not installed because of missing quotes. Replace the line near end

    Code:
    if [[ $(cru l | grep AdUpd | cut -d "#" -f2) != "AdUpd" ]] ; then
    by

    Code:
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
     
  18. jochen

    jochen LI Guru Member

    Please can someone explain what the purpose of these lines is:

    Code:
    TST ()
    { 
    sleep 15
    if sed -n -e '/^address=\/ad\..*\..*\/0\.0\.0\.0$/p' $GEN ; then
    
    Why the sleep?
    Why printing the lines matching some pattern to stdout?
     
  19. jochen

    jochen LI Guru Member

    It looks like the address= entries in dnsmasq block the whole second-level domain,not only subdomains. If this is true too much is blocked. Then I would prefer the hosts method.
     
  20. jochen

    jochen LI Guru Member

    The host version of the script has some bugs. First it has the same cron bug than the cache poison script. Second the script appends its data to the /etc/hosts file on every run, so it's growing bigger and bigger.

    I made some changes, now it works fine for me. If someone will try it:

    put this in dnsmasq custom configuration:

    Code:
    cache-size=2048
    log-async=5
    dhcp-authoritative
    addn-hosts=/tmp/hosts
    use this as wanup script:

    Code:
    ## ALL-U-NEED Ad Blocking By YAQUI 02/15/2009
    ## Host Method Edition v2.0c
    sleep 20
    
    ## Auto Update? (Y or N)
    AUPD="Y"
    
    ## Create ADBLOCK.sh 
    ADBLK="/tmp/ADBLOCK.sh"
    touch $ADBLK
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES 
    SRC1="http://www.mvps.org/winhelp2002/hosts.txt"                      # ~612K
    SRC2="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml"  # ~72K
    SRC3="http://someonewhocares.org/hosts/hosts"                         # ~208K
    SRC4="http://hostsfile.mine.nu/Hosts"                                 # ~2.59M
    GETSRC1="Y"
    GETSRC2="Y"
    GETSRC3="N"
    GETSRC4="N"
    USEWHITELIST="Y"
    # Enter sites in format below
    WHITE="editme.com
    editme.com
    editme.com"
    
    ## DO NOT EDIT BELOW
    NULLIP="0.0.0.0"
    GEN="/tmp/gen"
    TEMP="/tmp/temp"
    HOSTS="/tmp/hosts"
    
    ## FUNCTIONS 
    CLRDATA ()
    {
    rm -f $GEN
    rm -f $TEMP
    }
    
    DLSRC1 () 
    { 
    ##(Inline grep for 127.0.0.1 & Delete DOS Format Returns)
    if [[ $GETSRC1 == "Y" ]] ; then
    if wget $SRC1 -O - | grep 127.0.0.1 | tr -d "\r" >> $GEN ; then
    logger ADBLOCK Retrieved SRC1 $SRC1
    else
    logger ADBLOCK SRC1 ERROR
    fi
    fi
    }
    
    DLSRC2 ()
    {
    if [[ $GETSRC2 == "Y" ]] ; then
    if wget $SRC2 -O - | grep 127.0.0.1 | tr -d "\r" >> $GEN ; then
    logger ADBLOCK Retrieved SRC2 $SRC2
    else
    logger ADBLOCK SRC2 ERROR
    fi
    fi
    }
    
    DLSRC3 ()
    {
    if [[ $GETSRC3 == "Y" ]] ; then
    if wget $SRC3 -O - | grep 127.0.0.1 | tr -d "\r" >> $GEN ; then
    logger ADBLOCK Retrieved SRC3 $SRC3
    else
    logger ADBLOCK SRC3 ERROR
    fi
    fi
    }
    
    DLSRC4 ()
    {
    if [[ $GETSRC4 == "Y" ]] ; then
    if wget $SRC4 -O - | grep 127.0.0.1 | tr -d "\r" >> $GEN ; then
    logger ADBLOCK Retrieved SRC4 $SRC4
    else
    logger ADBLOCK SRC4 ERROR
    fi
    fi
    }
    
    CLEANUP ()
    { 
    sed -i -e 's/[[:cntrl:][:blank:]]//g' $GEN
    sed -i -e '/\#.*$/ s/\#.*$//' $GEN
    sed -i -e '/\[.*\]/ s/\[.*\]//' $GEN
    sed -i -e '/^$/d' $GEN
    sed -i -e '/127\.0\.0\.1/ s/127\.0\.0\.1//' $GEN
    #sed -i -e '/^www[0-9]*\./ s/^www[0-9]*\.//' $GEN
    #sed -i -e '/^[0-9]*www[0-9]*\./ s/^[0-9]*www[0-9]*\.//' $GEN
    #sed -i -e '/^www\./ s/^www\.//' $GEN
    sed -i -e '/</d' $GEN
    sed -i -e 's/^[ \t]*//;s/[ \t]*$//' $GEN
    sed -i -e '/localhost/d' $GEN
    logger ADBLOCK List Cleaned
    }
    
    DELDUPES ()
    {
    touch $GEN 
    cat $GEN | sort -u > $TEMP
    mv $TEMP $GEN
    rm -f $TEMP
    }
    
    FMTHOST ()
    {
    cp $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NULLIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    }
    
    LDWLIST ()
    {
    if [[ $USEWHITELIST == "Y" ]] ; then 
    for site in $WHITE
    do
    sed -i -e "/$site/d" $GEN
    done
    logger ADBLOCK Whitelist Applied
    fi
    }
    
    LOADHOSTS ()
    { 
    killall -1 dnsmasq
    logger ADBLOCK Restarting Dnsmasq
    }
    
    TESTPOPFMT ()
    { 
    ENDTOT=`wc -l $GEN | cut -d" " -f5` 
    logger ADBLOCK Host file contains $ENDTOT Entries
    }
    
    FAILSAFE ()
    { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running Failsafe Ignored
    else
    logger ADBLOCK Dnsmasq NOT Running Dnsmasq Restarting
    service dnsmasq stop
    killall -9 dnsmasq
    dnsmasq
    fi
    }
    
    ## Run Functions 
    CLRDATA
    DLSRC1
    DLSRC2
    DLSRC3
    DLSRC4
    CLEANUP
    DELDUPES
    LDWLIST
    FMTHOST
    LOADHOSTS
    TESTPOPFMT
    FAILSAFE
    CLRDATA
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADBLK
    
    AUTOUP ()
    {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 0 * * 2 $ADBLK"
    logger ADBLOCK Cronjob added
    fi
    fi
    }
    
    ## Run ADBLOCK.sh & AUTOUP
    chmod 777 $ADBLK
    $ADBLK
    AUTOUP
     
  21. jochen

    jochen LI Guru Member

    I now did some research on the behaviour of "address=" option in dnsmaq.
    The ip given for this address is used for exactly this domain and all subdomains.
    Example: an entry for "google.co.uk" is also used for "www.google.co.uk".

    This simplifies the list if a domain is known to have ads on all their subdomains.

    But not all lists are optimized for this behaviour. Most lists are used as hosts file and contain all subdomains of an ad serving domain. The only list from the 4 in this script that is optimized for this is the pgl.yoyo.org list. The other three are for hosts file.

    Knowing this I rewrote the "dns poisoning" script and optimized it a bit.
    For whitelisting I wrote a function that can get whitelist entries from a textfile from some server. This is easier to maintain than whitelist entries directly in the script. If you dont have a server where you can place text files, then you can replace the function with the original one.

    Here is my rewrite of Yaquis DNS Poisoning script:

    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## Dnsmasq Ed. v2.92
    sleep 20
    
    ## Auto Update? (Y or N)
    AUPD="Y"
    
    ## Create ADBLOCK.sh 
    rm -f /tmp/ADBLOCK.sh
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES 
    OPTIMISE="N"
    S1="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml"  
    S2="http://myserver.com/blacklist.txt"
    S3="http://www.mvps.org/winhelp2002/hosts.txt"
    S4="http://someonewhocares.org/hosts/hosts"
    GETS1="Y"
    GETS2="N"
    GETS3="N"
    GETS4="N"
    USEWHITELIST="N"
    WURL="http://myserver.com/whitelist.txt"
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    WFILE="/tmp/white"
    
    ## FUNCTIONS 
    CLR ()
    {
    rm -f $GEN
    rm -f $TMP
    rm -f $WFILE
    }
    
    FMEM ()
    { 
    service dnsmasq stop
    killall -9 dnsmasq
    logger ADBLOCK Freeing Memory
    }
    
    GET () 
    { 
    ##(Inline grep for 127.0.0.1 & Delete DOS Format Returns)
    if [[ $1 == "Y" ]] ; then
    if wget $2 -O - | tr -d "\r" >> $GEN ; then
    logger ADBLOCK Retrieved $2
    else
    logger ADBLOCK ERROR $2
    fi
    fi
    }
    
    CLN ()
    { 
    sed -i -e 's/\#.*$//' $GEN
    sed -i -e 's/\[.*\]//' $GEN
    sed -i -e 's/127\.0\.0\.1//' $GEN
    sed -i -e 's/^www[0-9]*\.//' $GEN
    sed -i -e 's/^[0-9]*www[0-9]*\.//' $GEN
    sed -i -e 's/^www\.//' $GEN
    sed -i -e '/</d' $GEN
    sed -i -e 's/^[ \t]*//;s/[ \t]*$//' $GEN
    sed -i -e '/^$/d' $GEN
    sort -u <$GEN >$TMP
    mv $TMP $GEN
    logger ADBLOCK List Cleaned
    }
    
    FDNSM ()
    { 
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    }
    
    LCFG ()
    { 
    cat /etc/dnsmasq.conf >> $GEN
    }
    
    OPT ()
    {
    if [[ $OPTIMISE == "Y" ]] ; then 
    cat >> $GEN <<EOF
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LWHT ()
    {
    if [[ $USEWHITELIST == "Y" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Retrieved Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    logger ADBLOCK Whitelist Applied
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    }
    
    LBLK ()
    {
    dnsmasq --conf-file=$GEN
    }
    
    TST ()
    { 
    TOT=$(wc -l $GEN | sed -e 's/[^0-9]//g') 
    logger ADBLOCK $TOT Entries
    }
    
    FS ()
    { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running Failsafe Ignored
    else
    logger ADBLOCK Dnsmasq NOT Running Dnsmasq Restarting
    service dnsmasq stop
    killall -9 dnsmasq
    dnsmasq
    fi
    }
    
    ## Run Functions 
    CLR
    FMEM
    GET $GETS1 $S1
    GET $GETS2 $S2
    GET $GETS3 $S3
    GET $GETS4 $S4
    CLN
    LWHT
    TST
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP ()
    {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 2 * * * $ADB"
    logger ADBLOCK Cronjob added
    fi
    fi
    }
    
    ## Run ADBLOCK.sh & AUP
    chmod 775 $ADB
    $ADB
    AUP
    
     
  22. seraphsf

    seraphsf LI Guru Member

    jochen,

    Can you post the script where you pull the whitelist from the script itself and not from a hosted file? I tried re-writing the script but got lost some where along the lines.

    Thanks.
     
  23. srouquette

    srouquette Network Guru Member

    @jochen:
    I still have 19000+ entries with MVPS and pgl.yoyo.org, should it reduce this number with your update?

    edit:
    ok, I compared your code, you updated CLN, FDNSM and LWHT.
    I updated my script with these updates.
    Remember to update the cron if you use mine...
    I also added the MDL hosts file.

    About the whitelist, I added your update in a elif.
    I didn't test it but it should work if you use USEWHITELIST="R", R stands for remote.

    edit2:
    Updated the header to thanks everyone who participated in it (I think).
    Merged hosts file and dnsmasq.conf into one script.
    If you want to use the hosts file, USEHOSTS="Y".
    If you optimised the script (OPTIMISE="Y"), addn-hosts=/tmp/hosts will be automatically added to the conf.

    edit3:
    Remove these lines from custom DNSMasq if you use optimise:
    Code:
    cache-size=2048
    log-async=5
    dhcp-authoritative
    addn-hosts=/tmp/hosts
    


    The script
    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## 10/10/10: updated by Syl, jochen, groosh, ng12345
    ## Dnsmasq Ed. v3.0
    
    sleep 10
    
    ## Auto Update? (Y or N)
    AUPD="Y"
    
    ## Create ADBLOCK.sh 
    rm -f /tmp/ADBLOCK.sh
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES 
    OPTIMISE="Y"
    USEHOSTS="Y"
    USEWHITELIST="Y" # N/Y/R for remote
    
    GETS="1 2 7"
    S1="http://www.mvps.org/winhelp2002/hosts.txt"                           #602K
    S2="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=hosts"   #72K
    S3="http://someonewhocares.org/hosts/hosts"                              #229K
    S4="http://hostsfile.mine.nu/Hosts"                                     #2641K
    S5="http://support.it-mate.co.uk/downloads/hosts.txt"                   #3851K
    S6="http://hosts-file.net/hphosts-partial.asp"                           #283K
    S7="http://www.malwaredomainlist.com/hostslist/hosts.txt"                 #94K
    
    # Enter sites in format below
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    ## FUNCTIONS 
    CLR ()
    {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    FMEM ()
    { 
    service dnsmasq stop
    killall -9 dnsmasq
    logger ADBLOCK Freeing Memory
    }
    
    DS ()
    {
    ##(Inline grep for 127.0.0.1 & Delete DOS Format Returns)
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | grep 127.0.0.1 | tr -d "\r" >> $GEN ; then
    logger ADBLOCK Retrieved $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    CLN ()
    {
    sed -i -e 's/\#.*$//' $GEN
    sed -i -e 's/\[.*\]//' $GEN
    sed -i -e 's/127\.0\.0\.1//' $GEN
    sed -i -e 's/^www[0-9]*\.//' $GEN
    sed -i -e 's/^[0-9]*www[0-9]*\.//' $GEN
    sed -i -e 's/^www\.//' $GEN
    sed -i -e '/</d' $GEN
    sed -i -e 's/^[ \t]*//;s/[ \t]*$//' $GEN
    sed -i -e '/^$/d' $GEN
    sort -u <$GEN >$TMP
    mv $TMP $GEN
    rm -f $TMP
    logger ADBLOCK List Cleaned
    }
    
    FDNSM ()
    { 
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG ()
    { 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT ()
    {
    if [[ "$USEHOSTS" == "Y" && "$OPTIMISE" == "Y" ]] ; then 
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    elif [[ "$OPTIMISE" == "Y" ]] ; then 
    cat >> $GEN <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LWHT ()
    {
    if [[ "$USEWHITELIST" == "Y" ]] ; then 
    for site in $WHITE
    do
    sed -i -e "/$site/d" $GEN
    done
    logger ADBLOCK Whitelist Applied
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Retrieved Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    logger ADBLOCK Whitelist Applied
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    }
    
    LBLK ()
    {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    TST ()
    { 
    TOT=$(wc -l $GEN | sed -e 's/[^0-9]//g') 
    logger ADBLOCK $TOT Entries
    }
    
    FS ()
    { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running Failsafe Ignored
    else
    logger ADBLOCK Dnsmasq NOT Running Dnsmasq Restarting
    service dnsmasq stop
    killall -9 dnsmasq
    dnsmasq
    fi
    }
    
    ## Run Functions 
    CLR
    FMEM
    DS
    CLN
    LWHT
    TST
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP ()
    {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    ## Run ADBLOCK.sh & AUP
    chmod 775 $ADB
    $ADB
    AUP
    
     
  24. Toink

    Toink Network Guru Member

    @srouquette

    I tried your script. My log files shows this:

    Oct 12 14:42:30 Linksys daemon.crit dnsmasq[779]: illegal repeated keyword at line 44 of /tmp/conf
    Oct 12 14:42:30 Linksys daemon.crit dnsmasq[779]: FAILED to start up
    Oct 12 14:42:30 Linksys user.notice root: ADBLOCK Dnsmasq NOT Running Dnsmasq Restarting

    What does it mean? Thanks!

    EDIT:

    I deleted this from the custom DNSMasq and the errors are gone:

    cache-size=2048
    log-async=5
    dhcp-authoritative
    addn-hosts=/tmp/hosts
     
  25. srouquette

    srouquette Network Guru Member

    Yes, no need to put these lines in dnsmasq.conf if you use OPTIMISE (that's what it does).
    It's easier this way, everyone uses these settings, I enabled it by default.


    PS: hosts file is definitely faster than dns poisoning, it's far better now :)
     
  26. tmr250z

    tmr250z LI Guru Member

    I'm starting to question the reliability of using the Hosts file versions of these adblocking scripts. Sometimes it works, sometimes it doesn't. I've noticed the blocking working on and off with srouquette's latest version and the one here: http://www.linksysinfo.org/forums/showthread.php?t=53133.
    I'm only referring to the scripts using the Hosts file method, when using the DNS Poisoning method in srouquette's latest version, i.e. USEHOSTS="N", adblocking works all the time, every time.

    Take the following link for example:

    http://www.jdoqocy.com/click-316035...com/Product/Product.aspx?Item=N82E16820148191

    It uses jdoqocy for click tracking. That site is blocked in the MVPS Hosts file, so if the script is working properly when you click the link above Newegg won't load because www.jdoqocy.com can't be resolved first. Using the Hosts file method, it just loads Newegg without blocking jdoqocy. Using the DNS Poisoning method, jdoqocy is blocked, therefore Newegg never loads and that's how it's supposed to work.

    Another one is ad.doubleclick.net. It is used by a lot of sites to serve ads in videos. If you block the ad, the video never plays. Since the MVPS Hosts file blocks ad.doubleclick.net, the video shouldn't play, if the script is working. That's not the case with the Hosts file method; you see the ad, and then the video plays. Using the DNS Poisoning method, this doesn't happen.

    Here are some examples: http://www.cbs.com/primetime/medium/video/?pid=Z0ZmDR0hdXCsQtvStb0olOG681H45CUQ or http://www.usanetwork.com/series/whitecollar/video/fullep/index.html?id=1247506

    Is anyone else having this issue with the Hosts file method?
     
  27. FattysGoneWild

    FattysGoneWild LI Guru Member

    Working fine for me here. I even disabled ABP for these tests. I am using the script in the first link you posted. Pictures for each link you provided in order.

    1st link

    http://tinypic.com/r/2ilc9rd/7

    2nd link

    http://tinypic.com/r/2rbz321/7

    3rd link video would not even load for me.

    http://tinypic.com/r/2dlp5d5/7

    I hit these links multiple times to see if it would eventually "miss" them as you described. It nailed them every time.

     
  28. tmr250z

    tmr250z LI Guru Member

    Thank you for trying this out, FattysGoneWild.

    OK, so I've discovered that this problem with the Host file method only occurs on my MacBook Pro. Everything that should be blocked, is blocked when using my Windows 7 and Windows XP laptops. I should have tested those too before I posted above.

    But this is so strange. All three computers are on the same network, accessing the same router, so why when using the hosts file method, does the adblocking work all the time on my 2 Windows computers but only some of the time on my Mac? And why does it work every time on all three computers only if I use the DNS poisoning method?

    Could someone with a Mac test this out too so I can see if this is happening to anyone else or if it's just me?
     
  29. FattysGoneWild

    FattysGoneWild LI Guru Member

    No problem. Yeah that is weird with the Mac having a issue. Hopefully others will post and test with their Mac's.
     
  30. srouquette

    srouquette Network Guru Member

    Maybe your Mac caches DNS. There's a command to clear Windows DNS (ipconfig /flushdns) but I don't know about Mac.

    You can also intercept DNS request (Advanced -> DHCP/DNS -> Intercept DNS port), maybe your Mac has a hard DNS settings and it bypasses your router.
     
  31. tmr250z

    tmr250z LI Guru Member

    Yes, I know the command to clear Mac's cache, it's:
    Code:
    dscacheutil -flushcache
    And I've done that multiple times, before and after each test. Still nothing.

    I just checked again and there aren't any hard coded DNS settings on my Mac. Everything goes through the router. But I'll try the intercept DNS request setting on the router and report back.

    EDIT: Nope, enabling the Intercept DNS port setting didn't help, adblocking still isn't working using the Host file method on a Mac. Crap.
     
  32. ~nephelim~

    ~nephelim~ LI Guru Member

    Thanks everybody involved in the adblock development (groosh , YAQUI, ng12345, and srouquette )

    Testing the CLN () function of the latest revisions I noticed that www(00). enties were not removed because hosts line had a leading whitespace and that few additional sed lines could be removed:

    sed -i -e 's/127\.0\.0\.1[ \t]*//' $GEN whitespace removal is needed to enable www. removal

    #sed -i -e 's/\[.*\]//' $GEN # Not needed. The lines containing square brackets are preceded by # (and thus everything after is removed by sed -i -e 's/\#.*$//' $GEN )

    #sed -i -e 's/^www[0-9]*\.//' $GEN # usually not needed (the statement below match this case) might be needed for "www.www.microsoft.com.org" in http://www.mvps.org/winhelp2002/hosts.txt

    sed -i -e 's/^[0-9]*www[0-9]*\.//' $GEN remove www. 00www. www00. 00www00. and the likes from the beginning of the line

    #sed -i -e 's/^www\.//' $GEN # usually not needed (the statement above match this case www.host.com)

    #sed -i -e '/</d' $GEN # probably not needed. Remove lines containing html <tags>

    There are also some glitches with host files that include localhost (eg http://www.mvps.org/winhelp2002/hosts.txt) or additional domains (http://someonewhocares.org/hosts/hosts)

    Applying sed -e '1,/hijack-sites/ d' only to http://someonewhocares.org/hosts/hosts will workaround some of these issues for that list (it will preserve entries from hijack-sites section onward)

    http://www.mvps.org/winhelp2002/hosts.txt and the likes will need additional sed statements to remove localhosts (eg 's/^localhost//' )

    http://support.it-mate.co.uk/downloads/hosts.txt will also have a IPV6 localhost :):1 localhost # IPv6) entry to be removed. (eg 's/^::1[ \t]*//' )

    I'm currently using a fork originally derived from groosh revision and recently updated to integrate the latest sed commands.

    It is a DNS poisoning adblock containing less features than the latest ones (eg no auto update).

    It contains minimal differences regarding sed parsing, configuration validation and dnsmasq configuration startup which might be of interest.

    Code:
    GEN="/tmp/gen" 
    TMP="/tmp/temp" 
    CST="/etc/dnsmasq.custom"
    
    NIP="0.0.0.0" 
    TESTW="N"
    USEW="Y"
    WHITE="intel.com 
    feedburner.com"
     
    S1="http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml" 
    F1(){ cat $TMP >> $GEN; } 
     
    S2="http://mirror1.malwaredomains.com/files/justdomains" 
    F2(){ F1; } 
    
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"
    F3(){ F6; }
    
    S4="http://someonewhocares.org/hosts/hosts"
    F4(){ sed -e '1,/hijack-sites/ d' -e 's/\#.*$//' -e 's/^127\.0\.0\.1[ \t]*//' \
    -e 's/^[0-9]*www[0-9]*\.//' -e 's/^download\.//' -e 's/^setup\.//' -e 's/[ \t]*$//' \
    -e '/^$/d' < $TMP >> $GEN ; }
    
    S5="http://www.mvps.org/winhelp2002/hosts.txt"
    F5(){ F6; }
    
    S6="http://support.it-mate.co.uk/downloads/hosts.txt"
    F6(){  sed -e 's/\#.*$//' -e 's/'$(echo -e "\x0D")$'//' -e 's/^127\.0\.0\.1[ \t]*//' \
    -e 's/^[0-9]*www[0-9]*\.//' -e 's/^download\.//' -e 's/^setup\.//' -e 's/^::1[ \t]*//' \
    -e 's/^localhost//' -e 's/[ \t]*$//' -e '/^$/d' < $TMP >> $GEN ; }
    
    S7="http://hosts-file.net/hphosts-partial.asp"
    F7(){ F6; }
    
    S8="http://hostsfile.mine.nu/Hosts"
    F8(){ F6; }
    
    GETS="1 2" 
    
    service dnsmasq stop
    for i in $GETS; do 
    eval url="\$S$i" 
    if wget $url -O $TMP ; then 
    logger ADBLOCK: Retrieved $url 
    if [ $TESTW == "Y" ] ; then
    for site in $WHITE; do
    if grep "$site" $TMP ; then
    logger ADBLOCK: Found \"$site\" in $url
    fi
    done
    fi
    eval F$i; rm -f $TMP 
    else 
    logger ADBLOCK: ERROR Retrieving $url 
    fi 
    done 
    
    mv -f $GEN $TMP
    if [ $USEW == "Y" ] ; then
    for site in $WHITE; do
    sed -i -e "/^$site/d" $TMP
    done
    logger ADBLOCK: Whitelist Applied
    fi
    
    touch $GEN ; sort -u -o $GEN $TMP 
    logger ADBLOCK: Duplicates Removed
    rm -f $TMP 
    
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN 
    logger ADBLOCK: Blacklist Finalized 
     
    if dnsmasq --test -C $GEN ; then 
    logger ADBLOCK: Blacklist Format PASS 
    TOT=$(wc -l < $GEN) 
    logger ADBLOCK: Blacklist Contains $TOT Entries 
    echo "conf-file=$GEN" > $CST 
    else 
    logger ADBLOCK: Blacklist Format ERROR 
    fi
    logger ADBLOCK: Restarting dnsmasq 
    service dnsmasq start
    rm -f $GEN ; rm -f $CST ; touch $GEN 
    
     
  33. srouquette

    srouquette Network Guru Member

    ok, thanks for the explanation :)
    Your script seems also more optimised, you don't clean the whole file, only the ones which need it.
    I'll try to update the other script with your update.

    The problem with groosh's update is that he removed too much stuff from the original one for his needs. The cron runs the WAN Up script, you can't have other stuff in it, you have to put some settings in custom dnsmasq, it's better to keep it user friendly IMO.

    edit:
    Is there a reason you use only 's/[ \t]*$//' instead of 's/^[ \t]*//;s/[ \t]*$//' ?

    What are these lines for?
    -e 's/^download\.//' \
    -e 's/^setup\.//' \

    Thanks.

    edit:
    You line with /hijack-sites/ doesn't seem to work...
    In fact, http://someonewhocares.org/hosts/hosts also has ::1 and localhost, so we can apply F6 to it.
    Too much problem with this list, there's some characters left on some lines, I think I will drop this hosts...
     
  34. ~nephelim~

    ~nephelim~ LI Guru Member

    I started using dnsmasq.custom as a workaround to avoid direct dnsmasq invocation through dnsmasq --conf-file and use service dnsmasq start

    When dnsmasq --conf-file is used on my router Tomato QOS\Details page cannot resolve the names of the hosts on my LAN (attempting a resolve will have all host on my lan get an IP instead of a name)

    I'm currently looking into the after-effects of using dnsmasq.custom in case dnsmasq is restarted from the console after the adblock scipt is completed.

    I'll have to confirm if using dnsmasq.custom will actually append the conf-file=$GEN to /etc/dnsmasq.conf or if it is possible to restart dnsmasq at any time from the console without having a 0 byte $GEN file in place

    I moved the equivalent of s/^[ \t]*// to e 's/127\.0\.0\.1[ \t]*//' so whitespace after 127.0.0.1 is removed before sed -i -e 's/^www[0-9]*\.//' $GEN and alike.

    Using 's/127\.0\.0\.1//' will have line start with whitespace and thus ^www. sed statement will not remove the www. parts

    This statement convert CRLF to LF. I made CRLF removal optional since some lists (someonewhocares, malwaredomainlist malwaredomainlist) are already formatted for linux
    these two are optional and works in the same way of ^www. removing download. and setup. from hosts.

    I thought of removing them since if a site has some download that ought to be blocked it would probably better to attempt to completely block it.

    Though many subdomains will be left even after those sed commands.
     
  35. ~nephelim~

    ~nephelim~ LI Guru Member

    Indeed parsing that list without removing the starting portion is too much trouble

    Code:
    #<localhost>
    [B]127.0.0.1 	localhost
    127.0.0.1 	localhost.localdomain
    255.255.255.255	broadcasthost
    ::1		localhost
    127.0.0.1 	local[/B]
    #</localhost>
    
    #<shock-sites>
    # For example, to block unpleasant pages, try:
    127.0.0.1 goatse.cx       # More information on sites such as 
    127.0.0.1 www.goatse.cx   # these can be found in this article
    127.0.0.1 oralse.cx       # en.wikipedia.org/wiki/List_of_shock_sites
    127.0.0.1 www.oralse.cx
    127.0.0.1 goatse.ca
    127.0.0.1 www.goatse.ca
    127.0.0.1 oralse.ca
    127.0.0.1 www.oralse.ca
    127.0.0.1 goat.cx
    127.0.0.1 www.goat.cx
    127.0.0.1 www.copyright-reform.info
    127.0.0.1 copyright-reform.info
    127.0.0.1 cshacks.partycat.us
    127.0.0.1 lemonparty.org
    127.0.0.1 nimp.org
    127.0.0.1 on.nimp.org
    127.0.0.1 sourmath.com
    #</shock-sites>
    
    #<shortcut-examples>
    # As well by specifying the ipaddress of a server, you can gain access
    #   to some of your favourite sites with a single letter, instead of
    #   using the whole domain name
    # It is perhaps a better solution to use Favourites/Bookmarks instead.
    [B]216.34.181.45   s        # slashdot.org
    64.233.187.104	g        # google.com[/B]
    #</shortcut-examples>
    
    [B]#<hijack-sites>[/B]
    # The sites ads234.com and ads345.com -- These sites hijack internet explorer 
    # and redirect all requests through their servers. You may need to spyware 
    # removal programs such as SpyBotS&amp;D, AdAware or HijackThis to remove this 
    # nasty parasite. It's possible that blocking these sites using a hosts
    # file may have not work, in which case you should remove the following 
    # lines from this file and try the tools listed above immediately. Don't
    # forget to reboot after a scan.
    127.0.0.1 ads234.com
    127.0.0.1 ads345.com
    127.0.0.1 www.ads234.com
    127.0.0.1 www.ads345.com
    #</hijack-sites>
    
    the statement

    Code:
    sed -e '1,/hijack-sites/ d' -e 's/\#.*$//' -e 's/^127\.0\.0\.1[ \t]*//' \
    -e 's/^[0-9]*www[0-9]*\.//' -e 's/^download\.//' -e 's/^setup\.//' -e 's/[ \t]*$//' \
    -e '/^$/d' < $TMP >> $GEN ;
    should work fine but it will probably fail if inplace (sed -i) editing and multiple sed commands are used.
    As it is it, will create a new file containing the changes after applying all sed commands in one run.

    Had some issues with sed delete command (/d) myself and adjusted the syntax over multiple trial and error attempts.

    Probably the failures are caused by some timing issue.
     
  36. srouquette

    srouquette Network Guru Member

    ok, thanks :)

    I let another person takes care of this list if someone really care...

    Here is the updated version with your features:

    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## 10/10/10: updated by Syl, jochen, groosh, ng12345, ~nephelim~
    ## Dnsmasq Ed. v3.0
    
    sleep 10
    
    ## Auto Update? (Y or N)
    AUPD="Y"
    
    ## Create ADBLOCK.sh 
    rm -f /tmp/ADBLOCK.sh
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES 
    OPTIMISE="Y"
    USEHOSTS="Y"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # Enter sites in format below
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    GETS="1 2 3 4"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"                 #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"                 #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"                           #620K
    S5="http://hosts-file.net/hphosts-partial.asp"                           #283K
    S6="http://hostsfile.mine.nu/Hosts"                                     #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"                   #3851K
    
    TRIM() {
    sed -i -e 's/\#.*$//' \
    -e 's/^127\.0\.0\.1[ \t]*//' \
    -e 's/^[0-9]*www[0-9]*\.//' \
    -e 's/^download\.//' \
    -e 's/^setup\.//' \
    -e 's/[ \t]*$//' \
    -e 's/^::1[ \t]*//' \
    -e 's/^localhost//' \
    -e '/^$/d' $TMP
    }
    
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    ## FUNCTIONS 
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    FMEM() {
    service dnsmasq stop
    killall -9 dnsmasq
    logger ADBLOCK Freeing Memory
    }
    
    DS() {
    ##(Delete DOS Format Returns)
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK Retrieved $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    logger ADBLOCK Sorting List...
    sort -u -o $TMP $GEN
    logger ADBLOCK List Sorted
    if [[ "$USEWHITELIST" == "Y" ]] ; then 
    for site in $WHITE
    do
    sed -i -e "/$site/d" $TMP
    done
    logger ADBLOCK Whitelist Applied
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Retrieved Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $TMP
    done
    logger ADBLOCK Whitelist Applied
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$USEHOSTS" == "Y" && "$OPTIMISE" == "Y" ]] ; then 
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    elif [[ "$OPTIMISE" == "Y" ]] ; then 
    cat >> $GEN <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK Dnsmasq IS NOT Running, Restarting Dnsmasq...
    service dnsmasq stop
    killall -9 dnsmasq
    dnsmasq
    fi
    }
    
    ## Run Functions 
    CLR
    FMEM
    DS
    LWHT
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    ## Run ADBLOCK.sh & AUP
    chmod 775 $ADB
    $ADB
    AUP
    
     
  37. Toink

    Toink Network Guru Member

    Thank you, srouquette for the update. It works :)

    For some reason, I tried ~nephelim~'s script 5 posts up on my WRT320N and it din't work at all.....
     
  38. ~nephelim~

    ~nephelim~ LI Guru Member

    It's kind of a fitting end for a list with such hostname :biggrin:

    Thanks for considering some of the changes. :)
     
  39. ~nephelim~

    ~nephelim~ LI Guru Member

    No clue about the failure reason here as well.

    Sorry for the inconvenience it might have caused.

    In the event that my script introduced some incompatibilities I would like to advise everybody to not bother testing it at all.
     
  40. srouquette

    srouquette Network Guru Member

    I didn't test your script but it seemed fine.
    That's why I took some ideas from it :)

    Finally, the slowest part in the script is the "sort -u", I don't know if there's a way to optimise that...
     
  41. QSxx

    QSxx LI Guru Member

    Good job people,

    it works flawlessly, xcept for the tiny little problem :)

    Code:
    Oct 15 12:46:30 nodeone user.notice root: ADBLOCK Dnsmasq is Running
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21204
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21205
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21206
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21207
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21208
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21209
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21210
    Oct 15 12:46:31 nodeone daemon.err dnsmasq[4464]: bad name at /tmp/hosts line 21211
    Oct 15 12:46:32 nodeone daemon.info dnsmasq[4464]: read /tmp/hosts - 30291 addresses
    
    This would be copy of my log...

    Code:
    0.0.0.0 popupguard2.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    0.0.0.0 popupguard3.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    0.0.0.0 popupguard4.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    0.0.0.0 popupguard5.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    0.0.0.0 popupguard6.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    0.0.0.0 popupguard7.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    0.0.0.0 popupguard8.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    0.0.0.0 popupguard9.net?f5f12e30ed=%9B%EC%E0%95k%A2%A1%A0%AE%96%C8%D7%C3%E4%A1%8B%7B%B3%8B%E0%D9r%97ai%8B%A2%A3%A2%DB%ED%CF%C7W%90%D9%D3%AD%D7%CF%B0%BA%99%98%D8%AC%9A%AB%95%A5%96i%9Dig%9Ehf%C6%8A%EA%EB%D8%A5%A3%E3%AC%A1%9B%E1%A4%9B%D5%DB%D8%A4%93%AB%8B%DA%CF%A1%CBnX%DB%9C%95%DC%A1%9D%E0%E7n%60%A1%A1%9E%AA%A6%95%AE%96%A0%96ma%94%D7%D9%CCr%CE%A5%A6%D5m_%94%D0%E7%DA%E2%95%95%DB%D6%DE%A3%DE%CC%E9%90%B1%C
    
    Theese are offending lines from /tmp/hosts file

    It doesn't break the script however, so nothing too serious. I just wanted to show you this one so you can adapt trimming accordingly :)

    And btw, thanks for the gread adblock!
     
  42. srouquette

    srouquette Network Guru Member

    yep, you're right, it comes from MDL.
    I contacted them to remove this error, instead of adding another temporary rule :)

    edit: They fixed it, you can update again :)
     
  43. ~nephelim~

    ~nephelim~ LI Guru Member

    That was my intention, really. :agree:

    Not much can be done other than disabling sort-u if only one S1 or S2 list is used.
    That might be the only condition where duplicate removal would not be needed.
     
  44. srouquette

    srouquette Network Guru Member

    hehe true :)
    I may think about something in this case...
     
  45. ~nephelim~

    ~nephelim~ LI Guru Member

    Sorry I edited my post while you were replying. :oops:

    After giving a second thought duplicate removal would be needed also for a single multi-MB list if www. and alike subdomains are removed.
     
  46. srouquette

    srouquette Network Guru Member

    ah, you're right :)
    but maybe in this case, the sort removes only few entries. It's really interesting when you merge multiple lists.

    edit:
    ok... I switch to TomatoUSB yesterday, and I got an error this evening after updating to build 52.
    They added default settings to dnsmasq.conf, which are duplicates with OPTIMISE. Which means the script doesn't work on TomatoUSB.
    I rewrote OPT completely because of that.
    OPTIMISE is disabled by default now, sorry for the trouble Toink :-/

    I still append automatically addn-hosts=/tmp/hosts if you USEHOSTS.

    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## 10/10/10: updated by Syl, jochen, groosh, ng12345, ~nephelim~
    ## Dnsmasq Ed. v3.0
    
    sleep 10
    
    ## Auto Update? (Y or N)
    AUPD="Y"
    
    ## Create ADBLOCK.sh 
    rm -f /tmp/ADBLOCK.sh
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="Y"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # Enter sites in format below
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    GETS="1 2 3 4"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"                 #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"                 #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"                           #620K
    S5="http://hosts-file.net/hphosts-partial.asp"                           #283K
    S6="http://hostsfile.mine.nu/Hosts"                                     #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"                   #3851K
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    if [[ "$USEHOSTS" == "N" ]] ; then
    sed -ie '
    s/^[0-9]*www[0-9]*\.//
    s/^download\.//
    s/^setup\.//' $TMP
    fi
    }
    
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    ## FUNCTIONS 
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    FMEM() {
    service dnsmasq stop
    killall -9 dnsmasq
    logger ADBLOCK Freeing Memory
    }
    
    DS() {
    ##(Delete DOS Format Returns)
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK Retrieved $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    logger ADBLOCK Sorting List...
    sort -u -o $TMP $GEN
    logger ADBLOCK List Sorted
    if [[ "$USEWHITELIST" == "Y" ]] ; then 
    for site in $WHITE
    do
    sed -i -e "/$site/d" $TMP
    done
    logger ADBLOCK Whitelist Applied
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Retrieved Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $TMP
    done
    logger ADBLOCK Whitelist Applied
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK Dnsmasq IS NOT Running, Restarting Dnsmasq...
    service dnsmasq stop
    killall -9 dnsmasq
    dnsmasq
    fi
    }
    
    ## Run Functions 
    CLR
    FMEM
    DS
    LWHT
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    ## Run ADBLOCK.sh & AUP
    chmod 775 $ADB
    $ADB
    AUP
    
     
  47. ~nephelim~

    ~nephelim~ LI Guru Member

    Looks like sorting http://hostsfile.mine.nu/Hosts alone removed 41631 entries (nearly all from www. removals)

    While I was looking at http://hostsfile.mine.nu/Hosts output to get an idea of host duplicates I found out a glitch affecting sites like localhost.mediaonenetwork.net (it was converted into .mediaonenetwork.net)

    changing -e 's/^localhost//' \ to -e 's/localhost$//' \ solve the issue.


    Code:
    # Parsing Test containing snippets from multiple hostlists
    TMP="/tmp/test"
    {
    cat <<'ENDF'
    ######################################################################
    # http://hostsfile.mine.nu/Hosts
    127.0.0.1	localhost
    [B]127.0.0.1	localhost.mediaonenetwork.net
    127.0.0.1	localhost.utopiad.com[/B]
    127.0.0.1	www.tudoparablogs2.hpg.com.br
    
    # IAC [adware]
    
    127.0.0.1	ads.ask.com
    ######################################################################
    # http://support.it-mate.co.uk/downloads/hosts.txt
    127.0.0.1	localhost #IPv4
    ::1	localhost # IPv6
    ######################################################################
    # http://www.mvps.org/winhelp2002/hosts.txt
    127.0.0.1  localhost
    
    #start of lines added by WinHelp2002
    # [Misc A - Z]
    127.0.0.1  www.spyarsenal.com #[Spyware.DesktopSpy][Spyware.FamilyKeylog]
    # [Ntt America][199.236.0.0 - 199.239.255.255]
    127.0.0.1  intelinet-global.net #[Win32/Adware.IntelinetSmartSecurity]
    127.0.0.1  www.intelinet-global.net
    127.0.0.1  www.tsgonline.com
    127.0.0.1  voena.net #[Google.Warning]
    ######################################################################
    #               MalwareDomainList.com Hosts List           #
    #   http://www.malwaredomainlist.com/hostslist/hosts.txt   #
    #         Last updated: Fri, 15 Oct 10 18:23:24 +0000      #
    
    127.0.0.1  localhost
    ######################################################################
    # hosts-Partial.txt
    #
    # The following hosts were added to hpHosts AFTER the last full release.
    #
    #Last FULL hpHosts update: 9/22/2010
    127.0.0.1	001.bladespoon.cn
    127.0.0.1	001escort.com
    ENDF
    } > $TMP
    
    
    sed -i -e 's/\#.*$//' \
    -e 's/^127\.0\.0\.1[ \t]*//' \
    -e 's/^[0-9]*www[0-9]*\.//' \
    -e 's/^download\.//' \
    -e 's/^setup\.//' \
    -e 's/[ \t]*$//' \
    -e 's/^::1[ \t]*//' \
    -e 's/localhost$//' \
    -e '/^$/d' $TMP
    
    cat $TMP
    rm -f $TMP
    
     
  48. jochen

    jochen LI Guru Member

    That's strange. With this setting the Mac should not be able to resolve the address using some other DNS server.

    Maybe the ad is in your browser cache? Try clearing the browser cache.

    What happens when you enter the following command?
    Code:
    dig www.jdoqocy.com
    It should not be able to resolve the address.
     
  49. jochen

    jochen LI Guru Member

    I will try to explain the difference between the two methods ("hosts file" and "dns poisoning").
    With the hosts file method, every single ad host has to be named in the hosts file. "www" and other prefixes should not be removed using this method.
    With the "dns poisoning" (why this name?) method, whole domains or subdomains can be blocked, it is not necessary to list each single server.
    If you use the hosts file lists (like MVP) with "dns poisoning", you have a lot of redundancy. Maybe the hosts file method is faster for large lists, because matching is simpler. The dns poisoning method should use less resources (memory) with appropriate lists (pgl.yoyo). With the pgl.yoyo list I only have 2700 entries which should block the same than your 19000+ hosts file.

    Take a look at http://www.mvps.org/winhelp2002/hosts.txt and search for "intellitxt.com". You find a lot of entries, all blocking hosts in the same domain:
    Code:
    127.0.0.1  intellitxt.com
    127.0.0.1  de.intellitxt.com
    127.0.0.1  uk.intellitxt.com
    127.0.0.1  us.intellitxt.com
    127.0.0.1  www.intellitxt.com
    127.0.0.1  cdcovers.cc.intellitxt.com
    127.0.0.1  ad-hoc-news.de.intellitxt.com
    127.0.0.1  atspace.de.intellitxt.com
    127.0.0.1  audio.de.intellitxt.com
    127.0.0.1  awardspace.de.intellitxt.com
    127.0.0.1  bild.de.intellitxt.com
    127.0.0.1  chip.de.intellitxt.com
    127.0.0.1  computerbase.de.intellitxt.com
    127.0.0.1  computerhilfen.de.intellitxt.com
    127.0.0.1  computerwoche.de.intellitxt.com
    127.0.0.1  digital-world.de.intellitxt.com
    127.0.0.1  ghacks.de.intellitxt.com
    127.0.0.1  golem.de.intellitxt.com
    127.0.0.1  gulli.de.intellitxt.com
    127.0.0.1  images.intellitxt.com
    127.0.0.1  inquake.de.intellitxt.com
    127.0.0.1  loady.de.intellitxt.com
    127.0.0.1  macwelt.de.intellitxt.com
    127.0.0.1  msmobiles.de.intellitxt.com
    127.0.0.1  pcwelt.de.intellitxt.com
    127.0.0.1  php-mag.de.intellitxt.com
    127.0.0.1  php-magnet.de.intellitxt.com
    127.0.0.1  softonic.de.intellitxt.com
    127.0.0.1  supernature-forum.de.intellitxt.com
    127.0.0.1  supportnet.de.intellitxt.com
    127.0.0.1  tecchannel.de.intellitxt.com
    127.0.0.1  winfuture.de.intellitxt.com
    127.0.0.1  actualite-de-stars.fr.intellitxt.com
    127.0.0.1  froggytest.fr.intellitxt.com
    127.0.0.1  generation-nt.fr.intellitxt.com
    127.0.0.1  hiphopgalaxy.fr.intellitxt.com
    127.0.0.1  infos-du-net.fr.intellitxt.com
    127.0.0.1  memoclic.fr.intellitxt.com
    127.0.0.1  neteco.fr.intellitxt.com
    127.0.0.1  pcinpact.fr.intellitxt.com
    127.0.0.1  pc-infopratique.fr.intellitxt.com
    127.0.0.1  presence-pc.fr.intellitxt.com
    127.0.0.1  programme-tv.fr.intellitxt.com
    127.0.0.1  reseaux-telecoms.fr.intellitxt.com
    127.0.0.1  tomshardware.fr.intellitxt.com
    127.0.0.1  zataz.fr.intellitxt.com
    127.0.0.1  techgadgets.in.intellitxt.com
    127.0.0.1  telefonino.it.intellitxt.com
    127.0.0.1  techworld.nl.intellitxt.com
    127.0.0.1  topdownloads.nl.intellitxt.com
    127.0.0.1  webwereld.nl.intellitxt.com
    127.0.0.1  compulenta.ru.intellitxt.com
    127.0.0.1  rbmods.se.intellitxt.com
    127.0.0.1  tomshardware.se.intellitxt.com
    127.0.0.1  4thegame.uk.intellitxt.com
    127.0.0.1  amygrindhouse.uk.intellitxt.com
    127.0.0.1  anorak.uk.intellitxt.com
    127.0.0.1  bink.uk.intellitxt.com
    127.0.0.1  bit-tech.uk.intellitxt.com
    127.0.0.1  biosmagazine.uk.intellitxt.com
    127.0.0.1  cbronline.uk.intellitxt.com
    127.0.0.1  computeractive.uk.intellitxt.com
    127.0.0.1  computing.uk.intellitxt.com
    127.0.0.1  contactmusic.uk.intellitxt.com
    127.0.0.1  digit-life.uk.intellitxt.com
    127.0.0.1  efluxmedia.uk.intellitxt.com
    127.0.0.1  femalefirst.uk.intellitxt.com
    127.0.0.1  ferrago.uk.intellitxt.com
    127.0.0.1  footymad.uk.intellitxt.com
    127.0.0.1  freedownloadcenter.uk.intellitxt.com
    127.0.0.1  freedownloadmanager.uk.intellitxt.com
    127.0.0.1  freewarepalm.uk.intellitxt.com
    127.0.0.1  futurepublications.uk.intellitxt.com
    127.0.0.1  gamesindustry.uk.intellitxt.com
    127.0.0.1  hexus.uk.intellitxt.com
    127.0.0.1  itpro.uk.intellitxt.com
    127.0.0.1  itreviews.uk.intellitxt.com
    127.0.0.1  knowyourmobile.uk.intellitxt.com
    127.0.0.1  letsgodigital.uk.intellitxt.com
    127.0.0.1  lse.uk.intellitxt.com
    127.0.0.1  mad.uk.intellitxt.com
    127.0.0.1  mobilecomputermag.uk.intellitxt.com
    127.0.0.1  monstersandcritics.uk.intellitxt.com
    127.0.0.1  newlaunches.uk.intellitxt.com
    127.0.0.1  nodevice.uk.intellitxt.com
    127.0.0.1  ok.uk.intellitxt.com
    127.0.0.1  pcadvisor.uk.intellitxt.com
    127.0.0.1  pcpro.uk.intellitxt.com
    127.0.0.1  pcw.uk.intellitxt.com
    127.0.0.1  physorg.uk.intellitxt.com
    127.0.0.1  playfuls.uk.intellitxt.com
    127.0.0.1  pocketlint.uk.intellitxt.com
    127.0.0.1  product-reviews.uk.intellitxt.com
    127.0.0.1  sharecast.uk.intellitxt.com
    127.0.0.1  softpedia.uk.intellitxt.com
    127.0.0.1  squarefootball.uk.intellitxt.com
    127.0.0.1  tcmagazine.uk.intellitxt.com
    127.0.0.1  teamtalk.uk.intellitxt.com
    127.0.0.1  thehollywoodnews.uk.intellitxt.com
    127.0.0.1  theinquirer.uk.intellitxt.com
    127.0.0.1  theregister.uk.intellitxt.com
    127.0.0.1  thetechherald.uk.intellitxt.com
    127.0.0.1  vitalfootball.uk.intellitxt.com
    127.0.0.1  vnunet.uk.intellitxt.com
    127.0.0.1  webuser.uk.intellitxt.com
    127.0.0.1  wi-fitechnology.uk.intellitxt.com
    127.0.0.1  windows7news.uk.intellitxt.com
    127.0.0.1  worldtravelguide.uk.intellitxt.com
    127.0.0.1  1src.us.intellitxt.com
    127.0.0.1  1up.us.intellitxt.com
    127.0.0.1  247wallstreet.us.intellitxt.com
    127.0.0.1  2snaps.us.intellitxt.com
    127.0.0.1  2spyware.us.intellitxt.com
    127.0.0.1  24wrestling.us.intellitxt.com
    127.0.0.1  411mania.us.intellitxt.com
    127.0.0.1  4x4review.us.intellitxt.com
    127.0.0.1  4w-wrestling.us.intellitxt.com
    127.0.0.1  5starsupport.us.intellitxt.com
    127.0.0.1  9down.us.intellitxt.com
    127.0.0.1  10best.us.intellitxt.com
    127.0.0.1  able2know.us.intellitxt.com
    127.0.0.1  accuweather.us.intellitxt.com
    127.0.0.1  aceshowbiz.us.intellitxt.com
    127.0.0.1  aclasscelebs.us.intellitxt.com
    127.0.0.1  activewin.us.intellitxt.com
    127.0.0.1  actionscript.us.intellitxt.com
    127.0.0.1  advancedmn.us.intellitxt.com
    127.0.0.1  adwarereport.us.intellitxt.com
    127.0.0.1  afterdawn.us.intellitxt.com
    127.0.0.1  afraidtoask.us.intellitxt.com
    127.0.0.1  ajc.us.intellitxt.com
    127.0.0.1  akihabaranews.us.intellitxt.com
    127.0.0.1  albinoblacksheep.us.intellitxt.com
    127.0.0.1  alive.us.intellitxt.com
    127.0.0.1  allcarselectric.us.intellitxt.com
    127.0.0.1  allgetaways.us.intellitxt.com
    127.0.0.1  allhiphop.us.intellitxt.com
    127.0.0.1  allrefer.us.intellitxt.com
    127.0.0.1  amdzone.us.intellitxt.com
    127.0.0.1  americanmedia.us.intellitxt.com
    127.0.0.1  andpop.us.intellitxt.com
    127.0.0.1  androidcommunity.us.intellitxt.com
    127.0.0.1  answerbag.us.intellitxt.com
    127.0.0.1  antimusic.us.intellitxt.com
    127.0.0.1  anythinghollywood.us.intellitxt.com
    127.0.0.1  appscout.us.intellitxt.com
    127.0.0.1  artistdirect.us.intellitxt.com
    127.0.0.1  askmen.us.intellitxt.com
    127.0.0.1  askmen2.us.intellitxt.com
    127.0.0.1  aquasoft.us.intellitxt.com
    127.0.0.1  architecturaldesigns.us.intellitxt.com
    127.0.0.1  autoforums.us.intellitxt.com
    127.0.0.1  autoforumuniverse.us.intellitxt.com
    127.0.0.1  automotive.us.intellitxt.com
    127.0.0.1  autospies.us.intellitxt.com
    127.0.0.1  away.us.intellitxt.com
    127.0.0.1  aximsite.us.intellitxt.com
    127.0.0.1  b5media.us.intellitxt.com
    127.0.0.1  backseatcuddler.us.intellitxt.com
    127.0.0.1  bargainpda.us.intellitxt.com
    127.0.0.1  baselinemag.us.intellitxt.com
    127.0.0.1  bastardly.us.intellitxt.com
    127.0.0.1  beautyden.us.intellitxt.com
    127.0.0.1  beliefnet.us.intellitxt.com
    127.0.0.1  betanews.us.intellitxt.com
    127.0.0.1  beyondhollywood.us.intellitxt.com
    127.0.0.1  bigbigforums.us.intellitxt.com
    127.0.0.1  bigsoccer.us.intellitxt.com
    127.0.0.1  bittenandbound.us.intellitxt.com
    127.0.0.1  blabbermouth.us.intellitxt.com
    127.0.0.1  blastro.us.intellitxt.com
    127.0.0.1  bleepingcomputer.us.intellitxt.com
    127.0.0.1  bootdaily.us.intellitxt.com
    127.0.0.1  boxingscene.us.intellitxt.com
    127.0.0.1  boxofficeprophets.us.intellitxt.com
    127.0.0.1  bradpittnow.us.intellitxt.com
    127.0.0.1  bricksandstonesgossip.us.intellitxt.com
    127.0.0.1  britneyexperts.us.intellitxt.com
    127.0.0.1  brothersoft.us.intellitxt.com
    127.0.0.1  bukisa.us.intellitxt.com
    127.0.0.1  bullz-eye.us.intellitxt.com
    127.0.0.1  bumpshack.us.intellitxt.com
    127.0.0.1  businessknowhow.us.intellitxt.com
    127.0.0.1  buzzfoto.us.intellitxt.com
    127.0.0.1  buzzhumor.us.intellitxt.com
    127.0.0.1  bolt.us.intellitxt.com
    127.0.0.1  canmag.us.intellitxt.com
    127.0.0.1  car-stuff.us.intellitxt.com
    127.0.0.1  cavemancircus.us.intellitxt.com
    127.0.0.1  cdreviews.us.intellitxt.com
    127.0.0.1  cdrinfo.us.intellitxt.com
    127.0.0.1  cdrom-guide.us.intellitxt.com
    127.0.0.1  celebrity-babies.us.intellitxt.com
    127.0.0.1  celebrityhack.us.intellitxt.com
    127.0.0.1  celebridiot.us.intellitxt.com
    127.0.0.1  celebrifi.us.intellitxt.com
    127.0.0.1  celebritynation.us.intellitxt.com
    127.0.0.1  celebrityodor.us.intellitxt.com
    127.0.0.1  celebrity-rightpundits.us.intellitxt.com
    127.0.0.1  celebritysmackblog.us.intellitxt.com
    127.0.0.1  celebscentral.us.intellitxt.com
    127.0.0.1  celebslam.us.intellitxt.com
    127.0.0.1  celebrity-gossip.us.intellitxt.com
    127.0.0.1  celebritypwn.us.intellitxt.com
    127.0.0.1  celebritywonder.us.intellitxt.com
    127.0.0.1  channelinsider.us.intellitxt.com
    127.0.0.1  cheatcc.us.intellitxt.com
    127.0.0.1  cheatingdome.us.intellitxt.com
    127.0.0.1  cmp.us.intellitxt.com
    127.0.0.1  cnet.us.intellitxt.com
    127.0.0.1  coedmagazine.us.intellitxt.com
    127.0.0.1  collegefootballnews.us.intellitxt.com
    127.0.0.1  comicbookmovie.us.intellitxt.com
    127.0.0.1  comicbookresources.us.intellitxt.com
    127.0.0.1  comingsoon.us.intellitxt.com
    127.0.0.1  complex.us.intellitxt.com
    127.0.0.1  compnet.us.intellitxt.com
    127.0.0.1  consumerreview.us.intellitxt.com
    127.0.0.1  cooksrecipes.us.intellitxt.com
    127.0.0.1  cooltechzone.us.intellitxt.com
    127.0.0.1  countryweekly.us.intellitxt.com
    127.0.0.1  courierpostonline.us.intellitxt.com
    127.0.0.1  coxtv.us.intellitxt.com
    127.0.0.1  crmbuyer.us.intellitxt.com
    127.0.0.1  csharpcorner.us.intellitxt.com
    127.0.0.1  csnation.us.intellitxt.com
    127.0.0.1  ctv.us.intellitxt.com
    127.0.0.1  dabcc.us.intellitxt.com
    127.0.0.1  dailystab.us.intellitxt.com
    127.0.0.1  damnimcute.us.intellitxt.com
    127.0.0.1  danasdirt.us.intellitxt.com
    127.0.0.1  daniweb.us.intellitxt.com
    127.0.0.1  darkhorizons.us.intellitxt.com
    127.0.0.1  darlamack.us.intellitxt.com
    127.0.0.1  dbtechno.us.intellitxt.com
    127.0.0.1  destructoid.us.intellitxt.com
    127.0.0.1  demonews.us.intellitxt.com
    127.0.0.1  denguru.us.intellitxt.com
    127.0.0.1  derekhail.us.intellitxt.com
    127.0.0.1  devshed.us.intellitxt.com
    127.0.0.1  digitalmediaonline.us.intellitxt.com
    127.0.0.1  digitalmediawire.us.intellitxt.com
    127.0.0.1  digitaltrends.us.intellitxt.com
    127.0.0.1  dlmag.us.intellitxt.com
    127.0.0.1  dnps.us.intellitxt.com
    127.0.0.1  doubleviking.us.intellitxt.com
    127.0.0.1  drdobbs.us.intellitxt.com
    127.0.0.1  driverguide.us.intellitxt.com
    127.0.0.1  drugscom.us.intellitxt.com
    127.0.0.1  eastsideboxing.us.intellitxt.com
    127.0.0.1  ecanadanow.us.intellitxt.com
    127.0.0.1  ecommercetimes.us.intellitxt.com
    127.0.0.1  eepn.us.intellitxt.com
    127.0.0.1  efanguide.us.intellitxt.com
    127.0.0.1  egotastic.us.intellitxt.com
    127.0.0.1  eharmony.us.intellitxt.com
    127.0.0.1  ehomeupgrade.us.intellitxt.com
    127.0.0.1  ehow.us.intellitxt.com
    127.0.0.1  electronista.us.intellitxt.com
    127.0.0.1  emaxhealth.us.intellitxt.com
    127.0.0.1  encyclocentral.us.intellitxt.com
    127.0.0.1  entrepreneur.us.intellitxt.com
    127.0.0.1  eontarionow.us.intellitxt.com
    127.0.0.1  estelle.us.intellitxt.com
    127.0.0.1  eten-users.us.intellitxt.com
    127.0.0.1  evilbeetgossip.us.intellitxt.com
    127.0.0.1  eweek.us.intellitxt.com
    127.0.0.1  examnotes.us.intellitxt.com
    127.0.0.1  excite.us.intellitxt.com
    127.0.0.1  experts.us.intellitxt.com
    127.0.0.1  expertvillage.us.intellitxt.com
    127.0.0.1  extntechnologies.us.intellitxt.com
    127.0.0.1  extremetech.us.intellitxt.com
    127.0.0.1  eztracks.us.intellitxt.com
    127.0.0.1  fangoria.us.intellitxt.com
    127.0.0.1  faqts.us.intellitxt.com
    127.0.0.1  fatbackandcollards.us.intellitxt.com
    127.0.0.1  fatbackmedia.us.intellitxt.com
    127.0.0.1  fatfreekitchen.us.intellitxt.com
    127.0.0.1  feedsweep.us.intellitxt.com
    127.0.0.1  fhmonline.us.intellitxt.com
    127.0.0.1  fightline.us.intellitxt.com
    127.0.0.1  filmdrunk.us.intellitxt.com
    127.0.0.1  filedudes.us.intellitxt.com
    127.0.0.1  filmstew.us.intellitxt.com
    127.0.0.1  filmthreat.us.intellitxt.com
    127.0.0.1  firingsquad.us.intellitxt.com
    127.0.0.1  fixya.us.intellitxt.com
    127.0.0.1  flashmagazine.us.intellitxt.com
    127.0.0.1  flexbeta.us.intellitxt.com
    127.0.0.1  forbes.us.intellitxt.com
    127.0.0.1  fortunecity.us.intellitxt.com
    127.0.0.1  foxnews.us.intellitxt.com
    127.0.0.1  foxsports.us.intellitxt.com
    127.0.0.1  foxtv.us.intellitxt.com
    127.0.0.1  freecodecs.us.intellitxt.com
    127.0.0.1  freewarefiles.us.intellitxt.com
    127.0.0.1  freewarehome.us.intellitxt.com
    127.0.0.1  friendtest.us.intellitxt.com
    127.0.0.1  futurelooks.us.intellitxt.com
    127.0.0.1  g2.us.intellitxt.com
    127.0.0.1  g3.us.intellitxt.com
    127.0.0.1  g4.us.intellitxt.com
    127.0.0.1  g5.us.intellitxt.com
    127.0.0.1  galleryoftheabsurd.us.intellitxt.com
    127.0.0.1  gamedev.us.intellitxt.com
    127.0.0.1  gamesradar.us.intellitxt.com
    127.0.0.1  gamerstemple.us.intellitxt.com
    127.0.0.1  gannettbroadcast.us.intellitxt.com
    127.0.0.1  gannettwisconsin.us.intellitxt.com
    127.0.0.1  gabsmash.us.intellitxt.com
    127.0.0.1  geek.us.intellitxt.com
    127.0.0.1  geekstogo.us.intellitxt.com
    127.0.0.1  girlsaskguys.us.intellitxt.com
    127.0.0.1  givememyremote.us.intellitxt.com
    127.0.0.1  gonintendo.us.intellitxt.com
    127.0.0.1  govpro.us.intellitxt.com
    127.0.0.1  graytv.us.intellitxt.com
    127.0.0.1  gsmarena.us.intellitxt.com
    127.0.0.1  gtmedia.us.intellitxt.com
    127.0.0.1  gtrforums.us.intellitxt.com
    127.0.0.1  guru3d.us.intellitxt.com
    127.0.0.1  guyism.us.intellitxt.com
    127.0.0.1  hackedgadgets.us.intellitxt.com
    127.0.0.1  hairboutique.us.intellitxt.com
    127.0.0.1  hardcoreware.us.intellitxt.com
    127.0.0.1  hardforum.us.intellitxt.com
    127.0.0.1  hardocp.us.intellitxt.com
    127.0.0.1  hardwareanalysis.us.intellitxt.com
    127.0.0.1  hardwaregeeks.us.intellitxt.com
    127.0.0.1  hardwarezone.us.intellitxt.com
    127.0.0.1  harmony-central.us.intellitxt.com
    127.0.0.1  helium.us.intellitxt.com
    127.0.0.1  highdefforum.us.intellitxt.com
    127.0.0.1  hiphoprx.us.intellitxt.com
    127.0.0.1  hiphopdx.us.intellitxt.com
    127.0.0.1  hitsquad.us.intellitxt.com
    127.0.0.1  hollyrude.us.intellitxt.com
    127.0.0.1  hollywood.us.intellitxt.com
    127.0.0.1  hollywoodbackwash.us.intellitxt.com
    127.0.0.1  hollywoodchicago.us.intellitxt.com
    127.0.0.1  hollywoodstreetking.us.intellitxt.com
    127.0.0.1  hollywoodtuna.us.intellitxt.com
    127.0.0.1  hometheaterhifi.us.intellitxt.com
    127.0.0.1  hondahookup.us.intellitxt.com
    127.0.0.1  hoopsvibe.us.intellitxt.com
    127.0.0.1  hoopsworld.us.intellitxt.com
    127.0.0.1  hoovers.us.intellitxt.com
    127.0.0.1  horoscope.us.intellitxt.com
    127.0.0.1  hostboard.us.intellitxt.com
    127.0.0.1  hothardware.us.intellitxt.com
    127.0.0.1  hotmommagossip.us.intellitxt.com
    127.0.0.1  hotonlinenews.us.intellitxt.com
    127.0.0.1  howardchui.us.intellitxt.com
    127.0.0.1  hq-celebrity.us.intellitxt.com
    127.0.0.1  htmlcenter.us.intellitxt.com
    127.0.0.1  huliq.us.intellitxt.com
    127.0.0.1  i4u.us.intellitxt.com
    127.0.0.1  iamnotageek.us.intellitxt.com
    127.0.0.1  icentric.us.intellitxt.com
    127.0.0.1  ichef.us.intellitxt.com
    127.0.0.1  icydk.us.intellitxt.com
    127.0.0.1  idontlikeyouinthatway.us.intellitxt.com
    127.0.0.1  iesb.us.intellitxt.com
    127.0.0.1  ign.us.intellitxt.com
    127.0.0.1  india-forums.us.intellitxt.com
    127.0.0.1  babes.ign.us.intellitxt.com
    127.0.0.1  cars.ign.us.intellitxt.com
    127.0.0.1  comics.ign.us.intellitxt.com
    127.0.0.1  cube.ign.us.intellitxt.com
    127.0.0.1  ds.ign.us.intellitxt.com
    127.0.0.1  filmforcedvd.ign.us.intellitxt.com
    127.0.0.1  gameboy.ign.us.intellitxt.com
    127.0.0.1  gear.ign.us.intellitxt.com
    127.0.0.1  music.ign.us.intellitxt.com
    127.0.0.1  psp.ign.us.intellitxt.com
    127.0.0.1  ps2.ign.us.intellitxt.com
    127.0.0.1  ps3.ign.us.intellitxt.com
    127.0.0.1  psx.ign.us.intellitxt.com
    127.0.0.1  revolution.ign.us.intellitxt.com
    127.0.0.1  sports.ign.us.intellitxt.com
    127.0.0.1  wireless.ign.us.intellitxt.com
    127.0.0.1  xbox.ign.us.intellitxt.com
    127.0.0.1  xbox360.ign.us.intellitxt.com
    127.0.0.1  idm.us.intellitxt.com
    127.0.0.1  i-hacked.us.intellitxt.com
    127.0.0.1  imnotobsessed.us.intellitxt.com
    127.0.0.1  impactwrestling.us.intellitxt.com
    127.0.0.1  imreportcard.us.intellitxt.com
    127.0.0.1  infopackets.us.intellitxt.com
    127.0.0.1  insidemacgames.us.intellitxt.com
    127.0.0.1  intermix.us.intellitxt.com
    127.0.0.1  internetautoguide.us.intellitxt.com
    127.0.0.1  intogossip.us.intellitxt.com
    127.0.0.1  intomobile.us.intellitxt.com
    127.0.0.1  investopedia.us.intellitxt.com
    127.0.0.1  ittoolbox.us.intellitxt.com
    127.0.0.1  itxt2.us.intellitxt.com
    127.0.0.1  itxt3.us.intellitxt.com
    127.0.0.1  itworld.us.intellitxt.com
    127.0.0.1  ivillage.us.intellitxt.com
    127.0.0.1  s.ivillage.us.intellitxt.com
    127.0.0.1  iwon.us.intellitxt.com
    127.0.0.1  jacksonsun.us.intellitxt.com
    127.0.0.1  jakeludington.us.intellitxt.com
    127.0.0.1  jkontherun.us.intellitxt.com
    127.0.0.1  joblo.us.intellitxt.com
    127.0.0.1  juicy-news.blogspot.us.intellitxt.com
    127.0.0.1  jupiter.us.intellitxt.com
    127.0.0.1  justjared.us.intellitxt.com
    127.0.0.1  justmovietrailers.us.intellitxt.com
    127.0.0.1  kaboose.us.intellitxt.com
    127.0.0.1  killerstartups.us.intellitxt.com
    127.0.0.1  knac.us.intellitxt.com
    127.0.0.1  laboroflove.us.intellitxt.com
    127.0.0.1  laptoplogic.us.intellitxt.com
    127.0.0.1  laptopmag.us.intellitxt.com
    127.0.0.1  lat34.us.intellitxt.com
    127.0.0.1  latinoreview.us.intellitxt.com
    127.0.0.1  linuxdevcenter.us.intellitxt.com
    127.0.0.1  linuxjournal.us.intellitxt.com
    127.0.0.1  lmcd.us.intellitxt.com
    127.0.0.1  lockergnome.us.intellitxt.com
    127.0.0.1  lohud.us.intellitxt.com
    127.0.0.1  longhornblogs.us.intellitxt.com
    127.0.0.1  lxer.us.intellitxt.com
    127.0.0.1  lyrics.us.intellitxt.com
    127.0.0.1  macdailynews.us.intellitxt.com
    127.0.0.1  macnewsworld.us.intellitxt.com
    127.0.0.1  macnn.us.intellitxt.com
    127.0.0.1  macgamefiles.us.intellitxt.com
    127.0.0.1  macmegasite.us.intellitxt.com
    127.0.0.1  madpenguin.us.intellitxt.com
    127.0.0.1  majorgeeks.us.intellitxt.com
    127.0.0.1  makeherup.us.intellitxt.com
    127.0.0.1  makemeheal.us.intellitxt.com
    127.0.0.1  makeushot.us.intellitxt.com
    127.0.0.1  masalatalk.us.intellitxt.com
    127.0.0.1  mazdaworld.us.intellitxt.com
    127.0.0.1  medicinenet.us.intellitxt.com
    127.0.0.1  medindia.us.intellitxt.com
    127.0.0.1  memphisrap.us.intellitxt.com
    127.0.0.1  meredithtv.us.intellitxt.com
    127.0.0.1  methodshop.us.intellitxt.com
    127.0.0.1  microsoft-watch.us.intellitxt.com
    127.0.0.1  military.us.intellitxt.com
    127.0.0.1  missjia.us.intellitxt.com
    127.0.0.1  mobile9.us.intellitxt.com
    127.0.0.1  mobileburn.us.intellitxt.com
    127.0.0.1  mobiletechreview.us.intellitxt.com
    127.0.0.1  mobilewhack.us.intellitxt.com
    127.0.0.1  mobilityguru.us.intellitxt.com
    127.0.0.1  mobilitysite.us.intellitxt.com
    127.0.0.1  morningstar.us.intellitxt.com
    127.0.0.1  motortrend.us.intellitxt.com
    127.0.0.1  moviehole.us.intellitxt.com
    127.0.0.1  movie-list.us.intellitxt.com
    127.0.0.1  movies.us.intellitxt.com
    127.0.0.1  movieweb.us.intellitxt.com
    127.0.0.1  msfn.us.intellitxt.com
    127.0.0.1  autos.msnbc.us.intellitxt.com
    127.0.0.1  business.msnbc.us.intellitxt.com
    127.0.0.1  news.msnbc.us.intellitxt.com
    127.0.0.1  sports.msnbc.us.intellitxt.com
    127.0.0.1  technology.msnbc.us.intellitxt.com
    127.0.0.1  travel-and-weather.msnbc.us.intellitxt.com
    127.0.0.1  mmafighting.us.intellitxt.com
    127.0.0.1  entertainment.msn.us.intellitxt.com
    127.0.0.1  muscleandfitnesshers.us.intellitxt.com
    127.0.0.1  mydigitallife.us.intellitxt.com
    127.0.0.1  myfavoritegames.us.intellitxt.com
    127.0.0.1  nasioc.us.intellitxt.com
    127.0.0.1  nationalledger.us.intellitxt.com
    127.0.0.1  nationalenquirer.us.intellitxt.com
    127.0.0.1  naturalhealth.us.intellitxt.com
    127.0.0.1  nbcuniversaltv.us.intellitxt.com
    127.0.0.1  neoseeker.us.intellitxt.com
    127.0.0.1  neowin.us.intellitxt.com
    127.0.0.1  newstoob.us.intellitxt.com
    127.0.0.1  ngohq.us.intellitxt.com
    127.0.0.1  nihoncar.us.intellitxt.com
    127.0.0.1  ninjadude.us.intellitxt.com
    127.0.0.1  ntcompatible.us.intellitxt.com
    127.0.0.1  obgyn.us.intellitxt.com
    127.0.0.1  octools.us.intellitxt.com
    127.0.0.1  ocworkbench.us.intellitxt.com
    127.0.0.1  officer.us.intellitxt.com
    127.0.0.1  okmagazine.us.intellitxt.com
    127.0.0.1  onlamp.us.intellitxt.com
    127.0.0.1  oocenter.us.intellitxt.com
    127.0.0.1  osdir.us.intellitxt.com
    127.0.0.1  ostg.us.intellitxt.com
    127.0.0.1  outofsightmedia.us.intellitxt.com
    127.0.0.1  outsidethebeltway.us.intellitxt.com
    127.0.0.1  overclockersonline.us.intellitxt.com
    127.0.0.1  overthelimit.us.intellitxt.com
    127.0.0.1  pal-item.us.intellitxt.com
    127.0.0.1  pcmag.us.intellitxt.com
    127.0.0.1  pcper.us.intellitxt.com
    127.0.0.1  pencomputing.us.intellitxt.com
    127.0.0.1  penton.us.intellitxt.com
    127.0.0.1  perezhilton.us.intellitxt.com
    127.0.0.1  phonearena.us.intellitxt.com
    127.0.0.1  physorg.us.intellitxt.com
    127.0.0.1  pickmeupnews.us.intellitxt.com
    127.0.0.1  pimprig.us.intellitxt.com
    127.0.0.1  planetgamecube.us.intellitxt.com
    127.0.0.1  planet-source-code.us.intellitxt.com
    127.0.0.1  playboy.us.intellitxt.com
    127.0.0.1  popdirt.us.intellitxt.com
    127.0.0.1  popoholic.us.intellitxt.com
    127.0.0.1  poponthepop.us.intellitxt.com
    127.0.0.1  popularmechanics.us.intellitxt.com
    127.0.0.1  portableplanet.us.intellitxt.com
    127.0.0.1  postchronicle.us.intellitxt.com
    127.0.0.1  prettyboring.us.intellitxt.com
    127.0.0.1  primedia.us.intellitxt.com
    127.0.0.1  priusonline.us.intellitxt.com
    127.0.0.1  profootballweekly.us.intellitxt.com
    127.0.0.1  pro-networks.us.intellitxt.com
    127.0.0.1  ps3news.us.intellitxt.com
    127.0.0.1  punchjump.us.intellitxt.com
    127.0.0.1  pwinsider.us.intellitxt.com
    127.0.0.1  qj.us.intellitxt.com
    127.0.0.1  quickpwn.us.intellitxt.com
    127.0.0.1  rankmytattoos.us.intellitxt.com
    127.0.0.1  rawstory.us.intellitxt.com
    127.0.0.1  rcpmag.us.intellitxt.com
    127.0.0.1  realitytvmagazine.us.intellitxt.com
    127.0.0.1  recipegoldmine.us.intellitxt.com
    127.0.0.1  recipeland.us.intellitxt.com
    127.0.0.1  redbalcony.us.intellitxt.com
    127.0.0.1  reelmovienews.us.intellitxt.com
    127.0.0.1  rickey.us.intellitxt.com
    127.0.0.1  ringsurf.us.intellitxt.com
    127.0.0.1  roadcatalogs.us.intellitxt.com
    127.0.0.1  rightpundits.us.intellitxt.com
    127.0.0.1  rojakpot.us.intellitxt.com
    127.0.0.1  rpg.us.intellitxt.com
    127.0.0.1  rx8club.us.intellitxt.com
    127.0.0.1  rydium.us.intellitxt.com
    127.0.0.1  screensavers.us.intellitxt.com
    127.0.0.1  sdcexecs.us.intellitxt.com
    127.0.0.1  sdtimes.us.intellitxt.com
    127.0.0.1  sheknows.us.intellitxt.com
    127.0.0.1  shoutwire.us.intellitxt.com
    127.0.0.1  siliconera.us.intellitxt.com
    127.0.0.1  slashfilm.us.intellitxt.com
    127.0.0.1  slashphone.us.intellitxt.com
    127.0.0.1  soccergaming.us.intellitxt.com
    127.0.0.1  socialitelife.us.intellitxt.com
    127.0.0.1  soft32.us.intellitxt.com
    127.0.0.1  softonic.us.intellitxt.com
    127.0.0.1  sohh.us.intellitxt.com
    127.0.0.1  somethingawful.us.intellitxt.com
    127.0.0.1  soundslam.us.intellitxt.com
    127.0.0.1  speedguide.us.intellitxt.com
    127.0.0.1  speedtv.us.intellitxt.com
    127.0.0.1  sprintusers.us.intellitxt.com
    127.0.0.1  spymac.us.intellitxt.com
    127.0.0.1  sqlservercentral.us.intellitxt.com
    127.0.0.1  starcasm.us.intellitxt.com
    127.0.0.1  starpulse.us.intellitxt.com
    127.0.0.1  stockgroup.us.intellitxt.com
    127.0.0.1  storknet.us.intellitxt.com
    127.0.0.1  stupidcelebrities.us.intellitxt.com
    127.0.0.1  supercars.us.intellitxt.com
    127.0.0.1  superherohype.us.intellitxt.com
    127.0.0.1  surebaby.us.intellitxt.com
    127.0.0.1  symbianone.us.intellitxt.com
    127.0.0.1  symbian-freak.us.intellitxt.com
    127.0.0.1  tastefulcelebs.us.intellitxt.com
    127.0.0.1  tbohiphop.us.intellitxt.com
    127.0.0.1  techeblog.us.intellitxt.com
    127.0.0.1  tech-faq.us.intellitxt.com
    127.0.0.1  techgage.us.intellitxt.com
    127.0.0.1  techguy.us.intellitxt.com
    127.0.0.1  techimo.us.intellitxt.com
    127.0.0.1  technewsworld.us.intellitxt.com
    127.0.0.1  techpowerup.us.intellitxt.com
    127.0.0.1  techspot.us.intellitxt.com
    127.0.0.1  techsupportforum.us.intellitxt.com
    127.0.0.1  tenmagazines.us.intellitxt.com
    127.0.0.1  tennessean.us.intellitxt.com
    127.0.0.1  tgdaily.us.intellitxt.com
    127.0.0.1  thathappened.us.intellitxt.com
    127.0.0.1  theadvertiser.us.intellitxt.com
    127.0.0.1  theblemish.us.intellitxt.com
    127.0.0.1  thebosh.us.intellitxt.com
    127.0.0.1  thecarconnection.us.intellitxt.com
    127.0.0.1  thecelebritycafe.us.intellitxt.com
    127.0.0.1  theeldergeek.us.intellitxt.com
    127.0.0.1  thefinalfantasy.us.intellitxt.com
    127.0.0.1  theforce.us.intellitxt.com
    127.0.0.1  thefutoncritic.us.intellitxt.com
    127.0.0.1  thegauntlet.us.intellitxt.com
    127.0.0.1  thehollywoodgossip.us.intellitxt.com
    127.0.0.1  themanroom.us.intellitxt.com
    127.0.0.1  theonenetwork.us.intellitxt.com
    127.0.0.1  thesuperficial.us.intellitxt.com
    127.0.0.1  thetechlounge.us.intellitxt.com
    127.0.0.1  thetechzone.us.intellitxt.com
    127.0.0.1  theunwired.us.intellitxt.com
    127.0.0.1  thinkcomputers.us.intellitxt.com
    127.0.0.1  thoughtsmedia.us.intellitxt.com
    127.0.0.1  threadwatch.us.intellitxt.com
    127.0.0.1  tmz.us.intellitxt.com
    127.0.0.1  todayshow.us.intellitxt.com
    127.0.0.1  toofab.us.intellitxt.com
    127.0.0.1  toms.us.intellitxt.com
    127.0.0.1  tomsforumz.us.intellitxt.com
    127.0.0.1  tomsnetworking.us.intellitxt.com
    127.0.0.1  topsocialite.us.intellitxt.com
    127.0.0.1  tothecenter.us.intellitxt.com
    127.0.0.1  tradingmarkets.us.intellitxt.com
    127.0.0.1  traileraddict.us.intellitxt.com
    127.0.0.1  trekweb.us.intellitxt.com
    127.0.0.1  tribal.us.intellitxt.com
    127.0.0.1  tutorialoutpost.us.intellitxt.com
    127.0.0.1  tvfanatic.us.intellitxt.com
    127.0.0.1  tv-now.us.intellitxt.com
    127.0.0.1  tv-rightcelebrity.us.intellitxt.com
    127.0.0.1  tweaktown.us.intellitxt.com
    127.0.0.1  tweakvista.us.intellitxt.com
    127.0.0.1  tweetsoup.us.intellitxt.com
    127.0.0.1  twitchguru.us.intellitxt.com
    127.0.0.1  ubergizmo.us.intellitxt.com
    127.0.0.1  ultimate-guitar.us.intellitxt.com
    127.0.0.1  uncoached.us.intellitxt.com
    127.0.0.1  upi.us.intellitxt.com
    127.0.0.1  vault9.us.intellitxt.com
    127.0.0.1  viaarena.us.intellitxt.com
    127.0.0.1  videocodezone.us.intellitxt.com
    127.0.0.1  vidnet.us.intellitxt.com
    127.0.0.1  vladtv.us.intellitxt.com
    127.0.0.1  voodoofiles.us.intellitxt.com
    127.0.0.1  warcry.us.intellitxt.com
    127.0.0.1  warp2search.us.intellitxt.com
    127.0.0.1  wdxcyber.us.intellitxt.com
    127.0.0.1  weightlossforall.us.intellitxt.com
    127.0.0.1  whatthetech.us.intellitxt.com
    127.0.0.1  whyfame.us.intellitxt.com
    127.0.0.1  wincert.us.intellitxt.com
    127.0.0.1  windowsbbs.us.intellitxt.com
    127.0.0.1  windowsitpro.us.intellitxt.com
    127.0.0.1  winmatrix.us.intellitxt.com
    127.0.0.1  winterrowd.us.intellitxt.com
    127.0.0.1  wiregirl.us.intellitxt.com
    127.0.0.1  withleather.us.intellitxt.com
    127.0.0.1  wm5fixsite.us.intellitxt.com
    127.0.0.1  womensforum.us.intellitxt.com
    127.0.0.1  worldnetdaily.us.intellitxt.com
    127.0.0.1  wowinterface.us.intellitxt.com
    127.0.0.1  wrestling-edge.us.intellitxt.com
    127.0.0.1  wwe.us.intellitxt.com
    127.0.0.1  wwtdd.us.intellitxt.com
    127.0.0.1  x17online.us.intellitxt.com
    127.0.0.1  xmlpitstop.us.intellitxt.com
    127.0.0.1  yeeeah.us.intellitxt.com
    127.0.0.1  yourtango.us.intellitxt.com
    127.0.0.1  zatznotfunny.us.intellitxt.com
    127.0.0.1  zeldalily.us.intellitxt.com
    127.0.0.1  zug.us.intellitxt.com
    That's pritty silly. These could all be blocked with a single entry "intellitxt.com" in the poisoning method.

    I prefer the poisoning method using an appropriate (and short) list (pgl.yoyo).

    The hosts lists should be optimized a little bit for the poisoning method, but this is not easy within a shell script. I think there is a little program or perl/python script necessary. Also a php script on some server my do the work.
    If some domain is present in the script (like "intellitxt.com") all subdomains should be removed from the list.
    If a have some time, I will think about a solution in PHP that can be placed on some server. The script should work the following way: it should be invoked with the url of a list, then it should retrieve and optimize this list, and deliver the filtered list.

    The following algorithm should be a good starting point:
    1. read the hosts file
    2. remove the ip (127.0.0.1)
    3. reverse the components (cdcovers.cc.intellitxt.com -> com.intellitxt.cc.cdcovers)
    4. sort the list
    5. if a line contains the preceding line at the beginning, remove it. ("com.intellitxt.cc.cdcovers" contains "com.intellitxt")
    6. reverse the components
     
  50. srouquette

    srouquette Network Guru Member

    @~nephelim~: ok, I updated the script.

    @jochen: you're right, we made a big mistake with the hosts file version, I'll update it to remove the "www., download., setup." only for the dns poisoning version.

    I thought it would be good to do like you said (removing hosts for dns poisoning), but we can't really do it for every sites.
    In http://www.mvps.org/winhelp2002/hosts.txt, search "free.fr".
    There are 2 entries, these hosts are personal websites. If you remove the host, it will block free.fr, and people won't have access to this site which is a big ADSL provider in France (http://www.free.fr/adsl/), we can't really do this.

    But the www removal was a mistake, I tested it with www2.hermoment.com, this site isn't blocked anymore with hosts file.

    edit:
    rewrote TRIM() until we find a solution.
    http://www.linksysinfo.org/forums/showpost.php?p=368862&postcount=146
     
  51. jochen

    jochen LI Guru Member

    I developed a little shell script (on my Mac) using sed and awk for doing the optimization describe in my previous post. Don't know if it works exactly in Tomato, there are often little differences in regex.

    Code:
    #!/bin/sh
    cat test.txt | tr -d "\r" | sed -e 's/#.*$//' -e '/^$/d' >test0.txt
    sed  -e 's/^[0-9.]*[ \t]*//' -e 's/[ ]*$//' test0.txt >test1.txt
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' test1.txt | sed -e 's/^\.*//' | sort >test2.txt
    awk 'BEGIN {dom = "hugo"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' test2.txt >test3.txt
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' test3.txt | sed -e 's/^\.*//' >test4.txt
    When processing the MVP list, there still remain some unwanted entries:
    Code:
    host1.adhese.be
    host2.adhese.be
    host3.adhese.be
    host4.adhese.be
    line01.metriweb.be
    line02.metriweb.be
    line03.metriweb.be
    line04.metriweb.be
    line05.metriweb.be
    line06.metriweb.be
    line07.metriweb.be
    line08.metriweb.be
    line09.metriweb.be
    line10.metriweb.be
    line11.metriweb.be
    line12.metriweb.be
    line13.metriweb.be
    line14.metriweb.be
    line15.metriweb.be
    line16.metriweb.be
    line17.metriweb.be
    line18.metriweb.be
    line19.metriweb.be
    line20.metriweb.be
    line24.metriweb.be
    line26.metriweb.be
    
    So my suggestion is:
    First add some personal blacklist entries, blocking the whole domain "adhese.be" and "metriweb.be" (and others if needed).
    Do the optimization at the last stage, when all blacklists are combined. So if one of the blacklists contain "metriweb.be" (which is missing in the MVP file), then all is fine at the end.

    My script leaves them both untouched!
    Code:
    stream.expedia.fr
    akcr.free.fr
    securinews.free.fr
    media.ftv-publicite.fr
    Hosts are only removed if the domain is also present in the file.
     
  52. srouquette

    srouquette Network Guru Member

    I found a bug in your script.
    I tested it on http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml

    at.oewabox is removed because of at.oewa

    Based on your work, I rewrote your script and found other useful ones.
    Code:
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"."A[i]};print S}' hosts | sort > tmp
    
    awk 'BEGIN{n=split("z z",d1);}{ split($0,d2,"."); for (i=1;i<=n;i++) if (d1[i]!=d2[i]) { n=split($0,d1,"."); print $0; break; } }' tmp > tmp2
    
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"."A[i]};print S}' tmp2 > tmp3
    
    If you can test it and if you think it's good, I will add it to the adblock script.

    edit:
    I'm testing it and it seems to work.
    Hosts file shrink from 33188 entries to 31609 if I use only "sort -u".
    And for dns poisoning, it shrinks from 33188 to 25027.
    Problem... the WAN Up script is now larger than 4096, which is the limit for Tomato :)
    Ho... and it takes 4 min to remove duplicates for dns poisoning ^^;

    edit2:
    Ok, done :)

    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~
    ## Dnsmasq Ed. v3.2
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    rm -f /tmp/ADBLOCK.sh
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # WHITELIST
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    GETS="1 2 3 4"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    FMEM() {
    service dnsmasq stop
    killall -9 dnsmasq
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    }
    
    CLN() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    logger ADBLOCK Sort...
    sort -u -o $TMP $GEN
    else
    logger ADBLOCK Swapping...
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"."A[i]};print S}' $GEN | sort > $TMP
    logger ADBLOCK Cleaning...
    awk 'BEGIN{n=split("z z",d1);}{ split($0,d2,"."); for (i=1;i<=n;i++) if (d1[i]!=d2[i]) { n=split($0,d1,"."); print $0; break; } }' $TMP > $GEN
    logger ADBLOCK Swapping back...
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"."A[i]};print S}' $GEN > $TMP
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK ERROR, Restarting Dnsmasq...
    service dnsmasq stop
    killall -9 dnsmasq
    dnsmasq
    fi
    }
    
    CLR
    #FMEM
    DS
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    FMEM
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    
    edit3:
    @jochen: thanks, removed this little bit aswell :)
     
  53. jochen

    jochen LI Guru Member

    @srouquette: thanks for your hard work.
    One comment, because you want to save space. ;-)

    The
    Code:
    rm -f /tmp/ADBLOCK.sh
    is unnecessary because
    Code:
    > $ADB
    overwrites any existing file.

    You should write this:

    Code:
    for site in $WHITE
    do
    sed -i -e "/$site/d" $GEN
    done
    as


    Code:
    for site in $WHITE
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    because sites contain dots, and dot means "any character" in regular expression. So you have to escape the dot.
     
  54. Beast

    Beast Network Guru Member

    I just installed 3.1 and it does not seem to work. I placed in the system tools and got this error:

    /tmp/ADBLOCK.sh: line 100: syntax error: unexpected "(" (expecting "fi")
    /tmp/.wxqhh1FL: line 217: [[Y: not found
     
  55. srouquette

    srouquette Network Guru Member

    @jochen: ok, I updated it again.

    @Beast: tsss, I did a silly mistake, a quote in a comment :)

    It's fixed, you can grab it again.

    And for the record, it takes 5 min on a WTR54GL @250MHz :-/
    Code:
    Oct 16 18:00:00 unknown user.notice root: ADBLOCK http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml
    Oct 16 18:00:02 unknown user.notice root: ADBLOCK http://mirror1.malwaredomains.com/files/justdomains
    Oct 16 18:00:05 unknown user.notice root: ADBLOCK http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 16 18:00:18 unknown user.notice root: ADBLOCK http://www.mvps.org/winhelp2002/hosts.txt
    Oct 16 18:00:20 unknown user.notice root: ADBLOCK 33190 Entries
    Oct 16 18:00:20 unknown user.notice root: ADBLOCK Swapping...
    Oct 16 18:02:25 unknown user.notice root: ADBLOCK Cleaning...
    Oct 16 18:03:49 unknown user.notice root: ADBLOCK Swapping back...
    Oct 16 18:04:56 unknown user.notice root: ADBLOCK Sorted.
    Oct 16 18:04:56 unknown user.notice root: ADBLOCK 25029 Entries
    Oct 16 18:05:01 unknown daemon.info dnsmasq[3542]: started, version 2.55 cachesize 4096
    Oct 16 18:05:01 unknown daemon.info dnsmasq[3542]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N DHCP no-scripts no-TFTP
    Oct 16 18:05:01 unknown daemon.info dnsmasq[3542]: asynchronous logging enabled, queue limit is 5 messages
    Oct 16 18:05:01 unknown daemon.info dnsmasq-dhcp[3542]: DHCP, IP range 192.168.42.110 -- 192.168.42.129, lease time 1d
    Oct 16 18:05:01 unknown daemon.info dnsmasq[3542]: reading /etc/resolv.dnsmasq
    Oct 16 18:05:01 unknown user.notice root: ADBLOCK Dnsmasq is Running
    Oct 16 18:05:01 unknown daemon.info dnsmasq[3542]: using nameserver 156.154.71.25#53
    Oct 16 18:05:01 unknown daemon.info dnsmasq[3542]: using nameserver 156.154.70.25#53
    Oct 16 18:05:01 unknown daemon.info dnsmasq[3542]: read /etc/hosts - 0 addresses
    Oct 16 18:05:02 unknown daemon.info dnsmasq[3542]: read /etc/hosts.dnsmasq - 11 addresses
    
    Maybe if someone complains about it, I'll add a test to skip this part. But it shouldn't bother anyone if it's done at 4AM.
     
  56. Beast

    Beast Network Guru Member

    Thanks,, could you please change the revision number, its back t 3.0.

    Its hard to keep up whats what.. lol


    Thanks for your hard work...
     
  57. srouquette

    srouquette Network Guru Member

    Yep sorry, 3.2 then :p
     
  58. jochen

    jochen LI Guru Member

    I think this is not a problem, since this is done at night when I'm sleeping. ;-)
     
  59. srouquette

    srouquette Network Guru Member

    I agree. The problem is when you start or reboot your router, you have to wait 5 min because dnsmasq isn't started :(
     
  60. jochen

    jochen LI Guru Member

    Is it really necessary to kill dnsmasq at the beginning?
     
  61. srouquette

    srouquette Network Guru Member

    I didn't try without, I removed the comment which says "Free memory", that's why he did it.
     
  62. jochen

    jochen LI Guru Member

    I just tried it. Moved the kill just before the restart, and that also worked.
    I cannot imagine that dnsmasq occupies so much memory that it makes a big difference when processing files with 600kb in ramfs.

    Does someone know what "service dnsmasq stop" does? It looks like the "killall" after that command is not necessary, since this gives an error "dnsmasq not running".

    Edit:
    Found this:
    http://busybox.net/~vda/init_vs_runsv.html

    Looks like it does the following:
    In /var/run are files with pids of services. "service xy stop" seems to kill the proces which pid is in this dir.
    So the "killall" is unnecessary.
     
  63. srouquette

    srouquette Network Guru Member

    Updates since 2.92
    * Updated sources and improved the way to fetch them
    * Merged hosts and dns poisoning code
    * Faster source cleaner
    * Better source cleaner for dns poisoning
    * Added remote whitelist

    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~
    ## Dnsmasq Ed. v3.3
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # WHITELIST
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    GETS="1 2 3 4"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    }
    
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "."a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,".");
    for (i=n;i>=1;i--)if(d1[i]!=d2[i]){ n=k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    service dnsmasq stop
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK ERROR, Restarting Dnsmasq...
    dnsmasq
    fi
    }
    
    CLR
    DS
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
     
  64. jochen

    jochen LI Guru Member

    Looks like "service stop" sends a SIGTERM (kill -15):
    Code:
    Oct 16 20:09:13 router daemon.info dnsmasq[5743]: exiting on receipt of SIGTERM
    That's really difficult. I have the "dig" command in Mac OS, that tells me the number of milliseconds that an answer takes. But this is not repeatable. It depends mainly on routing and server responses on the inet. I cannot see any big influences of dnsmasq on my router. So I think we should not concern about.
     
  65. ~nephelim~

    ~nephelim~ LI Guru Member

    @ srouquette:

    Found out a way to reduce the awk statement you provided and made few variants.

    I rearranged the second awk step to carry the 3rd step as well and reused one split($0,d2,".") statement


    All of them get $GEN for input and output a $TMP file. D1 looks slightly faster.

    Code:
    D1(){
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "."a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,".");
    for (i=n;i>=1;i--)if(d1[i]!=d2[i]){ n=k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    }
    
    These are the other variants.
    Code:
    D2(){
    awk '{n=split($0,a,".");S=a[n];
    {for(i=n-1;i>0;i--)S=S"."a[i]};print S}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{k=split($0,d2,".");
    for (i=n;i>=1;i--)if(d1[i]!=d2[i]){n=k;(d1[k]=d2[k]);S=d1[k];
    {for (i=k-1;i>0;i--) S=S"."(d1[i]=d2[i])};print S}}'>$TMP
    }
    
    D3(){
    awk '{n=split($0,a,".");S=a[n]; 
    {for (i=n-1;i>0;i--)S=S"."a[i]};print S}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,"."); 
    for (i=n;i>=1; i--)if(d1[i]!=d2[i]){ n=k;print(d1[k]=d2[k]);
    for (i=k-1;i > 0; i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    }
    
     
  66. srouquette

    srouquette Network Guru Member

    Ok, I'll try that, thanks.
    How much is it faster?
    What is your router?

    edit:
    yep, it's faster, 4min instead of 5min, good job :)
    It's also smaller. I updated the post above, upped the version to 3.3.

    FYI, I prefer to update a post on the latest page to not clutter this thread, it's easier to find the latest version this way.
     
  67. Beast

    Beast Network Guru Member

    Wow this script has been just setting around all this time and bam...
    All these changes in one day. Thanks again for all the hard work you all..
     
  68. tmr250z

    tmr250z LI Guru Member

    I have always had all my browsers set to delete their browsing history and browser cache when I close them, so I have a clean slate every time I reopen them.

    Using the lastest version on this script, Dnsmasq Ed. v3.3 USEHOSTS="Y", entering "dig www.jdoqocy.com" produces the following:
    Code:
    ; <<>> DiG 9.6.0-APPLE-P2 <<>> www.jdoqocy.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19281
    ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
    
    ;; QUESTION SECTION:
    ;www.jdoqocy.com.        IN    A
    
    ;; ANSWER SECTION:
    www.jdoqocy.com.    0    IN    A    0.0.0.0
    
    ;; Query time: 4 msec
    ;; SERVER: 192.168.1.1#53(192.168.1.1)
    ;; WHEN: Sat Oct 16 21:28:42 2010
    ;; MSG SIZE  rcvd: 49
    
    But I can still navigate to the jdoqocy click tracking link that I posted previously using the hosts file method in Firefox, Chrome and Safari on my Mac.

    Using the dns poisoning method, USEHOSTS="N", that link and everything else that should be blocked is blocked. I have no clue what's going on here.
     
  69. jochen

    jochen LI Guru Member

    No problem here on my Mac. All working as designed.
    I found that jdoqocy has a CNAME. Maybe this is cached somewhere on your Mac, maybe in Webkit?

    Code:
    www.jdoqocy.com.        668     IN      CNAME   track.cj.akadns.net.
    track.cj.akadns.net.    288     IN      A       63.215.202.74
    
    Since this does not happen on my Mac I cannot help you. Use the poisoning method. :wink:
     
  70. jochen

    jochen LI Guru Member

    @srouquette:
    I added these two functions to the script, to avoid eval errors.
    Code:
    F1() { return 0; }
    F2() { return 0; }
     
  71. Larix

    Larix LI Guru Member

    The script doesn't block the ads on all sides I surf.
    I have added the script in the wan up under Administration -> Script and the options for dnsmasq under Advanced -> DHCP/DNS and then I reboot the router. Nothing changed I can the ad on all homepages.

    The log said:
    Code:
    Oct 17 09:41:16 Stargate user.notice root: ADBLOCK http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml
    Oct 17 09:41:18 Stargate user.notice root: ADBLOCK http://mirror1.malwaredomains.com/files/justdomains
    Oct 17 09:41:21 Stargate user.notice root: ADBLOCK http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 17 09:41:31 Stargate user.notice root: ADBLOCK http://www.mvps.org/winhelp2002/hosts.txt
    Oct 17 09:41:31 Stargate user.notice root: ADBLOCK 33305 Entries
    Oct 17 09:41:31 Stargate user.notice root: ADBLOCK Sort...
    Oct 17 09:41:47 Stargate cron.err crond[88]: time disparity of 21455021 minutes detected
    Oct 17 09:46:15 Stargate user.notice root: ADBLOCK Sorted.
    Oct 17 09:46:15 Stargate user.notice root: ADBLOCK 25144 Entries
    Oct 17 09:46:19 Stargate daemon.info dnsmasq[96]: exiting on receipt of SIGTERM
    Oct 17 09:46:20 Stargate daemon.crit dnsmasq[460]: illegal repeated keyword at line 25166 of /tmp/gen
    Oct 17 09:46:20 Stargate daemon.crit dnsmasq[460]: FAILED to start up
    Oct 17 09:46:20 Stargate user.notice root: ADBLOCK ERROR, Restarting Dnsmasq...
    
    I have optimise="Y" and USEWHITELIST="N".
    The version that I use is from srouquette post on the last page 17.
     
  72. ~nephelim~

    ~nephelim~ LI Guru Member

    You're welcome. :)

    My router ( Buffalo WHR-G54S ) is not a fast horse. IIRC is clocked at 200 Mhz.

    The awk approach to selective subdomain removal was really illuminating.

    As per jochen suggestion it would be possible to include a blackist ( "adhese.be" and "metriweb.be" ) for the DNS poisoning part.

    This could come at no cost by appending a variable content before sed parsing (echo "$BLACK" >> $GEN) while the remote version could be implicitly carried by adding a custom entry to the S[0-9] list (eg S8 ="http://myhost/mycustomblacklist")
     
  73. jochen

    jochen LI Guru Member

    I corrected the little bug in my first awk script. This runs faster than the version with loops in awk.

    Code:
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' $GEN |
    sed -e 's/^\.*//' -e 's/$/\./' | sort |
    awk 'BEGIN {dom = "zz"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' |
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' | sed -e 's/^\.*//' >$TMP
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
     
  74. ~nephelim~

    ~nephelim~ LI Guru Member

    What about using awk to add that dot?
    Code:
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1"."}' $GEN |
    sed -e 's/^\.*//' | sort |
    awk 'BEGIN {dom = "zz"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' |
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' | sed -e 's/^\.*//' >$TMP
     
  75. srouquette

    srouquette Network Guru Member

    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~
    ## Dnsmasq Ed. v3.3a
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # WHITELIST
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    GETS="1 2 3 4"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    F1() { return 0; }
    F2() { return 0; }
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    }
    
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "."a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,".");
    for (i=n;i>=1;i--)if(d1[i]!=d2[i]){ n=k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    service dnsmasq stop
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK ERROR, Restarting Dnsmasq...
    dnsmasq
    fi
    }
    
    CLR
    DS
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    
    @~nephelim~: added your fix, we were short on space, that's why I didn't bother with this :)

    @jochen: ok, I'll try it again. But I see you still use substr with length(dom), the problem I reported came from this. at.oewabox was removed because it matches the condition, at.oewa is shorter. Did the sed fix this problem?

    @Larix:
    Code:
    Oct 17 09:46:20 Stargate daemon.crit dnsmasq[460]: illegal repeated keyword at line 25166 of /tmp/gen
    You still have something left in custom dnsmasq conf.
    If you use OPTIMISE, it adds:
    Code:
    dhcp-authoritative
    cache-size=2048
    log-async=5
    
    And if you USEHOSTS:
    Code:
    addn-hosts=/tmp/hosts
    You shouldn't have these in custom dnsmasq conf if you use OPTIMISE.
     
  76. Larix

    Larix LI Guru Member

    Thanks, the OPTIMISE was the problem. :) It doesn't block all ads. ;*(
     
  77. jochen

    jochen LI Guru Member

    Yes. The only change I did: I added a dot at the end of every line, so oewabox and oewa can be distinguished.

    I found a bug in your other awk script (don't know who introduced it):
    The sort order is not correct for the algorithm to work. Look at the lines for kazaa.com:

    Code:
    com.kazaa
    com.kazaa-firewall.www
    com.kazaa-proxy.www
    com.kazaa.download
    com.kazaa.images
    com.kazaa.www
    
    After com.kazaa (the Domain) comes com.kazaa-firewall.www, so the subdomains of kazaa.com are not removed. My script works fine at this point.
     
  78. jochen

    jochen LI Guru Member

    Since the sed is needed to remove dots at the beginning of the line, it makes no difference if sed adds one at the end or awk does it. I find it cleaner because both awks for reversing can be identical when sed adds the dot.
     
  79. srouquette

    srouquette Network Guru Member

    I tried your script and it crashed my router, maybe it's due to the big pipe. I don't really know how Linux handles this, so it's a supposition.

    The leading point is definitely an improvement, that's why your sort is better.
    I'll try to add this in the previous script.
     
  80. jochen

    jochen LI Guru Member

    The pipe runs fine on my WRT54GL. Which router do you have?
    You can split the long pipe into 2 parts.
    In general pipes are more efficient in Unix, because the data must not be written to disk on every step. A pipe is a block in memory, usually 8k big. One process writes into the pipe, the other process reads from the pipe. If the pipe is full, the writer waits until the reader made enough room. The disadvantage: there must be enough RAM for all processes to fit in memory, otherwise the system begins swapping.

    Version with small pipes:
    Code:
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' $GEN |
    sed -e 's/^\.*//' -e 's/$/\./' | sort >$TMP
    awk 'BEGIN {dom = "hugo"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' $TMP >$GEN
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' $GEN | sed -e 's/^\.*//' >$TMP
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
     
  81. ~nephelim~

    ~nephelim~ LI Guru Member

    Code:
    #BEWARE THIS IS NOT AN ADBLOCK AND TAKES A LONG TIME TO RUN
    TESTWHAT="D1 D4 D5 D6"
    LINECNT="Y"
    CATOUT="N"
    GETS="1 2 3 4" 
    
    TMP="/tmp/tmp1"
    GEN="/tmp/tmp2"
    
    TRIM(){
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/^::1[ \t]*//
    s/[ \t]*$//
    s/localhost$//
    /^$/d' $TMP 
    }
    
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"                 #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"                 #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"                           #620K
    S5="http://hosts-file.net/hphosts-partial.asp"                           #283K
    S6="http://hostsfile.mine.nu/Hosts"                                     #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"                   #3851
    
    
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    LOAD(){
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK Retrieved $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    rm -f $TMP 
    }
    
    D1(){
    awk '{ORS="";n=split($0,a,".");print a[n];
    for(i=n-1;i>0;i--)print "#"a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,"#");
    for(i=n;i>0;i--)if(d1[i]!=d2[i]){n=k;print(d1[k]=d2[k]);
    for(i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP
    }
    
    D2(){
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"#"A[i]};print S}' $GEN | sort > $TMP
    awk 'BEGIN{n=split("z z",d1);}{ split($0,d2,"#"); for (i=1;i<=n;i++) if (d1[i]!=d2[i]) { n=split($0,d1,"#"); print $0; break; } }' $TMP > $GEN
    awk '{n=split($0,A,"#");S=A[n];{for(i=n-1;i>0;i--)S=S"."A[i]};print S}' $GEN > $TMP
    
    }
    
    D3(){
    awk '{n=split($0,a,".");S=a[n];
    {for(i=n-1;i>0;i--)S=S"#"a[i]};print S}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{k=split($0,d2,"#");
    for (i=n;i>=1;i--)if(d1[i]!=d2[i]){n=k;(d1[k]=d2[k]);S=d1[k];
    {for (i=k-1;i>0;i--) S=S"."(d1[i]=d2[i])};print S}}'>$TMP
    }
    
    D4(){
    awk '{n=split($0,a,".");S=a[n]; 
    {for (i=n-1;i>0;i--)S=S"#"a[i]};print S}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,"#"); 
    for (i=n;i>=1; i--)if(d1[i]!=d2[i]){ n=k;print(d1[k]=d2[k]);
    for (i=k-1;i > 0; i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    }
    
    
    D5(){
    awk -F '.' '{print $9 "~" $8 "~" $7 "~" $6 "~" $5 "~" $4 "~" $3 "~" $2 "~" $1}' $GEN |
    sed -e 's/^~*//' -e 's/$/~/' | sort >$TMP
    awk 'BEGIN {dom = "hugo"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' $TMP >$GEN
    awk -F '~' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' $GEN | sed -e 's/^\.*//' >$TMP
    }
    
    D6() {
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "."a[i];print".\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,".");
    for (i=n;i>0;i--)if(d1[i]!=d2[i]){n=--k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    }
    
    
    
    logger START BATCH TESTING
    for x in $TESTWHAT; do
    LOAD
    
    if [ $LINECNT == "Y" ] ; then
    TOT=$(echo ": ";wc -l < $GEN; echo lines)
    fi
    
    logger start "$x" "$TOT"
    
    ACOUNT=$(date +%s)
    eval $x
    BCOUNT=$(date +%s)
    
    TRUN=$(($BCOUNT-$ACOUNT))
    
    logger end "$x" Runtime: "$(($TRUN%3600/60)) min, $(($TRUN%60)) sec [$TRUN ss]"
    
    if [ $LINECNT == "Y" ] ; then
    TOT=$(echo "After parsing: ";wc -l < $TMP; echo lines) 
    logger "$TOT" 
    fi
    
    if [ $CATOUT == "Y" ] ; then
    mv -f $TMP $GEN
    cat $GEN
    fi
    
    rm -f $TMP ;  rm -f $GEN
    
    done
    
    edit: damn. I wiped the entire post by mistake. :wall:
    I added a function that calculate elapsed time automatically
     
  82. jochen

    jochen LI Guru Member

    Maybe we should think about some other delimiter? How about "~", the last character in ascii alphabet?

    Code:
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk -F '.' '{print $9 "~" $8 "~" $7 "~" $6 "~" $5 "~" $4 "~" $3 "~" $2 "~" $1}' $GEN |
    sed -e 's/^~*//' -e 's/$/~/' | sort >$TMP
    awk 'BEGIN {dom = "hugo"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' $TMP >$GEN
    awk -F '~' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' $GEN | sed -e 's/^\.*//' >$TMP
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
     
  83. ~nephelim~

    ~nephelim~ LI Guru Member

    I'm not quite clear on this and I need some more testing but the dot might work just fine just as you mentioned.


    I've been using this protototype test the parsing on a per case basis.

    I've not updated it yet to account for kazaa case.

    Edit: Updated.
    Will start testing ASAP
    Edit2: updated again. (wrong domans)
    Code:
    
    TMP="/tmp/test"
    GEN="/tmp/test2"
    {
    cat <<'ENDF'
    kazaa.com
    www.kazaa-firewall.com
    www.kazaa-proxy.com
    download.kazaa.com
    images.kazaa.com
    www.kazaa.com
    www.oewa.at
    www.oewa.de
    oewa.de
    oewa.at
    de.oewa.at
    oewabox.at
    aa.bb.cc.dd
    aa.bbb.cc
    aa.bb
    bb.cc
    bb.aa.cc
    intellitxt.com
    de.intellitxt.com
    uk.intellitxt.com
    us.intellitxt.com
    medicinenet.us.intellitxt.com
    medindia.us.intellitxt.com
    memphisrap.us.intellitxt.com
    meredithtv.us.intellitxt.com
    computerbase.de.intellitxt.com
    computerhilfen.de.intellitxt.com
    computerwoche.de.intellitxt.com
    digital-world.de.intellitxt.com
    methodshop.us.intellitxt.com
    microsoft-watch.us.intellitxt.com
    xmlpitstop.us.intellitxt.com
    yeeeah.us.intellitxt.com
    yourtango.us.intellitxt.com
    zatznotfunny.us.intellitxt.com
    zeldalily.us.intellitxt.com
    zug.us.intellitxt.com
    anandtech.us.smarttargetting.com
    betanews.us.smarttargetting.com
    xbitlabs.us.smarttargetting.com
    www.smarttargetting.com
    vibrantmedia.com
    itxt.vibrantmedia.com
    www.vibrantmedia.com
    ENDF
    } > $GEN
    
    
    
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' $GEN |
    sed -e 's/^\.*//' -e 's/$/\./' | sort |
    awk 'BEGIN {dom = "zz"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' |
    awk -F '.' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' | sed -e 's/^\.*//' >$TMP
    mv -f $TMP $GEN
    
    cat $GEN
    
    rm -f $GEN 
    rm -f $TMP
    
     
  84. srouquette

    srouquette Network Guru Member

    Here is the updated version with the trailing dot, if you want to test it before I update the script.
    Code:
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "."a[i];[COLOR="red"]print".\n"[/COLOR]}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,".");
    for (i=n;[COLOR="Red"]i>0[/COLOR];i--)if(d1[i]!=d2[i]){ [COLOR="red"]k--;[/COLOR] n=k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
    
    The dot is the good character. Without it, sort thinks '-' is before '.', that's why kazaa wasn't removed.

    I also have a WRT54GL, maybe my router crashed because there wasn't enough memory. I'll see what happens with nephelim's script tomorrow and see if it crashes without killing dnsmasq.
     
  85. jochen

    jochen LI Guru Member

  86. ~nephelim~

    ~nephelim~ LI Guru Member

    Good job!
    The new revision works fine here.

    Edit: using a different separator appear to work fine as well

    Code:
    awk '{ORS="";n=split($0,a,".");print a[n];
    for(i=n-1;i>0;i--)print "#"a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,"#");
    for(i=n;i>0;i--)if(d1[i]!=d2[i]){n=k;print(d1[k]=d2[k]);
    for(i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    
     
  87. QSxx

    QSxx LI Guru Member

    Giving it a spin too, hold for results - i'll update here when done)

    Update1

    I'm running it with:
    Code:
    AUPD="N"  ## No autoupdate because my net reconnects every 24h, forcing script to reload
    OPTIMISE="Y" ## I use OPTIMISE because I have no entries in dnsmasq config field 
    USEHOSTS="N" ## This one is new to me :)
    USEWHITELIST="N" ## No need for whiteys yet
    
    Log said this:
    Code:
    Oct 17 12:32:48 nodeone user.notice root: ADBLOCK http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml
    Oct 17 12:32:51 nodeone user.notice root: ADBLOCK http://mirror1.malwaredomains.com/files/justdomains
    Oct 17 12:32:54 nodeone user.notice root: ADBLOCK http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 17 12:33:08 nodeone user.notice root: ADBLOCK http://www.mvps.org/winhelp2002/hosts.txt
    Oct 17 12:33:08 nodeone user.notice root: ADBLOCK 33305 Entries
    Oct 17 12:33:08 nodeone user.notice root: ADBLOCK Sort...
    Oct 17 12:35:46 nodeone user.notice root: ADBLOCK Sorted.
    Oct 17 12:35:46 nodeone user.notice root: ADBLOCK 25131 Entries
    Oct 17 12:35:48 nodeone daemon.info dnsmasq[966]: exiting on receipt of SIGTERM
    Oct 17 12:35:49 nodeone daemon.info dnsmasq[1435]: started, version 2.55 cachesize 2048
    Oct 17 12:35:49 nodeone daemon.info dnsmasq[1435]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N DHCP no-scripts TFTP
    Oct 17 12:35:49 nodeone daemon.info dnsmasq[1435]: asynchronous logging enabled, queue limit is 5 messages
    Oct 17 12:35:49 nodeone daemon.info dnsmasq-dhcp[1435]: DHCP, IP range 192.168.1.200 -- 192.168.1.200, lease time 1d
    Oct 17 12:35:50 nodeone user.notice root: ADBLOCK Dnsmasq is Running
    Oct 17 12:35:50 nodeone daemon.info dnsmasq[1435]: reading /etc/resolv.dnsmasq
    Oct 17 12:35:50 nodeone daemon.info dnsmasq[1435]: using nameserver 195.29.166.118#53
    Oct 17 12:35:50 nodeone daemon.info dnsmasq[1435]: using nameserver 208.67.220.220#53
    Oct 17 12:35:50 nodeone daemon.info dnsmasq[1435]: using nameserver 208.67.222.222#53
    Oct 17 12:35:50 nodeone daemon.info dnsmasq[1435]: read /etc/hosts - 0 addresses
    Oct 17 12:35:50 nodeone daemon.info dnsmasq[1435]: read /etc/hosts.dnsmasq - 6 addresses
    
    Looking good so far - Sort did it's job in just over 2 min 30 sec - you can see hardware in my sig
     
  88. srouquette

    srouquette Network Guru Member

    I'll post 3.3b here.
    Code:
    
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~
    ## Dnsmasq Ed. v3.3b
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # WHITELIST
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    GETS="1 2 3 4"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    F1() { return 0; }
    F2() { return 0; }
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    }
    
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "."a[i];print".\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,".");
    for (i=n;i>0;i--)if(d1[i]!=d2[i]){ k--; n=k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    service dnsmasq stop
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK ERROR, Restarting Dnsmasq...
    dnsmasq
    fi
    }
    
    CLR
    DS
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    
    LAN party today, I will post later. I have some trouble with Toastman's QoS with VPN build, maybe that's why I had a memory problem. I disabled QoS and the VPN worked after that. I'll test again with your scripts, but maybe you'll find the fastest one before I come back :)
     
  89. QSxx

    QSxx LI Guru Member

    Theese might be a stupid questions but i'll go ahead anyways:

    What does the USEHOST actually do? Add that one line to dnsmasq config?
    Is it better to use OPTIMISE and USEHOST or add those 3/1 lines in dnsmasq config manually?

    Testing shows NO difference in speed?

    P.S. 3.3b is ATM fastest one... at least on my router (check post #187 - 2:38)
     
  90. jochen

    jochen LI Guru Member

    Script with substr:
    Code:
    Oct 17 12:30:39 router user.notice root: ADBLOCK 33306 Entries
    Oct 17 12:30:39 router user.notice root: ADBLOCK Sort...
    Oct 17 12:32:26 router user.notice root: ADBLOCK Sorted.
    Oct 17 12:32:26 router user.notice root: ADBLOCK 25132 Entries
    
    1 minute 47 seconds
    
    Script with loop on domain parts:
    Code:
    Oct 17 12:53:26 router user.notice root: ADBLOCK 33306 Entries
    Oct 17 12:53:26 router user.notice root: ADBLOCK Sort...
    Oct 17 12:56:11 router user.notice root: ADBLOCK Sorted.
    Oct 17 12:56:11 router user.notice root: ADBLOCK 25132 Entries
    
    2 minutes 45 seconds
     
  91. Larix

    Larix LI Guru Member

    On my WRT54G v2.2 it needs 4 minutes 59 seconds. I load the default lists 1-4

    Code:
    Oct 17 13:00:11 Stargate user.notice root: ADBLOCK 33304 Entries
    Oct 17 13:00:11 Stargate user.notice root: ADBLOCK Sort...
    Oct 17 13:05:10 Stargate user.notice root: ADBLOCK Sorted.
    Oct 17 13:05:10 Stargate user.notice root: ADBLOCK 25130 Entries
     
  92. jochen

    jochen LI Guru Member

    I think this is because you use native Tomato. I use Victeks mod which has a lot of more free memory, and I do overclocking on 250 MHz.
    Code:
    Total / Free Memory	14.14 MB / 4,164.00 KB (28.76%)
    CPU Frequency (MHz)	250
    The 4MB free memory are already with full domain list loaded (25.000 entries).
     
  93. srouquette

    srouquette Network Guru Member

    USEHOSTS uses the hosts file. It blocks based on hosts and not domain, which means if in your hosts file you have "example.com", it won't block "ad.example.com", that's why you have to add multiple lines.
    DNS Cache Poisoning blocks domain. "example.com" will block everything from example.com.

    Speed and memory wise, I don't know how it works. I thought the hosts file wasn't loaded when you run dnsmasq, it may be an on-access thing. For DNS Cache Poisoning, it's definitely loaded when you run dnsmasq.

    About OPTIMISE, it adds those lines to dnsmasq.conf. If you didn't add them manually in custom conf, you can enable OPTIMISE.
    I use TomatoUSB and these settings are enabled by default, so I disabled OPTIMISE and put nothing in custom conf. If there's duplicate entries, you'll get an error in the log.
     
  94. Larix

    Larix LI Guru Member

    I use Victecs Mod: Tomato RAF Firmware v1.28.8515 _RAF ND Std

    Can I overclock my WRT v2.2 to 250Mhz without a cooler?
     
  95. srouquette

    srouquette Network Guru Member

    My results...

    D1: 3m19s
    Code:
    Oct 17 14:27:59 unknown user.notice root: start D1 :  33320 lines
    Oct 17 14:31:18 unknown user.notice root: end D1
    Oct 17 14:31:18 unknown user.notice root: After parsing:  25145 lines
    
    D2: 3m22s
    Code:
    Oct 17 14:33:06 unknown user.notice root: start D2 :  33320 lines
    Oct 17 14:36:28 unknown user.notice root: end D2
    Oct 17 14:36:28 unknown user.notice root: After parsing:  25145 lines
    
    D3: 3m28s
    Code:
    Oct 17 14:37:21 unknown user.notice root: start D3 :  33320 lines
    Oct 17 14:40:49 unknown user.notice root: end D3
    Oct 17 14:40:49 unknown user.notice root: After parsing:  25145 lines
    
    D4: 3m16s
    Code:
    Oct 17 14:41:48 unknown user.notice root: start D4 :  33320 lines
    Oct 17 14:45:04 unknown user.notice root: end D4
    Oct 17 14:45:04 unknown user.notice root: After parsing:  25145 lines
    
    D5: 3m37s
    Code:
    Oct 17 14:23:09 unknown user.notice root: start D5 :  33320 lines
    Oct 17 14:26:46 unknown user.notice root: end D5
    Oct 17 14:26:46 unknown user.notice root: After parsing:  25145 lines
    
    D5 again: 3m40s
    Code:
    Oct 17 14:46:10 unknown user.notice root: start D5 :  33320 lines
    Oct 17 14:49:50 unknown user.notice root: end D5
    Oct 17 14:49:50 unknown user.notice root: After parsing:  25145 lines
    
    I have a WRT54GL with TomatoUSB k24 NoUSB VPN (build 52) @250MHz.
    Is there a command to check the free memory?
    In summary, I can see something between 600k and 1500k.

    edit: ho, D4 was faster than D1... I'll do more tests...
     
  96. srouquette

    srouquette Network Guru Member

    3.3c: D1 with # instead of trailing dot.

    Code:
    
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~
    ## Dnsmasq Ed. v3.3c
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # WHITELIST
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    
    GETS="1 2 3 4"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    F1() { return 0; }
    F2() { return 0; }
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN
    done
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    }
    
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "#"a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,"#");
    for (i=n;i>0;i--)if(d1[i]!=d2[i]){ n=k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    service dnsmasq stop
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK ERROR, Restarting Dnsmasq...
    dnsmasq
    fi
    }
    
    CLR
    DS
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    
     
  97. jochen

    jochen LI Guru Member

    Yes, 250 MHz is no problem.
     
  98. jochen

    jochen LI Guru Member

    You can check it with the "free" command.
     
  99. ~nephelim~

    ~nephelim~ LI Guru Member

    These are my results.

    WHR-G54S BCM5352 200MHZ 16MB running Tomato NoUSB Std no VPN build 51 K24

    I added one more variant but I forgot to fill the container fuction (D6) so it took 0 seconds to run :frown:


    Code:
    Oct 17 14:28:02 router user.notice root: ADBLOCK Retrieved http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml
    
    [B]Hosts used for testing (removed in the rest of the post)[/B]
    Oct 17 14:28:06 router user.notice root: ADBLOCK Retrieved http://mirror1.malwaredomains.com/files/justdomains
    Oct 17 14:28:08 router user.notice root: ADBLOCK Retrieved http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 17 14:28:18 router user.notice root: ADBLOCK Retrieved http://www.mvps.org/winhelp2002/hosts.txt
    
    Oct 17 14:28:19 router user.notice root: start D1 :  33320 lines
    Oct 17 14:32:20 router user.notice root: end D1
    Oct 17 14:32:20 router user.notice root: After parsing:  25145 lines
    Runtime= 4:01 min:sec
    
    
    Oct 17 14:32:37 router user.notice root: start D5 :  33320 lines
    Oct 17 14:37:05 router user.notice root: end D5
    Oct 17 14:37:05 router user.notice root: After parsing:  25145 lines
    Runtime= 4:28 min:sec
    
    Oct 17 14:37:44 router user.notice root: start D1 :  33320 lines
    Oct 17 14:41:45 router user.notice root: end D1
    Oct 17 14:41:45 router user.notice root: After parsing:  25145 lines
    Runtime= 4:01 min:sec
    
    Oct 17 14:42:05 router user.notice root: start D5 :  33320 lines
    Oct 17 14:46:33 router user.notice root: end D5
    Oct 17 14:46:34 router user.notice root: After parsing:  25145 lines
    Runtime= 4:28 min
    
    Oct 17 14:47:13 router user.notice root: start D1 :  33320 lines
    Oct 17 14:51:15 router user.notice root: end D1
    Oct 17 14:51:15 router user.notice root: After parsing:  25145 lines
    Runtime= 4:02 min:sec
    
    Oct 17 14:51:34 router user.notice root: start D5 :  33320 lines
    Oct 17 14:56:02 router user.notice root: end D5
    Oct 17 14:56:02 router user.notice root: After parsing:  25145 lines
    Runtime= 4:28 min:sec
    
    
     
  100. Toink

    Toink Network Guru Member

    3.3c working great so far. Thank you for the continuous development of the script, guys! :)

    Now will someone tell me how I could include this in the black list:

    render.talk4free.com

    I hate the pesky ads in my MagicJack.

    Thank in advance!
     

Share This Page