1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ALL-U-NEED Ad Blocking by Yaqui

Discussion in 'Tomato Firmware' started by yaqui, Jul 9, 2008.

  1. jochen

    jochen LI Guru Member

    Strange. On my router D5 is definitively faster.

    Code:
    Oct 17 16:04:10 router user.notice root: start D1 :  33320 lines
    Oct 17 16:06:50 router user.notice root: end D1
    Oct 17 16:06:50 router user.notice root: After parsing:  25145 lines
    
    160 seconds
    
    Oct 17 16:06:50 router user.notice root: start D4 :  33320 lines
    Oct 17 16:09:23 router user.notice root: end D4
    Oct 17 16:09:23 router user.notice root: After parsing:  25145 lines
    
    153 seconds
    
    Oct 17 16:09:24 router user.notice root: start D5 :  33320 lines
    Oct 17 16:11:10 router user.notice root: end D5
    Oct 17 16:11:11 router user.notice root: After parsing:  25145 lines
    
    106 seconds
    
    Oct 17 16:11:11 router user.notice root: start D6 :  33320 lines
    Oct 17 16:13:54 router user.notice root: end D6
    Oct 17 16:13:55 router user.notice root: After parsing:  25145 lines
    
    163 seconds
    Test on WRT54GL with Victeks v1.25.8515 .2RAF ND @250MHz
     
  2. QSxx

    QSxx LI Guru Member

    3.3b vs 3.3c

    3.3b - 2:35
    Code:
    Oct 17 16:26:28 nodeone user.notice root: ADBLOCK 33320 Entries
    Oct 17 16:26:28 nodeone user.notice root: ADBLOCK Sort...
    Oct 17 16:29:03 nodeone user.notice root: ADBLOCK Sorted.
    Oct 17 16:29:04 nodeone user.notice root: ADBLOCK 25145 Entries
    
    3.3c - 2:34
    Code:
    Oct 17 16:19:00 nodeone user.notice root: ADBLOCK 33320 Entries
    Oct 17 16:19:00 nodeone user.notice root: ADBLOCK Sort...
    Oct 17 16:21:34 nodeone user.notice root: ADBLOCK Sorted.
    Oct 17 16:21:34 nodeone user.notice root: ADBLOCK 25145 Entries
    
    tested 3 times each on hardware below (theese are best results) - now how come you guys go that much slower?
     
  3. QSxx

    QSxx LI Guru Member

    If you have some webspace somewhere put a simple txt file up there with domains you want to block. Let's say for ex. blacklist.txt

    Then find its full URL (ex. http://yourserver.something/somedir/blacklist.txt) - put it in the script as follows:

    Code:
    GETS="1 2 3 4 8"
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    S8="http://yourserver.something/somedir/blacklist.txt" #your own blacklist
    
    Note the first and the last line :)

    Don't forget to reconnect :)

    P.S. Or bug the guys to add full-blown blacklist :) :) :)
     
  4. Toink

    Toink Network Guru Member

    3.3c = 01:43
    Code:
    Oct 17 21:41:48 Linksys user.notice root: ADBLOCK 33301 Entries
    Oct 17 21:41:48 Linksys user.notice root: ADBLOCK Sort...
    Oct 17 21:43:31 Linksys user.notice root: ADBLOCK Sorted.
    Oct 17 21:43:31 Linksys user.notice root: ADBLOCK 25136 Entries
    
    @QSxx

    I will try to save it in my Synology and see it it works. Graci!
     
  5. srouquette

    srouquette Network Guru Member

    Does it happen to anyone else?
    When I run the adblock script manually, sometimes it crashes the router. The memory still has 2000kB left (from 3400), maybe it's too heavy for the router (load = 5). But the crash happens after waiting 24h, it doesn't occur after a fresh reboot if I try to run the script multiple times...
     
  6. ~nephelim~

    ~nephelim~ LI Guru Member

    I updated to NoUSB build 52 and noticed the new firmware set DNS cache to 4096 addresses by default.

    It might be worth trying to change it back to 2048 though Advanced\DNS gui settings. (thereafter running the adblock script again is needed)
     
  7. srouquette

    srouquette Network Guru Member

    thanks, I'll try that.
     
  8. QSxx

    QSxx LI Guru Member

    Not here, everything working fine on 3.3c since yesterday, BUT i have my router on reconnect every 24hours (round 4:30 in the morning - lowest load) so it might be that...
     
  9. Toink

    Toink Network Guru Member

    Last night I updated my WRT320M to Tomato Firmware v1.28.9052 MIPSR2-beta23 K26 Std and I notice this in the logs:

    Code:
    Oct 18 19:29:11 Linksys user.notice root: ADBLOCK 33579 Entries
    Oct 18 19:29:11 Linksys user.notice root: ADBLOCK Sort...
    [B]Oct 18 19:29:25 Linksys cron.err crond[392]: time disparity of 21456689 minutes detected
    [/B]Oct 18 19:30:56 Linksys user.notice root: ADBLOCK Sorted.
    
    It wasn't there in the previous build.

    Code:
    Build 52 - 10/15/2010
    [B]Changes common for kernel 2.4 and kernel 2.6 based builds:[/B]
    [LIST]
    [*]Fixed WAN Up script not being executed (was broken in build 51).
    [*]Fixed obtaining DHCP lease from Corbina Beeline and possibly other ISPs (was broken in build 51).
    [*]Fixed DHCP renewal processing to not replace the default gateway and DNS servers for PPTP/L2TP connections.
    [*]Fixed errors in routing for PPTP/L2TP connections.
    [*]Added "Use Default Gateway on Remote Network" option for PPTP/L2TP connection types (turned on by default). Turn it off to use gateway obtained via DHCP for internet routing, and only use PPTP/L2TP gateway for VPN subnet (custom routing may be required in case if VPN network contains multiple subnets).
    [*]Resolve sporadic PPPoE disconnect issues with some ISPs.
    [*]Allow to configure custom TTL values in the Web GUI.
    [*]Make sure PPTP/L2TP is gracefully disconnected after firmware upgrades and after restoring configuration to prevent connection problems after reboot.
    [*]Size optimization of some large applications.
    [*]Software updates: PPTP kernel driver 0.8.5, [URL="http://busybox.net/"]Busybox 1.17.3[/URL].
    [*]Cosmetics.
    [/LIST]
     
  10. srouquette

    srouquette Network Guru Member

    It's not related to adblock, I always had this.
    I think you get this message until your router sync his internal clock with NTP server (Basic -> Time).
     
  11. Toink

    Toink Network Guru Member

    Oh. I see. Thought it's related to the script not working coz a few times I have to open a website, then close the browser for the script to finish sorting... Or I had to reboot the router....

    Thanks, Srouquette!
     
  12. ~nephelim~

    ~nephelim~ LI Guru Member

    Made some more awk tests.

    D11 below appear to be smaller and faster.


    Code:
    awk -F '.' 'BEGIN{ORS=""}{print $NF;
    for(i=--NF;i>0;i--)print "#"$i;print "#\n"}' $GEN|sort|
    awk -F '#' 'BEGIN{ORS="";d = "%"}{if(index($0,d)!=1){d=$0;print $--NF;
    for(i=--NF;i>0;i--)print "."$i;print "\n"}}'  > $TMP
    
    Batch testing script updated as well (includes other variants)
    Code:
    #BEWARE THIS IS NOT AN ADBLOCK AND TAKES A LONG TIME TO RUN
    TESTWHAT="D1 D11"
    LINECNT="Y"
    CATOUT="N"
    GETS="1 2 3 4" 
    
    TMP="/tmp/tmp1"
    GEN="/tmp/tmp2"
    
    TRIM(){
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/^::1[ \t]*//
    s/[ \t]*$//
    s/localhost$//
    /^$/d' $TMP 
    }
    
    S1="http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"                 #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"                 #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"                           #620K
    S5="http://hosts-file.net/hphosts-partial.asp"                           #283K
    S6="http://hostsfile.mine.nu/Hosts"                                     #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"                   #3851K
    
    
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    LOAD(){
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK Retrieved $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    rm -f $TMP 
    }
    
    D0(){
    awk 'BEGIN{ORS=""}{n=split($0,a,".");print a[n];
    for(i=--n;i>0;i--)print "#"a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{ORS="";n=split("% %",d1)}{k=split($0,d2,"#");
    for(i=n;i>0;i--)if(d1[i]!=d2[i]){n=k;print d1[k]=d2[k];
    for(i=--k;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP
    }
    
    D1(){
    awk '{ORS="";n=split($0,a,".");print a[n];
    for(i=n-1;i>0;i--)print "#"a[i];print"\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,"#");
    for(i=n;i>0;i--)if(d1[i]!=d2[i]){n=k;print d1[k]=d2[k];
    for(i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP
    }
    
    D2(){
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"#"A[i]};print S}' $GEN | sort |
    awk 'BEGIN{n=split("z z",d1);}{ split($0,d2,"#"); for (i=1;i<=n;i++) if (d1[i]!=d2[i]) { n=split($0,d1,"#"); print $0; break; } }' |
    awk '{n=split($0,A,"#");S=A[n];{for(i=n-1;i>0;i--)S=S"."A[i]};print S}' > $TMP
    
    }
    
    D3(){
    awk '{n=split($0,a,".");S=a[n];
    {for(i=n-1;i>0;i--)S=S"#"a[i]};print S}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{k=split($0,d2,"#");
    for (i=n;i>=1;i--)if(d1[i]!=d2[i]){n=k;(d1[k]=d2[k]);S=d1[k];
    {for (i=k-1;i>0;i--) S=S"."(d1[i]=d2[i])};print S}}'>$TMP
    }
    
    D4(){
    awk '{n=split($0,a,".");S=a[n]; 
    {for (i=n-1;i>0;i--)S=S"#"a[i]};print S}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,"#"); 
    for (i=n;i>=1; i--)if(d1[i]!=d2[i]){ n=k;print(d1[k]=d2[k]);
    for (i=k-1;i > 0; i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    }
    
    
    D5(){
    awk -F '.' '{print $9 "~" $8 "~" $7 "~" $6 "~" $5 "~" $4 "~" $3 "~" $2 "~" $1}' $GEN |
    sed -e 's/^~*//' -e 's/$/~/' | sort |
    awk 'BEGIN {dom = "hugo"} { if ( substr($0,1,length(dom)) != dom ) {print $0;dom = $0} }' |
    awk -F '~' '{print $9 "." $8 "." $7 "." $6 "." $5 "." $4 "." $3 "." $2 "." $1}' | sed -e 's/^\.*//' >$TMP
    }
    
    D6() {
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=n-1;i>0;i--) print "."a[i];print".\n"}' $GEN|sort|
    awk 'BEGIN{n=split("% %",d1)}{ORS="";k=split($0,d2,".");
    for (i=n;i>0;i--)if(d1[i]!=d2[i]){n=--k;print(d1[k]=d2[k]);
    for (i=k-1;i>0;i--)print "."(d1[i]=d2[i]);print"\n"}}'>$TMP 
    }
    
    D7() {
    awk '{ORS="";n=split($0,a,"."); print a[n];
    for (i=--n;i>0;i--)print "#"a[i];print"#\n"}' $GEN|sort| 
    awk 'BEGIN{dom = "%%%"}{if(substr($0,1,length(dom))!=dom){print $0;dom=$0}}' |
    awk '{ORS="";n=split($0,a,"#");print a[--n];
    for(i=--n;i>0;i--)print "."a[i];print"\n"}' > $TMP 
    }
    
    D8() {
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"#"A[i]};print S"#"}' $GEN|sort| 
    awk 'BEGIN{dom = "%%%"}{if(substr($0,1,length(dom))!=dom){print $0;dom=$0}}' |
    awk '{n=split($0,A,"#");S=A[--n];{for(i=--n;i>0;i--)S=S"."A[i]};print S}' > $TMP 
    }
    
    D9() {
    awk '{n=split($0,A,".");S=A[n];{for(i=n-1;i>0;i--)S=S"#"A[i]};print S"#"}' $GEN|sort| 
    awk 'BEGIN{d = "%%%";k=length(d)}{if(substr($0,1,k)!=d){print d=$0;k=length(d)}}' |
    awk '{n=split($0,A,"#");S=A[--n];{for(i=--n;i>0;i--)S=S"."A[i]};print S}' > $TMP 
    }
    D10(){
    awk -F '.' 'BEGIN{ORS=""}{print $NF;for(i=--NF;i>0;i--)print "#"$i;print "#\n"}' $GEN|sort|
    awk 'BEGIN{d = "%%%";k=length(d)}{if(substr($0,1,k)!=d){print d=$0;k=length(d)}}' |
    awk -F '#' 'BEGIN{ORS=""}{print $--NF; for(i=--NF;i>0;i--)print "."$i;print "\n"}'  > $TMP 
    }
    
    D11(){
    awk -F '.' 'BEGIN{ORS=""}{print $NF;
    for(i=--NF;i>0;i--)print "#"$i;print "#\n"}' $GEN|sort|
    awk -F '#' 'BEGIN{ORS="";d = "%"}{if(index($0,d)!=1){d=$0;print $--NF;
    for(i=--NF;i>0;i--)print "."$i;print "\n"}}'  > $TMP 
    }
    
    D12() {
    awk -F '.' 'BEGIN{ORS=""}{print $NF;for(i=--NF;i>0;i--)print "#"$i;print "#\n"}' $GEN|sort|
    awk -F '#' 'BEGIN{ORS="";d = "%%%";k=length(d)}
    {if(substr($0,1,k)!=d){d=$0;k=length(d);print $--NF; for(i=--NF;i>0;i--)print "."$i;print "\n"}}' 
    }
    
    logger START BATCH TESTING
    LOAD
    for x in $TESTWHAT; do
    
    
    if [ $LINECNT == "Y" ] ; then
    TOT=$(echo ": ";wc -l < $GEN; echo lines)
    fi
    
    logger start "$x" "$TOT"
    
    ACOUNT=$(date +%s)
    eval $x
    BCOUNT=$(date +%s)
    
    TRUN=$(($BCOUNT-$ACOUNT))
    
    logger end "$x" Runtime: "$(($TRUN%3600/60)) min, $(($TRUN%60)) sec [$TRUN ss]"
    
    if [ $LINECNT == "Y" ] ; then
    TOT=$(echo "After parsing: ";wc -l < $TMP; echo lines) 
    logger "$TOT" 
    fi
    
    if [ $CATOUT == "Y" ] ; then
    cat $TMP
    fi
    
    rm -f $TMP  
    
    done
    rm -f $GEN
    
     
  13. jochen

    jochen LI Guru Member

    Code:
    ct 19 07:00:17 router user.notice root: start D1 :  33634 lines
    Oct 19 07:02:59 router user.notice root: end D1 Runtime: 2 min, 42 sec [162 ss]
    Oct 19 07:02:59 router user.notice root: After parsing:  25316 lines
    Oct 19 07:02:59 router user.notice root: start D5 :  33634 lines
    Oct 19 07:04:47 router user.notice root: end D5 Runtime: 1 min, 48 sec [108 ss]
    Oct 19 07:04:47 router user.notice root: After parsing:  25316 lines
    Oct 19 07:04:47 router user.notice root: start D11 :  33634 lines
    Oct 19 07:06:34 router user.notice root: end D11 Runtime: 1 min, 47 sec [107 ss]
    Oct 19 07:06:34 router user.notice root: After parsing:  25316 lines
     
  14. Beast

    Beast Network Guru Member

    WRTSL54GS @ 264 MHZ

    start D1 : 33634 lines
    end D1 Runtime: 2 min, 36 sec [156 ss]
    After parsing: 25316 lines
    start D11 : 33634 lines
    end D11 Runtime: 1 min, 49 sec [109 ss]
    After parsing: 25316 lines
     
  15. srouquette

    srouquette Network Guru Member

    Code:
    Oct 19 12:04:40 unknown user.notice root: end D1 Runtime: 4 min, 30 sec [270 ss]
    Oct 19 12:04:41 unknown user.notice root: After parsing:  25318 lines
    Oct 19 12:04:41 unknown user.notice root: start D5 :  33637 lines
    Oct 19 12:11:35 unknown user.notice root: end D5 Runtime: 6 min, 54 sec [414 ss]
    Oct 19 12:11:35 unknown user.notice root: After parsing:  25318 lines
    Oct 19 12:11:36 unknown user.notice root: start D11 :  33637 lines
    Oct 19 12:13:59 unknown user.notice root: end D11 Runtime: 2 min, 23 sec [143 ss]
    Oct 19 12:13:59 unknown user.notice root: After parsing:  25318 lines
    Yep, it's faster, good job :)

    I also updated the script to TRIM only the sources after 2, and removed the duplicated functions.
    I replaced [[ and ]] by [ and ], don't know why there were 2 of them, it works fine with 1.

    Adblock Script 3.5e
    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~, ray123, mstombs
    ## Dnsmasq Ed. v3.5e
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    (
    cat <<'ENDF'
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    USEWHITELIST="Y" # N/Y/R for remote
    
    # WHITELIST
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    BLACK=""
    
    USEPIXELSERV="N"
    PXL_ADR=192.168.1.2
    PXL_URL="http://myserver.com/pixelserv"
    
    GETS="1 2 3 4"
    TRIM_BEGIN=3
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    ## DO NOT EDIT BELOW
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    PXL="/tmp/pixelserv"
    
    PXL_SETUP() {
    if [[ "$USEPIXELSERV" == "Y" ]] ; then
    if [[ ! -x $PXL ]] ; then
    wget -O $PXL $PXL_URL
    chmod +x $PXL
    fi
    ifconfig br0:0 $PXL_ADR
    if [[ "$(pidof pixelserv)x" == "x" ]] ; then $PXL $PXL_ADR ; sleep 1 ; fi
    if [[ "$(pidof pixelserv)x" == "x" ]] ; then
    logger ADBLOCK ERROR cannot start pixelserv
    else
    eval "NIP=$PXL_ADR"
    fi
    fi
    }
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    if [[ $i -ge $TRIM_BEGIN ]] ; then TRIM ; fi
    cat $TMP >> $GEN
    logger ADBLOCK $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK Whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN
    done
    else
    logger ADBLOCK Cannot get whitelist $WURL
    fi
    fi
    echo "$BLACK" |sed 's/[ \t]*/\n/g'|sed '/^$/d' >> $GEN
    }
    
    CLN() {
    logger ADBLOCK Sort...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk -F '.' 'BEGIN{ORS=""}{for(i=NF;i>0;i--)print $i"#";print "\n"}' $GEN|sort|
    awk -F '#' 'BEGIN{ORS="";d = "%"}{if(index($0,d)!=1){d=$0;print $--NF;
    for(i=--NF;i>0;i--)print "."$i;print "\n"}}' > $TMP
    fi
    logger ADBLOCK Sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK $TOT Entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i 's|^.*$|address=/&/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    service dnsmasq stop
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK Dnsmasq is Running
    else
    logger ADBLOCK ERROR, Restarting Dnsmasq...
    dnsmasq
    fi
    }
    
    PXL_SETUP
    CLR
    DS
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ## End of ADBLOCK.sh
    ENDF
    ) > $ADB
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    
    edit:
    Added pixelserv support, based on these threads:
    http://tomatousb.org/forum/t-262221/adblock-setup-install-package
    http://www.linksysinfo.org/forums/showthread.php?t=62758&page=7

    Update these variables if you want to use it:
    Code:
    USEPIXELSERV="N"
    PXL_ADR=192.168.1.2
    PXL_URL="http://myserver.com/pixelserv"
    
    pixelserv will be downloaded from the specified URL. Instead of 0.0.0.0, dnsmasq will use the pixelserv IP.

    edit (b):
    updated blacklist sed.

    edit (c):
    removed -w when starting pixelserv.

    edit (d):
    fixed an error in the test to start pixelserv

    edit (e):
    updated the awk line: http://www.linksysinfo.org/forums/showpost.php?p=369182&postcount=227
     
  16. Larix

    Larix LI Guru Member

    After that my WRT54G v2.2 bricked. :( I have buy me a WRT320N.
    With that the Ad block Script 3.4a runs... :)

    Oct 19 13:31:07 Stargate user.notice root: ADBLOCK 33636 Entries
    Oct 19 13:31:07 Stargate user.notice root: ADBLOCK Sort...
    Oct 19 13:32:19 Stargate user.notice root: ADBLOCK Sorted.
    Oct 19 13:32:20 Stargate user.notice root: ADBLOCK 25317 Entries
     
  17. ~nephelim~

    ~nephelim~ LI Guru Member

    Thanks.

    I made also a quick test under Victek's tomato.RAF-K24-1.28.8515_lite (did not reset th NVRAM though) and found out that RAF firmware are able to perform the awk commands faster.

    To carry this little experiment I almost bricked my router when I flashed NoUSB build 52 back (it was probably my fault though I don't really understand what happened :confused: )

    That's good and it will squeeze some more space as well.

    It might be desirable to shorten S1 to "http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml" even using this no header will be added.

    In addition it would be possible to shorten
    sed -i -e 's|^|address=/|' -e 's|$|/'$NIP'|' $GEN
    to sed -i 's|^.*$|address=/&/'$NIP'|' $GEN

    And probably also

    esite=$(echo $site|sed -e 's/\./\\\./g')
    sed -i -e "/$esite/d" $GEN

    to
    sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN


    This would allow some space to set an inline blackist.

    eg:
    BLACK=""

    echo "$BLACK" |sed '/^$/d' >> $GEN

    to be applied just before CLN
    Found out a faq about the differences.

    I'm still confused about whenever [ is builtin or not.
    Probably [ is a builtin command as well in tomato whereas in some older shells it might be implemented though a [ command. :confused:
     
  18. srouquette

    srouquette Network Guru Member

    Ok, thanks.
    I added your suggestions and reverted [ back to [[ (I couldn't find an explanation, thanks for the link). If [[ has an impact on tests, it's safer to keep it the old way.
    I didn't test if "$BLA" == "Y" was working with [ ].
     
  19. ~nephelim~

    ~nephelim~ LI Guru Member

    Thanks :)

    I used single square brackets for the benchmark script and it seemed to work but it was not intentional.

    The table in the faq also made it look that == cannot be used with single square brackets (though I did and it worked :confused: )

    It will probably be safer to keep the syntax like it used to be.
    It's fine. Emptying the backlist variable will do just fine to disable it and the extra space could be used to add more entries if actually needed.

    edit: Just a word of caution for those who intend to use the inline backlist.

    The hosts are to be separated by newlines without spaces anywhere in between the double quotes

    eg:

    BLACK="hosta.com
    hostb.com" #is fine

    BLACK="
    hosta.com
    hostb.com
    hostc.con
    " #is fine


    BLACK="hosta.com
    hostb.com " #is not fine

    BLACK="hosta.com hostb.com hostc.con" #is not fine

    edit2:
    a whitespace-friendlier revision, just in case the above mentioned limitation cause some concern

    echo "$BLACK" |sed 's/[ \t]*/\n/g'|sed '/^$/d' >> $GEN
     
  20. ~nephelim~

    ~nephelim~ LI Guru Member

    I adapted an idea to compress NVRAM for use with copy&paste tomato scripts.

    It needs openssl (base64 enc/dec) and gzip to work (both are available on build 52 NoUSB)

    To test if a router support this approach all that is needed is to past this in Tomato's tools\system

    Code:
    [B]{
    cat <<'ENDF'|gzip|openssl enc -base64|openssl enc -base64 -d|gunzip[/B]
    gzip/base64 supported!
    [B]ENDF
    }[/B]
    

    to compress something
    Code:
    {
    [B]cat <<'ENDF'|gzip|openssl enc -base64 [/B]
    something
    [B]ENDF
    }[/B]
    
    This will output the encoded text on stdout.

    The resulting adblock based on this approach is something like this
    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~, ray123, mstombs
    ## Dnsmasq Ed. v3.5b
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    [B]{
    cat <<'ENDF' > $ADB[/B]
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    USEWHITELIST="N" # N/Y/R for remote
    
    # WHITELIST
    WURL="http://myserver.com/whitelist.txt"
    WHITE="feedburner.com
    intel.com"
    BLACK=""
    
    USEPIXELSERV="N"
    PXL_ADR=192.168.1.2
    PXL_URL="http://myserver.com/pixelserv"
    
    GETS="1 2 3 4"
    TRIM_BEGIN=3
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    ## DO NOT EDIT BELOW
    [B]ENDF
    }[/B]
    
    [B]{
    cat <<'ENDF'|openssl enc -base64 -d|gunzip  >> $ADB[/B]
    H4sIAIcqv0wCA6VWbVPiSBD+nl/RO+YusLeTwJb3UmKsUgxKiUAleO6VqJVNBpha 
    SLgkinvif7+Zngkgi1dWXQky6e7pp9873XbfJTUb/4hx5nVd4hSzuTNmCTEGl339 
    WLDZnBjNXrelCVGajIhx3gsGgaZM0rzIiXHdanc8TVpMeMGI0f/S0YQ5f2LTnGWP 
    xJDU+8AbXPUrVXg2+AhuboCYV4HXb3/xOoHn/0nAdYH8ReD2FhpQTFiixT4AfQJT 
    KNjgLMasANpTZPnv/srvGNFklsbwi5I2RlwokJbzMXzNagc1JXh86q/wK3MepyNY 
    GVp9ElYAedowQkHQxeqyoOdTxuZQFyfEeLcuY5qOxyyD49OTTq95AZ7v93yIwiRJ 
    C8iLMCvW1w35Y7DHcAqkK9JWwhPpl/i8GEaz42M0sxnQEZgineVRpLI8yiyWZ8yW 
    vDnw25d4NWcxUM7AMnJnuGd/NB1HnO7qn38f2jX81G9gWNx+RLo6apmDg03WNI3C 
    qSwKyXXuTCe2lBkC7TRArFGaAQeeSEsHQQPiVLn3kE1dMjQDkxMZTEytKYgyvxSW 
    UGRAYyDDjMAR6nxdHiYHOmaCIXy6P/HO2t2N5EmiSlMUFury0ZEK1VYuJKKK+c4k 
    tcQXRYSqOE2Y9KtzfT7YLufr8/bA67SDwY5ylgHIRY9gDFBQqNIpAOKYFRZNUjCl 
    yFJSLZET2xmK79iqOjFRdiM6m76N6W9W7zqi16JDdocUy+KNEr2WTT3luVZQWktF 
    yeiCgVXlWIqsk6+K7ZXPFUwCMqr/w/Ufc9RULST9XGwZrLoF1RPzpHPcvCBQQuj6 
    HSYCRdG08WWNYJN1McdbiEGaFbZtb6Yex+OOtOdCFOgD0FQVIGpGL8LFN6AtsGwL 
    LKzc554vBix5eZ5nPBEedFsNGcEKdykVZ35Ua3BKq4pL9ojJG+V5mJAXC3UvJeBy 
    pXxvS3kjlmPpJwHCRxWexOypYtY+xdUPbr36HLtmTas0EfI/4O0NeIku4FV/yoDv 
    CBeLbWP2WM6ndYC7qokGvYFrVhYR0CkcIrf6Q48KGfCSIuMsl1dbp93gcrsD30pD 
    NC+nJJgoo9fFb/v7JWGjuPPl3dIyxeC1YGnt4Ne5mJG4SOuwmn0rQcyulpaqRHss 
    wzjOWJ67zs+OUrxU6dLDvNNsnb3TFcBh5rAicuIkn4X537Zcc1i1OO4lu3yAw0Ov 
    1zIEekJxabvr/W1IDtr6tsK1hb3+62knntuX7cDbZeB7nFC7TbjtoqFkc+MhVUDj 
    tlu50zrT3sSTaE7Dh2KSZrwIC/4oPYgmjOb8H+Z+ru3/IUuHhvn3JHJ/RTd1lE86 
    F3rxZY88YqD9Fds3nb/L6vICpTJEdMSnTNmvzN/JXsewhasQ4zPPxSAeZ+JFgnpA 
    9D2ySUvSr2n8naygt7rhVEPxHPyHJOHJ+O399Ql8hi8YQqq8KMeXhtXWrV7S5KuF 
    WNu44WSDyimIv9hxWKuyHDCcwicU/xdL1+bqXAoAAA== 
    [B]ENDF
    }[/B]
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    

    If most tomato variants support this approach it will be possible to squeeze some more space :rockon:
     
  21. srouquette

    srouquette Network Guru Member

    hehe, that's awesome :)
    next time we'll hit the limit, we'll use this ;)
     
  22. ~nephelim~

    ~nephelim~ LI Guru Member

    It looks like the latest revision is 4165 bytes.
     
  23. srouquette

    srouquette Network Guru Member

    yes, but it seems to work with Tomato when I save the WAN Up script...
     
  24. ~nephelim~

    ~nephelim~ LI Guru Member

    You're right.

    Firefox and chrome works fine.

    Opera trigger a warning. :confused:

    Was the 4096 bytes limit superseded in recent builds?
     
  25. srouquette

    srouquette Network Guru Member

    I don't know, noone reported a problem ^^;
     
  26. ~nephelim~

    ~nephelim~ LI Guru Member

    Probably they don't use opera or this affect only the latest 10.70 snapshot.

    As it turned out the script is 3946 bytes if LF formatted and 4192 if CRLF formatted.

    Firefox and Chrome store the script in a LF formatted variable while Opera and Internet Explorer use a CRLF formatted one.
     
  27. jochen

    jochen LI Guru Member

    In D11 the first awk statement:
    Code:
    awk -F '.' 'BEGIN{ORS=""}{print $NF;
    for(i=--NF;i>0;i--)print "#"$i;print "#\n"}' $GEN|sort|
    can be written shorter:
    Code:
    awk -F '.' 'BEGIN{ORS=""}{for(i=NF;i>0;i--)print $i"#";print "\n"}' $GEN|sort|
     
  28. srouquette

    srouquette Network Guru Member

    Is it faster?
    I updated the script with it.
     
  29. jochen

    jochen LI Guru Member

    No, this mod has no influence on speed. It's just a bit cleaner.
     
  30. ~nephelim~

    ~nephelim~ LI Guru Member

    It performs slightly slower for me.

    Code:
    Oct 23 17:56:45 router user.notice root: START BATCH TESTING
    Oct 23 17:56:45 router user.notice root: ADBLOCK Retrieved http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Oct 23 17:56:48 router user.notice root: ADBLOCK Retrieved http://mirror1.malwaredomains.com/files/justdomains
    Oct 23 17:56:50 router user.notice root: ADBLOCK Retrieved http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 23 17:56:59 router user.notice root: ADBLOCK Retrieved http://www.mvps.org/winhelp2002/hosts.txt
    Oct 23 17:56:59 router user.notice root: start PM 
    Oct 23 17:58:13 router user.notice root: end PM Runtime: 1 min, 13 sec [73 ss]
    Oct 23 17:58:13 router user.notice root: start P11 
    Oct 23 17:59:14 router user.notice root: end P11 Runtime: 1 min, 1 sec [61 ss]
    Oct 23 17:59:14 router user.notice root: start P5 
    Oct 23 18:01:07 router user.notice root: end P5 Runtime: 1 min, 53 sec [113 ss]
    Tested with (P)artial implementations of the first transformation step without sorting

    Code:
    #BEWARE THIS IS NOT AN ADBLOCK AND TAKES A LONG TIME TO RUN
    TESTWHAT="P11MOD P11 P5"
    LINECNT="N"
    CATOUT="N"
    GETS="1 2 3 4" 
    
    TMP="/tmp/tmp1"
    GEN="/tmp/tmp2"
    
    TRIM(){
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/^::1[ \t]*//
    s/[ \t]*$//
    s/localhost$//
    /^$/d' $TMP 
    }
    
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"                 #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"                 #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"                           #620K
    S5="http://hosts-file.net/hphosts-partial.asp"                           #283K
    S6="http://hostsfile.mine.nu/Hosts"                                     #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"                   #3851K
    
    
    F3() { TRIM; }
    F4() { TRIM; }
    F5() { TRIM; }
    F6() { TRIM; }
    F7() { TRIM; }
    
    LOAD(){
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    eval F$i
    cat $TMP >> $GEN
    logger ADBLOCK Retrieved $url
    else
    logger ADBLOCK ERROR FOR $url
    fi
    done
    rm -f $TMP 
    }
    
    P5(){
    awk -F '.' '{print $9 "~" $8 "~" $7 "~" $6 "~" $5 "~" $4 "~" $3 "~" $2 "~" $1}' $GEN |
    sed -e 's/^~*//' -e 's/$/~/' > $TMP
    }
    
    
    P11(){
    awk -F '.' 'BEGIN{ORS=""}{print $NF;
    for(i=--NF;i>0;i--)print "#"$i;print "#\n"}' $GEN  > $TMP
    }
    
    P11MOD(){
    awk -F '.' 'BEGIN{ORS=""}{for(i=NF;i>0;i--)print $i"#";print "\n"}' $GEN  > $TMP
    }
    
    logger START BATCH TESTING
    LOAD
    for x in $TESTWHAT; do
    
    
    if [ $LINECNT == "Y" ] ; then
    TOT=$(echo ": ";wc -l < $GEN; echo lines)
    fi
    
    logger start "$x" "$TOT"
    
    ACOUNT=$(date +%s)
    eval $x
    BCOUNT=$(date +%s)
    
    TRUN=$(($BCOUNT-$ACOUNT))
    
    logger end "$x" Runtime: "$(($TRUN%3600/60)) min, $(($TRUN%60)) sec [$TRUN ss]"
    
    if [ $LINECNT == "Y" ] ; then
    TOT=$(echo "After parsing: ";wc -l < $TMP; echo lines) 
    logger "$TOT" 
    fi
    
    if [ $CATOUT == "Y" ] ; then
    cat $TMP
    fi
    
    rm -f $TMP  
    
    done
    rm -f $GEN
    
     
  31. srouquette

    srouquette Network Guru Member

    Maybe the order in your tests is important.

    Code:
    Oct 23 18:26:53 unknown user.notice root: start P11 
    Oct 23 18:28:50 unknown user.notice root: end P11 Runtime: 1 min, 57 sec [117 ss]
    Oct 23 18:28:51 unknown user.notice root: start P11MOD 
    Oct 23 18:30:05 unknown user.notice root: end P11MOD Runtime: 1 min, 14 sec [74 ss]
    Oct 23 18:30:05 unknown user.notice root: start P5 
    Oct 23 18:31:51 unknown user.notice root: end P5 Runtime: 1 min, 46 sec [106 ss]
    
     
  32. ~nephelim~

    ~nephelim~ LI Guru Member

    Probably results are more likely to be affected if the router carry other tasks while the batch test run.

    Code:
    Oct 23 18:41:40 router user.notice root: START BATCH TESTING
    Oct 23 18:41:41 router user.notice root: ADBLOCK Retrieved http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Oct 23 18:41:43 router user.notice root: ADBLOCK Retrieved http://mirror1.malwaredomains.com/files/justdomains
    Oct 23 18:41:46 router user.notice root: ADBLOCK Retrieved http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 23 18:41:56 router user.notice root: ADBLOCK Retrieved http://www.mvps.org/winhelp2002/hosts.txt
    Oct 23 18:41:56 router user.notice root: start P11 
    Oct 23 18:42:57 router user.notice root: end P11 Runtime: 1 min, 1 sec [61 ss]
    Oct 23 18:42:57 router user.notice root: start P11MOD 
    Oct 23 18:44:10 router user.notice root: end P11MOD Runtime: 1 min, 13 sec [73 ss]
    Oct 23 18:44:10 router user.notice root: start P5 
    Oct 23 18:46:03 router user.notice root: end P5 Runtime: 1 min, 53 sec [113 ss]
    Oct 23 18:46:03 router user.notice root: start P11MOD 
    Oct 23 18:47:16 router user.notice root: end P11MOD Runtime: 1 min, 13 sec [73 ss]
    Oct 23 18:47:16 router user.notice root: start P11 
    Oct 23 18:48:18 router user.notice root: end P11 Runtime: 1 min, 2 sec [62 ss]
    I disable wireless and don't use the router while the test is running.
    I just leave the script running and walk away without surfing the internet meantime.
     
  33. srouquette

    srouquette Network Guru Member

    I really don't understand how awk works ^^;
    In P11MOD, there's one less print, and one less operation (--NF), but it takes more time...
     
  34. ~nephelim~

    ~nephelim~ LI Guru Member

    No clue here as well.

    I though about the loop but even removing the sed statement from P5

    Code:
    P5(){
    awk -F '.' '{print $9 "~" $8 "~" $7 "~" $6 "~" $5 "~" $4 "~" $3 "~" $2 "~" $1}' $GEN > $TMP
    }

    I got these results

    Code:
    Oct 23 19:28:32 router user.notice root: START BATCH TESTING
    Oct 23 19:28:32 router user.notice root: ADBLOCK Retrieved http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Oct 23 19:28:35 router user.notice root: ADBLOCK Retrieved http://mirror1.malwaredomains.com/files/justdomains
    Oct 23 19:28:37 router user.notice root: ADBLOCK Retrieved http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 23 19:28:46 router user.notice root: ADBLOCK Retrieved http://www.mvps.org/winhelp2002/hosts.txt
    Oct 23 19:28:46 router user.notice root: start P11 
    Oct 23 19:29:47 router user.notice root: end P11 Runtime: 1 min, 1 sec [61 ss]
    Oct 23 19:29:48 router user.notice root: start P5 
    Oct 23 19:31:30 router user.notice root: end P5 Runtime: 1 min, 42 sec [102 ss]
    Oct 23 19:31:30 router user.notice root: start P5 
    Oct 23 19:33:13 router user.notice root: end P5 Runtime: 1 min, 43 sec [103 ss]
    Oct 23 19:33:13 router user.notice root: start P11 
    Oct 23 19:34:15 router user.notice root: end P11 Runtime: 1 min, 2 sec [62 ss]
    
    Oct 23 19:39:34 router user.notice root: START BATCH TESTING
    Oct 23 19:39:35 router user.notice root: ADBLOCK Retrieved http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Oct 23 19:39:38 router user.notice root: ADBLOCK Retrieved http://mirror1.malwaredomains.com/files/justdomains
    Oct 23 19:39:40 router user.notice root: ADBLOCK Retrieved http://www.malwaredomainlist.com/hostslist/hosts.txt
    Oct 23 19:39:49 router user.notice root: ADBLOCK Retrieved http://www.mvps.org/winhelp2002/hosts.txt
    Oct 23 19:39:49 router user.notice root: start P5 
    Oct 23 19:41:32 router user.notice root: end P5 Runtime: 1 min, 43 sec [103 ss]
    Oct 23 19:41:32 router user.notice root: start P11 
    Oct 23 19:42:33 router user.notice root: end P11 Runtime: 1 min, 1 sec [61 ss]
    Oct 23 19:42:33 router user.notice root: start P11 
    Oct 23 19:43:34 router user.notice root: end P11 Runtime: 1 min, 1 sec [61 ss]
    Oct 23 19:43:34 router user.notice root: start P5 
    Oct 23 19:45:29 router user.notice root: end P5 Runtime: 1 min, 55 sec [115 ss]
    
     
  35. zatoom

    zatoom Addicted to LI Member

    Good day.

    (Translated ImTranslator 3.3.5) I can read English but not write.
    (Linksys WRT54GL)
    (newbie)

    I used not long Tomato, my current version is v1.28.1816
    I've seen the latest version of Adblock script ver. 3.5e is
    My question is: where should I drop the script.
    [Admin / scripts / Wan Up?} Or somewhere else?
    It is now on [admin / scripts / Wan UP] but does not work, I think. :confused:

    I've just caught a URL from the list: http://support.it-mate.co.uk/downloads/hosts.txt. > Cyclus.net < I could do the site, which I think may be impossible.
    How can I fix this. please with examples.

    Greetings Ed.
     
  36. srouquette

    srouquette Network Guru Member

    Yes, you should put the script in Wan UP.
    Did you reboot your router after copying the script?
    What is the error? Do you have a popup which say "script is too big"?
    If it's the problem, you can try to use Nephelim's script, which is compressed, or you can try to save the script with firefox or chrome.

    About the URL, your router doesn't have enough memory to read it, you mustn't use S6 and S7 if your router doesn't have at least 16 MB (I think).
    By default, S6 and S7 aren't enabled. Read the log to see if Adblock is active, then try a websites from S1 to S4.
     
  37. zatoom

    zatoom Addicted to LI Member

    Thanks for the reply.
    Yes I did a reboot and got no error on a script that is too large.
    I've tried a few other URLs and S6 and S7 alone did not.
    Which I have switched off with a ##

    Thanks, it works just fine.

    Ed.
     
  38. srouquette

    srouquette Network Guru Member

    You don't need to comment anything.
    This line GETS="1 2 3 4" gets the first four URL, it doesn't get 5, 6 and 7.
    If you only want to download S1, write GETS="1".
    It's written in the logs which URL are downloaded... :)
     
  39. zatoom

    zatoom Addicted to LI Member

    That makes everything clear and understandable.

    Thanks.

    Ed.
     
  40. ~nephelim~

    ~nephelim~ LI Guru Member

    I made an hash check to abort script execution meant to be placed before DS

    Code:
    TST(){
    MD5=$(md5sum $GEN|cut -d " " -f1)
    if [[ -f $GEN.md5 ]] && [[ $MD5 == $(cat $GEN.md5) ]];  then
    logger ADBLOCK: "(NO CHANGE) MD5 $MD5"
    CLR; exit
    else
    echo $MD5>$GEN.md5
    logger ADBLOCK: "(UPDATED) MD5 $MD5" 
    fi
    }
    There is a catch though:

    TST is unable to acknowledge if dnsmasq was restarted using advanced-dhcpdns.asp (this mean no active blacklist) and will abort regardless of that fact each time the previous hash (for a blacklist that is not available anymore) match what DS just downloaded.

    Using an alternate dnsmasq loading (instead of directly launching dnsmasq --conf-file=$GEN)

    Code:
    DNS="/etc/dnsmasq"
    echo "conf-file=$GEN" > $DNS.custom
    service dnsmasq start
    rm -f $DNS.custom
    
    it would be possible to detect this and remove the old md5 file just before TST executes:
    Code:
    if ! grep $GEN $DNS.conf ; then 
    rm -f $GEN.md5
    fi
    Due to the alternate loading method /etc/dnsmasq.conf will contain a conf-file=/tmp/gen line as long the service is not restarted using advanced-dhcpdns.asp page or service dnsmasq start/restart (from the console)

    The additional conf-file=/tmp/gen line will not be removed and cause an error if dnsmasq is manually restarted without using the service command.



    EDIT It might be possible to retain dnsmasq --conf-file=$GEN usage though if a #comment is appended to /etc/dnsmasq.conf
    and used to detect advanced-dhcpdns.asp restarts without incurring in failures when dnsmasq is not invoked through the service command


    Code:
    DNS="/etc/dnsmasq"
    TOKEN='###uniqueid###'
    if ! grep $TOKEN $DNS.conf ; then 
    rm -f $GEN.md5
    echo $TOKEN>>$DNS.conf
    fi
     
  41. jochen

    jochen LI Guru Member

    What is the benefit when testing if there was a change? The main work is downloading and optimizing the list. I see no advantage in not restarting dnsmasq every time.
     
  42. jochen

    jochen LI Guru Member

    Double post
     
  43. ~nephelim~

    ~nephelim~ LI Guru Member

    For lists S1-S4 the downloading + sed-processing steps take nearly 10 seconds.

    It's too bad most lists don't provide an hash file or else it would have been possible to download only the hash files and compare with a local copy to skip full download and sed processing at once.




    The hashing operation itself takes about one second on a 570.7K post processed file (S1 to S4) same result with a 1.7M post processed file (S6)

    With such a small overhead it can provide notification about MD5 changes in the downloaded list.




    With an 1 second hashing overhead it would be also possible to skip the second awk/sort processing step (which takes minutes) and (optional) dnsmasq stop, though.

    (In my case dnsmasq is stopped just before the second awk/sort-processing step to prevent domain resolution until a new list is loaded.)




    Overall, though hash checking can simply provide an informative benefit, the related abort routine would be able to save minutes wasted in unnecessary processing and related (optional) downtime.

    Time-saving benefits would be proportional to the frequency of daily updates and list-size.

    Of course these time-saving considerations don't apply if wan or dnsmasq are manually restarted (eg using tomato GUI): in that case the list will be applied even if the MD5 hash would have been matched, upon grep-detection of such before-script restart condition (can be optionally notified in the log as well since in non-wan dnsmasq-restart cases it means the blacklist was not active when the script started)
     
  44. srouquette

    srouquette Network Guru Member

    Interesting... maybe you could add this test in TST: [[ $# == 0 ]]
    It tests if there's a argument on the command line, this way you can force the update with "./ADBLOCK.sh --force".
    I'll add it when everything is settled.


    There's a problem in the script, if someone can figure out how to fix it. When I use pixelserv, pixelserv is restarted a lot of time, and multiple instances end up in memory. The test in PXL_SETUP seems valid, but the problem should come from there (unless pixelserv fork itself...) because the process is restarted with the IP from command line.
    Can someone else check that please?
     
  45. ~nephelim~

    ~nephelim~ LI Guru Member

    It might be possible to test specifically for this before the grep-test

    Code:
    if [[ $# != 0 ]] ; then
    logger ADBLOCK: Forced Execution
    rm -f $GEN.md5
    fi
    
    The removal of $GEN.md5 file will force execution during TEST() operation ([ -f $GEN.md5] )

    compound statement would be something like
    Code:
    if !  grep $GEN $DNS.conf || [[ $# != 0 ]] ; then 
    rm -f $GEN.md5
    fi
    
    I ran multiple times
    Code:
    ifconfig br0:0 192.168.1.2
    /tmp/pixelserv 192.168.1.2
    ps
    in tools-shell.asp.

    Even without pid check there is only one pixelserv instance running.

    Perhaps the issue is caused by pixelserv hanging :confused:
     
  46. srouquette

    srouquette Network Guru Member

    Ok thanks for the test.

    adblock 3.6b.
    You need OpenSSL (available on TomatoUSB) on your router to use this one.
    Use 3.5e if you don't have OpenSSL.
    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by Syl, jochen, groosh, ng12345, ~nephelim~, ray123, mstombs
    ## Dnsmasq Ed. v3.6b
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    {
    cat <<'ENDF' >$ADB
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    
    USEWHITELIST="Y" # N/Y/R for remote
    WURL="http://example.com/whitelist.txt"
    WHITE="intel.com"
    BLACK=""
    
    USEPIXELSERV="N"
    PXL_ADR=192.168.1.2
    PXL_URL="http://example.com/pixelserv"
    
    GETS="1 2 3 4"
    TRIM_BEGIN=3
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    ENDF
    }
    
    {
    cat <<'ENDF'|openssl enc -base64 -d|gunzip >>$ADB
    H4sIACIhzEwCA6VWbW/aSBD+7l8x3fhi6NU2VO3dCepIeYEUlQDC5NJTaCvXXmBV
    sKltEnoh//1mZ9fgpKSKdApJltndeeblmZk9Hp77nnlg9DoDj9Uc+mHGeavnMTdf
    LN0pj5kxuhjorzlfLJlx2u+1tSBM4gkz3vf9ka8lsyTLM2ZctTvdlhbdzkTOmTH4
    2NWCpVjzecbTG2ZI6Re/NbocVKpwZ4gJXF8DMy/91qDzsdX1W8O/GXgesH8YfPoE
    TchnPNbHXoC9BhMVlHZupzwHu6/E8s+Xy2HXCGeLJILf1WljIlCBtFxM4Wtaa9TU
    weOz4Ra/shRRMoGtodW1smJdsqKEgVdRmM05X0IdV4TwfE3GPJlOeQrHZyfd/ukH
    aA2H/WEDwiCOkxyyPEjznQJD/jP4TTAHJtNWGMCkX/i5N4zT7pCimS7AnoCJ6SyW
    mMpiKbNYrClb8uZo2LmgqxmPwBYcLCNzxwfOS9N1cfW5/vrPsVOjT/0axvmnlyRX
    S32m0ShvzZMwmEtSyF33s+lGljID0c58wpokKQgQsbR05DchSpR7q3TusbHpm4LJ
    cFJqTRTK/NqwgTwFOwI2Thkckc6H9DAF2FOOG+jTl5PWeadXyp0UqkSFQa4uHx2p
    UD1MRoMgVdB/mabCOpmDKIk5hdMfVaoFq81jLDZ44UFN2rHZFPzI0JVpisyxMdy2
    LYlpFSIrTr4m0Q+rqkgjOaN93OXWWURvJejF2VvPrOCXbLWgjU24yilC+GNP6lVt
    R+maNOTwkIzD2xLCrFA89H6V8PZxtAFZsOCAt15B9k0slyKewmoZBTl3JP+awNci
    11wNZwkBHJXNxfB0r96PHpf91fvOqNXt+KM9ZS+JkmEvIa7QQYy0piow16woJHlk
    I6UWctdxx/g7tapuxFR+KTl8/jTmsFzkO+ZdYSfZTz0qn/213ADqfnORaQ2FuTLZ
    urJgW2KWEusqUVX5wGmVHdqo/g/fn8Plx3ar7kIozDzpHp9+YFAg6XofxwimZNqH
    oqaoKfUo1z/xKElzSR4aHI7jlMlAg2UPEeQdsFdgJ6p0CYPcCm6/gd0Gy7HAopq/
    6w9xNLH7O4xjRXi9dlMc1ZrCtqvLVMTonGAHrKnWbByze0vVjoTYbNUdPFLXjABt
    +g3ViklFxBFfV8zaq6j6wqtX7yLPrGmNpm0joqGwaf0QnTnMFGV0hFe9TAb7cago
    RBQwHjnG4qZo6bsY91Q9jfojbAW3IdhzeEe71Z/bGh4CHuep4Jm82z7r+RePq/Gp
    BITLoo2ASWf0iP3jzZtCUOJ5tvm8sUwcVtjXrD37dYFzhR4fddjOi+1Byqs+LVVh
    pWyCKEp5lnnuoasUb1TaisZy2j5/pitAA8DleehGcbYIsu+O7MDEXBqRcrv4Au/e
    tfptA9Fjm5Lh7d48htwhW59WuLOwP3jY+fB756Ljt/YZ+Bwn1HsA3fbIUFZ+JZAU
    oemFsHWnfa69iWbh0g5W+SxJRR7k4kZ6EM64nYl/ufe69uYvyR07yH7EofeW3NRR
    Pul+0I+F9EaEHLS/+GJJls+yurigpp49EXOu7Ffm793exbBNzweKT2mItoDpe6ws
    U4OUbaEfl0OBJTJIV3GMHelXfRLJJ19lsm8VScbOpZfavO3LVs5DfOvI1wCNPFmm
    sh3Sfyo7IqzkBMUUHaMrxn/iIe1CngsAAA==
    ENDF
    }
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    
    gzip block generated with:
    Code:
    #!/bin/sh
    {
    cat <<'ENDF'|gzip|openssl enc -base64
    ARGS=$#
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    PXL="/tmp/pixelserv"
    
    PXL_SETUP() {
    if [[ "$USEPIXELSERV" == "Y" ]] ; then
    if [[ ! -x $PXL ]] ; then
    wget -O $PXL $PXL_URL
    chmod +x $PXL
    fi
    ifconfig br0:0 $PXL_ADR
    if [[ "$(pidof pixelserv)x" == "x" ]] ; then $PXL $PXL_ADR ; sleep 1 ; fi
    if [[ "$(pidof pixelserv)x" == "x" ]] ; then
    logger ADBLOCK ERROR: cannot start pixelserv
    else
    eval "NIP=$PXL_ADR"
    fi
    fi
    }
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    if [[ $i -ge $TRIM_BEGIN ]] ; then TRIM ; fi
    cat $TMP >> $GEN
    logger ADBLOCK: $url
    else
    logger ADBLOCK ERROR: cannot get $url
    fi
    done
    }
    
    TST(){
    if [[ $ARGS != 0 ]] || [[ "$(ps | grep -e '--conf' | grep 'nobody')" == "" ]]; then
    rm -f $GEN.md5
    fi
    MD5=$(md5sum $GEN|cut -d " " -f1)
    if [[ -f $GEN.md5 ]] && [[ $MD5 == $(cat $GEN.md5) ]];  then
    logger ADBLOCK: same MD5, skipping update.
    CLR; exit
    else
    echo $MD5>$GEN.md5
    fi
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK: whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN
    done
    else
    logger ADBLOCK ERROR: cannot get whitelist $WURL
    fi
    fi
    echo "$BLACK" |sed 's/[ \t]*/\n/g'|sed '/^$/d' >> $GEN
    }
    
    CLN() {
    logger ADBLOCK: sorting hosts...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk -F '.' 'BEGIN{ORS=""}{for(i=NF;i>0;i--)print $i"#";print "\n"}' $GEN|sort|
    awk -F '#' 'BEGIN{ORS="";d = "%"}{if(index($0,d)!=1){d=$0;print $--NF;
    for(i=--NF;i>0;i--)print "."$i;print "\n"}}' > $TMP
    fi
    logger ADBLOCK: hosts sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK: $TOT entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i 's|^.*$|address=/&/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    service dnsmasq stop
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK: dnsmasq is running
    else
    logger ADBLOCK ERROR: restarting dnsmasq...
    dnsmasq
    fi
    }
    
    PXL_SETUP
    CLR
    DS
    TST
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ENDF
    }
    
    I added the test in TST. I needed to store ARGS, I suppose $# in a function is relative to this function.

    edit (a):
    Added test with grep --conf

    edit (b):
    fixed some stuff.
     
  47. ~nephelim~

    ~nephelim~ LI Guru Member

    Sorry, I didn't realize that. :frown:

    Might be necessary to include also the test for dnsmasq restart before adblock.sh execution.
    Code:
    DNS="/etc/dnsmasq.conf"
    TOKEN='###uniqueid###'
    if ! grep $TOKEN $DNS ; then 
    rm -f $GEN.md5
    echo $TOKEN>>$DNS
    fi
    
    at the beginning of TST()

    This will remove $GEN.md5 and force execution if dnsmasq lost its blacklist after being restarted (eg advanced-dhcpdns.asp).
     
  48. srouquette

    srouquette Network Guru Member

    I don't understand this blacklist part.
    When does it happen?
    When you save new settings in advanced dns?
    If you do that, dnsmasq is restarted without adblock, so you have to run it manually.
     
  49. jochen

    jochen LI Guru Member

    If you have a web server in your LAN (most NAS have one builtin), you can do what I did:

    I configured the server to use virtual hosts and made one which I do not use otherwise the default host. This default host contains only a 404 error page with this php script:

    Code:
    <?php
    header("HTTP/1.0 200 OK");
    
    $url = $_SERVER['REQUEST_URI'];
    $host = $_SERVER['HTTP_HOST'];
    
    if (preg_match('/^[0-9.]*$/',$host)) { // called by IP address
      sleep(600);
    }
    
    if (preg_match('/\.(gif|jpg|png)$/i',$url) || strpos($url,'.') === FALSE) {
      header("Content-Type: image/gif");
      readfile('1pixel.gif');
    }
    
    ?>
    
    Now I forward all blocked sites to my server. This responds either with an empty html file or with a 1 pixel gif, depending on the called uri.

    If the script is called with a IP address instead of a hostname, it is invoked by a hacker scanning ip adresses and ports. I block the connection with a sleep in this case.

    I also count blocked sites in a mysql database, so I can gather statistics about blocking. But this is not contained in the code above.
     
  50. ~nephelim~

    ~nephelim~ LI Guru Member

    If I remember correctly saving (without changing anything) in advanced-dhcpdns.asp (Or perhaps tomato might also restart dnsmasq for other unknown reasons) will restart dnsmasq while the blacklist is not available anymore

    if a $GEN.md5 file is there TST will not acknowledge that no blackist is actually running and testing $ADS.md5 is moot.
    In that case if $GEN.md5 match what DS downloaded adblock.sh would abort even if no blacklist is actually active.

    Instead of testing a comment added to dsnmasq.conf I guess it would be possible to grep for dnsmasq --conf in ps output (which is a callsign of a running blacklist). in no dnsmasq --conf is running $GEN.md5 will be removed before the core TST routine
     
  51. srouquette

    srouquette Network Guru Member

    Good idea, I updated the script.
    edit: updated too quickly, it doesn't work, I'll get back to you in a sec.
    edit2: fixed.
     
  52. ~nephelim~

    ~nephelim~ LI Guru Member

    This is a script meant for the firewall section which got an 8kb size limit.

    Not sure if this one will be much of use as it will probably require too much space for those running custom QOS scripts.


    Code:
    # pixelserv v16 by mstombs
    # http://www.linksysinfo.org/forums/showpost.php?p=365338&postcount=69
    # base64 decoder by Danny Chouinard's
    PXL="/tmp/pixelserv"
    dec="openssl enc -base64 -d"
    
    b64(){ awk 'BEGIN{b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
    while(getline<"/dev/stdin"){for(i=1;i<=length($0);i++){c=index(b64,substr($0,i,1));
    if(c--){for(b=0;b<6;b++){o=o*2+int(c/32);c=(c*2)%64;if(++obc==8){if(o)
    {printf"%c",o}else{system("echo -en \"\\0\"")}obc=0;o=0;}}}}}}';}
    
    if [[ ! -x $PXL ]] ; then
    [[ "$(echo MA==|$dec)" != "0" ]] && dec="b64" 
    {
    cat <<'ENDF' | $dec |gzip -d > $PXL
    H4sIAB0AzEwCA+0abWwUx/Xtec8Yg2EBg4/ghL1w+A4Cx5k4keMQ6cDmw4kBJyFt 
    qhCdz3fr84X70u4CdoTUI6EJPyzsqqgiUtRaGAd+RDqH5Ed+EGFVqRRVVcuPSkVq 
    pfpHpFRRPxKpSt2K9Pre7OzdeLm9QH9WjD03+2be17x5OzM7b364f+CAJElgJw80 
    AUGDK6LQhWXfowBeLLtAhUYIQRtswBJYWxdEWZ7HZ8qEJ2NuwPwPlqMst+JzK2/j 
    kgpN+NAkRVn2YYWPtws4mKIsdz8ELBN/UHi7ChwChWTfwtzIazzULFGOsvwUNlBe 
    xvnvyqSHd2WSO0/24kMibOTDEaK8Do6kFkEhWZvxZyWWx5dFoYWUb4syPgvIm3S6 
    7Y0y2aGGKDRjeRbxV3DlNiA6lsp6zpNsg9KIb0GybFbwWLoXqK9IXyA912D5GJZr 
    SRyW67BcBfeftmJ+jD+vxhwAyw7LSV9eT31p57p18DrFksnSNsyr+BhR/7ZgRreA 
    h3n7Zl628NIev7W8JNu1WX0CHEZmm3rJD9YYbhLq1lu2ZH70iFDf6KBt4uVGzNu5 
    T9ZKqmAfSsH7tOvLUsVM+LMCtkmWvwCz29dlWarageDd3B+B2eHrckzAVxB+HuGF 
    ZVHOtAnWS4y5YuGvhNck9o5V8Mljr62Mcjs3QcLR/rRk+Zkt74zE3pFK+ykH/ufk 
    lyuilXGTHLaiMVGF9hTRCfBz9CDAOcxfNVfhTfy9seDl8Hd6f4T2PnqHBXiQ5hQB 
    /g3mOQH+E+ZphCMc/hnmKaH9+5iLAhwj+QK8iHlIgJ8hmQJ8gWYeAX6H7CPAr5M9 
    BPg22UOA1/P5woKXwSw9CO1fkn2WV+G3yR4CTJPR/Irq+P6C7CO0f0vtAvwE2UeA 
    IfbsqdgLWiptmJrem4kbhmZALJ1LmxAbwQJiMWv6ixlmXDdj2Xg6B6fjabOQToKR 
    TuXiGTDGjUw+BdoYEmW1rKGZYJh6LlEYh3xBy1FbMq5l8zlIaWY8mdTTuZE8pOJp 
    5Klrup7XwcgnThCZZtJTvmDCcDqXhAyplYMRXdMqdCg0MZpJxkbjuWRG0wnWsgVz 
    nIlNp+IJM42S4omEhlxG8voJSGTyhoa8kaExetJM5k/nQEeGUECO5gjE+n5wZO/h 
    /t7YQP+R5/qPHMQ+vzDQFzu8dxCqE38sVYgl00ahgg2xgwNH9+0diB09cODF/cdi 
    x/buG9gfQ6uZ2hgZLxk340QFMc16jA0bhmVFbMVnbECNaHxWt+/5dOzdoH/hnYaF 
    8idBBW6U2uFmyUdl8Z1JwllseopNPcsX1QaozL03WJuNM7WhDg7cnLTe0U+CTbDQ 
    4JynSIeoQwdf8Q1GMw/7cDnYN2XrFIRoq01n48gWzoRV2zth17XBkFLFPctx/Yj7 
    6Pv+z4Zq6vpdPP9aHqrIl8ATsPnOW3yn3fpKfTx+2sXOdWhCLjS+4jlGE4JzAQv7 
    jQlrHYtW+hzhfVkm1Pk/i97V73ryr526P50ZT/QJ/9hCA8nZDr6ACG9zwCEH3OGA 
    tzrggAN+1AH7HbDqgDc74Ecc8MMOuN0Bb3LADzngjQ7Y54DbHPAGB7zeAa9zwGsd 
    8BoHrDjgVQg73zcJFNke389MGt+fN/hpsYErkzLMBiV4K1CMNIIa+SUUi29P/hnH 
    vo/7kgKXS138nRf9yId+UC6DUmsfQnKGzNp+VC7LAX+IdGoMvFuDL/kn4XxTjrbb 
    dfX8ddFYKkcCb8CWlXHh7wU50F78EWu71TYbGHDBkxBvj0ubLX+My4/++5MgyfzY 
    e7M05/2o9L73w9I17/XSFe8HpY+8N0rT3jmcXy2bLrTtC1jzy8EJmmtoXvFPFXFF 
    bQxcgyvBKe/lkv/SNO4TPcxWV+BqaRpmShexfrDmWNhzs8X/K87fA71TOKdd/FC+ 
    EiQ+Lcj/ujwzMydfvnovfMBnzYmfcvqVSC/BmsC87K7HdmW3ZxPOP/5pex9fwO27 
    98JO7AvATJHqiuoauCENV3xnGvusIO8hF54StAZKMD0jQWcH8ZdB6fiyXF0X2vnc 
    qPjOBbbg+PuQV9K1fyMX7P61Y/9sP+jyzeLz2+x50PdFoBHHnnzfxj3ms2zainUD 
    LrzJRn7cmHbXta0lL4ny8Os1MCpFZRvnljwXRHoP8XGzrwfteBvt/zvMZDcvFDeI 
    69MtrCcd5uTrQX+EeEU9E96rwagLvzYaK9Xao4259SsSreh9DvVuQNsMuI6VHFgn 
    VeeFCe/lSWudOjAhQ99EE+yfmHChjaAt/Oej3G+UAOlVLnu7ZLTTFPZBgblSwYVW 
    ASmgEr1ifUXNy1fRlqqH6DciPY1lwVXnxsAaYRyIrhH96Ltk0vvaUIPu4nfQfQNV 
    GrIP0bXiPKCgjfyhIfhKnimZLrRfwFJ5j6C84y64f1wi58rkIvrXOTff9OK8w339 
    DvrQRXcf94Di2bMWxxn9CxYR99g9+PtF9JsptNf7S/Sv7+viXBRauY/mQs9qLvdO 
    HbnT8F5wUpBj6zCNOvi7ox5vHf9VYSZIY0C2LeL7dMn7XvASzrvnvTOlM242KVbf 
    j49Qhoz8NUH+yy50K9m7x44VYEcdff4grLOklxsu8ar0deNs4CLa298NcN4Fn/Tc 
    5RgPWj8PueCvwr24P1TFncO9A8g4h7n5Oa7hIv6hOnrX6RPsuQf/mvfNuo7pr9B+ 
    tp3rjftZwc4231/fw3z3+xp0tznd10KbBLPFHPpUAeeqAbis2HOVBFs6tsDlynq2 
    XdGhgOOwRtjH05pH6+icbK2V/vND4LZe3pSnZ7r+I+pkrY+f4/p4ybUfnR0hgcav 
    /AR/90/YtPJGd1oA7YKN50O8ORnfmbpjZuHuQdx5eTZ4qY5ti0vst6WDbKOw+fxu 
    28zTPqWOXX6Ldvnbt7XtkkN7f1xnPzuVtvZ5t+3vVZzf7W/HdvxODOCeU2X73A9K 
    d8p9E+VyQ+BOeTCwFt4Uvknf5PgJbEsgzhoslY7a36YeOu6Bf5afb/X7rDURZeC3 
    2weTKHsyVGMv/JfR+/92s2nnRuufCSws+9+/9+mwRs3n1Bf7Dx7b/8JhiITZX6W9 
    O7IUf1hfWlFIj2l6xtD0UwDPnsyoare6O9IZUXfv7nki0tPZzc5MDRX/E/lsIZ3R 
    kj0EjOj5LJYA3+t8kvFgLDqfDCcQfyROeKqZV63jovTr2g6VFEXSLJ0lVk+PiBmg 
    7r2HBvpY6wA7KUrnUtSprUYPybDOgCxiOv3hbID3mUF7h/O6zf9ogY6O1J1bE6oZ 
    P6EZyEpT43rqZFbLmc0AL+VO5PKnc2qeMTXoiBteMuIpDaWpr+xMv6q+0j+IPzsL 
    2EplTu0/8GozpNVn1HROM5NqNp/ELvUPqnldHc0bZi6e1ai/1jnXDrWAmAVUSD2S 
    38Xa+g8wWlPTR+IJTWV1oaQ2ggK2NYvjcejYscFdneFOHIWIevS5lubePFLlzJ3m 
    eEHrUdNZVHNXKj3S0ryXWWWnHs+lNKNHHR43NaOKntFyKXO0R+16nNXlNHac1qOy 
    M7SW5pbmg/0Hup+KS/ji43dDGT83cV76l8yOgXdYEwI+ezx9EjwND9KD9CA9SA/S 
    /0UKwN17sbK1BFRKRypasc/okkqKHxYjrU0UJ2SxQcwUA6S4H8X6KL5HMT2K41Hs 
    juJ1FKOjuBzF4ij+RjE3irNRbI3iaSyGhiXFyig+RjExioNR7IviXTzGNnVH0FPl 
    senN1ZijazuPsbm2h1bWb7disJunFoX2eW+13Y4BuiU7JuqawsaoYepmfBjCbL9Q 
    gLCupVjELJwcx41DOgHh0bgxykBjPGuVpk74aRPCLGwVZjG/sJ5nIauw9atnkrFs 
    HBlqo7ERHbcgEE6Yed3Adqt4LYFcUnkkNyiuFWY/hSRWZpPa8MlUOD6cfnz3PfvZ 
    Ch5X9lTuhljxfHDcG1jH70MQnnVHxLp7AMJ9BR+/s9AI9n2PKLvzMS/cCZD5fYCt 
    /B6Ah9/ZWEC8Q1L1boB97+QxrqOH3+UI4e56zmPxkzgOvSiP87sSHn7347bXuvPh 
    7MfT/LmR+8HxZQBnHP2gtFfAozsAC4g3tVqI+fDyoIBHfkMXQ16uwe+wYOdbiHcL 
    8UY9d/N7ieM12NXq0onAfn5FwKM7CfQpLdeQmxTwcFyVVhe8nIDno6NktXpvQ8Q7 
    KeCRaqoL3hkBL4R4IRe5Z6t4Ct2NiCDelKO/hPsW95sG626RMq86Yzz8nK+KBzZe 
    Lfv92HGfgvAWauj3UwfetL+2XGHMKnhXauD9F+e6ouzcJQAA 
    ENDF
    }
    chmod +x $PXL
    fi
     
  53. ~nephelim~

    ~nephelim~ LI Guru Member

    I made some tests for lookahead update detection that rely on netcat (nc) and md5sum

    nc availability on tomato versions/variants is unclear.

    While testing repeatedly, rarely some host caused netcat test to hang (if GETS includes hosts 5 to 7 as well)

    I added a timeout (-w 10) option to netcat to attempt to prevent this.

    I will check if this occurs again meanwhile.

    Code:
    ## BEWARE this is NOT an ADBLOCK
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
    # !!! TODO: use  www.malwaredomains.com if it is confirmed 
    # that mirror1 sometimes do not send a last modified date
    S2="http://mirror1.malwaredomains.com/files/justdomains"      
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    
    if ! type nc || ! type md5sum ; then #test for commands
    echo nc/md5sum unavailable 
    exit
    fi
    
    GEN="/tmp/gen" 
    GETS="1 2 3 4" # Servers from 5 to 7 take a long time for unknown reasons
    
    CNTA=$(date +%s) #Performance counter
    for i in $GETS; do 
    eval url="\$S$i" 
    P1=$(echo $url| sed 's|^http[s]*://[^/]*\(/.*\)$|\1|') #get path. Do not handle user:password@host urls
    H1=$(echo $url| sed 's|^http[s]*://\([^/]*\)/.*$|\1|') #get host. Do not handle user:password@host urls
    
    
    
    
    	for x in 1 2 3; do #Address sporadic timoeouts.  Retry up to 3 times. 
    		time=$(echo -e "HEAD $P1 HTTP/1.1\r\nHost: $H1\r\nConnection: close\r\n"|
    		nc -w 5 $H1 80|grep -i Last-Modified:|tr -d "\r") #LookaHEAD (rarely no response is received after dozen of seconds)
    		if [ "$time" == "" ]; then #condition must be handled to avoid incorrect results out of sporadic timeouts
    			NEW=U #Undefined
    			logger ERROR $url #connection timeout or no lastmodified date 
    		else
    			echo $url
    			break;
    		fi
    	done  #inner loop
    
    echo $time>>$GEN.new
    done #outer loop
    
    if [ "$NEW" == "U" ]; then
    echo Undefined condition for some urls. see log
    #TODO: specific handler for this condition:
    #Alternatives:
    #1. fall back to $GEN.md5 method 
    #2. postpone adblock.sh execution + exit.
    #3. have adblock.sh continue thinking the list were updated (might not be the case since one or more list were unreachable)
    exit
    fi
    
    if [ -f $GEN.last ]; then  # if no $GEN.last start afresh
    	MD1=$(md5sum $GEN.last|cut -d " " -f1) 
    	MD2=$(md5sum $GEN.new|cut -d " " -f1) 
    	if [ "$MD1" == "$MD2" ]; then
    		rm -f $GEN.new
    		NEW=N # Same MD5
    	else
    		echo new $MD2 && cat $GEN.new && echo old  $MD1 && cat $GEN.last
    		mv -f $GEN.new $GEN.last
    		NEW=Y  # Different MD5 
    	fi
    else
    	NEW=Y # NO previous $GEN.last
    	mv -f $GEN.new $GEN.last
    fi
    
    CNTB=$(date +%s) #Performance counter
    echo "NEW=$NEW Runtime:  $(($CNTB-$CNTA)) sec"
    
    [ -f $GEN.new ] && echo leftover $GEN.new  && ls /tmp
    
     
  54. srouquette

    srouquette Network Guru Member

    It takes between 2 and 4 sec on my router.
    Do you run adblock more than once a day?
    mirror1.malwaredomains.com doesn't return all the time the latest update?
     
  55. QSxx

    QSxx LI Guru Member

    3.6b doesn't work on stock Tomato 1.28.1816 @ WRT54G v2. It simply won't start.

    I checked /tmp directory and found a part of ADBLOCK.sh there with only header part and ADB is never created.

    It seems that the problem lies in openssl
    Code:
    openssl:Error: 'enc' is an invalid command.
    
    Standard commands
    req            rsa            x509
    
    Any fix for that? :)

    (In the meantime i'm back to 3.5e)
     
  56. ~nephelim~

    ~nephelim~ LI Guru Member

    Not yet though I'm considering it.

    ATM only S5 list appear to be updated frequently during a day though shortening update intervals might provide a way to get other blacklists changes earlier.

    Incidentally S5 is also the list that caused more timeouts on repeated tests.

    During early tests mirror1.malwaredomains.com did not reply with a last modified field necessary to acknowledge that the list was updated without actually downloading the file.

    It looks like the lack of last modified it is not happening anymore at the moment.
     
  57. ~nephelim~

    ~nephelim~ LI Guru Member

    Openssl is needed for base64 decoding.

    On build 52 there is no uudecode command.


    Probably yes.

    It should be slightly slower but it would be possible to carry base64 decoding using awk.

    stripping the encode part of that awk script and removing excess whitespace from the awk function should be able to carry the decoding without increasing much the script size.
     
  58. srouquette

    srouquette Network Guru Member

    I searched a little bit and BusyBox should have uudecode, but TomatoUSB doesn't have it.
    I updated my sig to keep a link to 3.5e.
     
  59. mraneri

    mraneri LI Guru Member

    With all due respect, isn't this all a little overkill?
    How often do you guys really need to update your adblock lists? Why do you care if your router's CPU usage takes a hit once a week at 3AM?

    I get black lists. I get White lists. I even get combining two lists. (Though I don't bother.) This code has gotten so complex that it has to be gzipped to fit in 8k? Really?

    Is it all worth it? I'm still using the adblock script, which I guess started this all, from May 2007, with the whitelist upgrade in January 2009. It's fundamentally 10 lines of code. (Plus however many whitelisted sites you have.) And it's never failed. Even through all the firmware upgrades.

    I guess I'm gonna get flamed for this, so, let me have it...
     
  60. QSxx

    QSxx LI Guru Member

    No, you won't get flamed over it :tongue:

    Shortest answer would be - for the fun of it all :) pushing the code to the limit..

    Okay, okay, it is WAAAAAY too advanced for the "simple" adblock script right now but still...

    I, for example find pixelserv addition very welcome - no more 404pagenotfoundwhatever error where ads were supposed to be,

    It's just a couple of reeeeeaaallly talented coders --- having fun, tuning the engine ... (yes, i mean you ~nephelim~ :) )



    On the other hand, it is true that even basic script works JUUUST fine... - everyone can choose his own flavour :cool:
     
  61. srouquette

    srouquette Network Guru Member

    What QSxx, for the fun of it.

    mraneri, aren't you the one who was complaning about the lack of hosts version instead of dns poisoning?
    We added that, with a cleaning method to remove duplicates (nearly 10000 entries for dns poisoning).
    The script is larger than 4k, that's why we used the gzip method (down to 3k now).
    Feel free to use what you want, we don't force anyone to use it.
    But you popup every year to say the same thing... what's the point?
     
  62. ~nephelim~

    ~nephelim~ LI Guru Member


    It looks like chances for additions are over. :frown:

    openssl enc or uudecode are not amendable requirements to continue further feature addition.

    I tested that awk-based base64 conversion script I mentioned earlier and found it to be incompatible with busybox's awk:

    Since tomato's awk cannot output NULL (0x00) bytes the script cripples gzip compressed data.

    Found no workaround for this. :noway:

    I hope openssl enc support will not be removed in future noUSB builds as well. :frown:

    EDIT: found out a workaround to print null bytes but it needs a printf command.

    it's an hack based on awk's system command where busybox's printf is invoked without specifying a variable (proper way to call printf would be printf %c,variable )


    EDIT2. using cat with a file containg a null character works as well and busybox printf is not needed.


    Code:
    # base64 decoder by Danny Chouinard's
    
    if openssl enc 2>&1 | grep invalid ; then
    z="/tmp/zero"
    echo -ne "\0"> $z
    b64="b64"
    else
    b64="openssl enc -base64 -d"
    fi
    
    
    b64(){
    awk -vz=$z 'BEGIN{b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
    while(getline<"/dev/stdin"){for(i=1;i<=length($0);i++){c=index(b64,substr($0,i,1));
    if(c--){for(b=0;b<6;b++){o=o*2+int(c/32);c=(c*2)%64;if(++obc==8){if(o)
    {printf "%c",o}else{system("cat "z)}obc=0;o=0;}}}}}}'
    }
    
    
    {
    cat <<'ENDF'| $b64 |gzip -d >>$ADB
    H4sIACIhzEwCA6VWbW/aSBD+7l8x3fhi6NU2VO3dCepIeYEUlQDC5NJTaCvXXmBV
    sKltEnoh//1mZ9fgpKSKdApJltndeeblmZk9Hp77nnlg9DoDj9Uc+mHGeavnMTdf
    LN0pj5kxuhjorzlfLJlx2u+1tSBM4gkz3vf9ka8lsyTLM2ZctTvdlhbdzkTOmTH4
    2NWCpVjzecbTG2ZI6Re/NbocVKpwZ4gJXF8DMy/91qDzsdX1W8O/GXgesH8YfPoE
    TchnPNbHXoC9BhMVlHZupzwHu6/E8s+Xy2HXCGeLJILf1WljIlCBtFxM4Wtaa9TU
    weOz4Ra/shRRMoGtodW1smJdsqKEgVdRmM05X0IdV4TwfE3GPJlOeQrHZyfd/ukH
    aA2H/WEDwiCOkxyyPEjznQJD/jP4TTAHJtNWGMCkX/i5N4zT7pCimS7AnoCJ6SyW
    mMpiKbNYrClb8uZo2LmgqxmPwBYcLCNzxwfOS9N1cfW5/vrPsVOjT/0axvmnlyRX
    S32m0ShvzZMwmEtSyF33s+lGljID0c58wpokKQgQsbR05DchSpR7q3TusbHpm4LJ
    cFJqTRTK/NqwgTwFOwI2Thkckc6H9DAF2FOOG+jTl5PWeadXyp0UqkSFQa4uHx2p
    UD1MRoMgVdB/mabCOpmDKIk5hdMfVaoFq81jLDZ44UFN2rHZFPzI0JVpisyxMdy2
    LYlpFSIrTr4m0Q+rqkgjOaN93OXWWURvJejF2VvPrOCXbLWgjU24yilC+GNP6lVt
    R+maNOTwkIzD2xLCrFA89H6V8PZxtAFZsOCAt15B9k0slyKewmoZBTl3JP+awNci
    11wNZwkBHJXNxfB0r96PHpf91fvOqNXt+KM9ZS+JkmEvIa7QQYy0piow16woJHlk
    I6UWctdxx/g7tapuxFR+KTl8/jTmsFzkO+ZdYSfZTz0qn/213ADqfnORaQ2FuTLZ
    urJgW2KWEusqUVX5wGmVHdqo/g/fn8Plx3ar7kIozDzpHp9+YFAg6XofxwimZNqH
    oqaoKfUo1z/xKElzSR4aHI7jlMlAg2UPEeQdsFdgJ6p0CYPcCm6/gd0Gy7HAopq/
    6w9xNLH7O4xjRXi9dlMc1ZrCtqvLVMTonGAHrKnWbByze0vVjoTYbNUdPFLXjABt
    +g3ViklFxBFfV8zaq6j6wqtX7yLPrGmNpm0joqGwaf0QnTnMFGV0hFe9TAb7cago
    RBQwHjnG4qZo6bsY91Q9jfojbAW3IdhzeEe71Z/bGh4CHuep4Jm82z7r+RePq/Gp
    BITLoo2ASWf0iP3jzZtCUOJ5tvm8sUwcVtjXrD37dYFzhR4fddjOi+1Byqs+LVVh
    pWyCKEp5lnnuoasUb1TaisZy2j5/pitAA8DleehGcbYIsu+O7MDEXBqRcrv4Au/e
    tfptA9Fjm5Lh7d48htwhW59WuLOwP3jY+fB756Ljt/YZ+Bwn1HsA3fbIUFZ+JZAU
    oemFsHWnfa69iWbh0g5W+SxJRR7k4kZ6EM64nYl/ufe69uYvyR07yH7EofeW3NRR
    Pul+0I+F9EaEHLS/+GJJls+yurigpp49EXOu7Ffm793exbBNzweKT2mItoDpe6ws
    U4OUbaEfl0OBJTJIV3GMHelXfRLJJ19lsm8VScbOpZfavO3LVs5DfOvI1wCNPFmm
    sh3Sfyo7IqzkBMUUHaMrxn/iIe1CngsAAA==
    ENDF
    }
    rm -f $z
    
     
  63. mstombs

    mstombs Network Guru Member

    Hi Jochen,

    Do you find this test for image request avoids the issue with Internet Explorer displaying a script error when a website tries to load a script from a host blocked by adblock/ dnsmasq but redirected to pixelserv? If so all users of pixelserv.c could benefit - will experiment later!
    You must use additional IP address filters to block numeric IP address requests - these won't be be forwarded by dnsmasq dns poisoning scripts like those discussed here?
     
  64. mraneri

    mraneri LI Guru Member

  65. srouquette

    srouquette Network Guru Member

    @nephelim:
    hey, good job! :)
    I'll update the script tomorrow.

    @mraneri:
    Yes, I know you wrote the original one.
    We don't really ditch your work, but we took this script and made some improvement because it already had more features.

    You asked a few years ago why we used this one. Now I can answer you since we added more sources. This script can merge multiple sources and clean duplicated entries. It isn't a simple adblock anymore, it also block malware domains. MVPS isn't enough, MDL has an up to date malware domain list.

    Did you find an anwer about which one is more efficient between hosts file and dns poisoning?
    I'm interested. I asked that a few days ago, but noone knows...
    hosts file has 33k entries, while dns poisoning has 24k entries. I don't know if dnsmasq load the hosts file in memory, but it does with dns poisoning (takes up to 4k in memory).
     
  66. dkirk

    dkirk Network Guru Member

    PixelServ

    Been playing with the PixelServ posted a page or two back and still don't have it working right. Would any of you that have it working kindly post a few words of wisdom for us trying? What should the following values be set to:

    PXL_ADR=192.168.1.1
    PXL_URL="/tmp/pixelserv"
     
  67. srouquette

    srouquette Network Guru Member

    PXL_URL is the URL from where pixelserv will be downloaded. You have to store it on a server (or use nephelim's script to store it in nvram).
    And PXL_ADR is the address pixelserv will use to redirect blocked ads. 192.168.1.1 may be used by your router, you have to set an unused address.

    nephelim, I tried your script but it doesn't work. The first test with openssl hang up. I think it's because openssl open a prompt.
     
  68. mraneri

    mraneri LI Guru Member

    I presume you mean 4MB? At one point I figured a 14000 element host file consumed 1MB of RAM in my router. Never could detect any performance penalty, but I never did a DNS benchmark either. Not sure how you could benchmark it accurately and reliably. Maybe I'll check out the MDL list and see if I can see how much memory a consolidated list consumes the hosts way...

    (Is this what you're asking?)
     
  69. srouquette

    srouquette Network Guru Member

    Yes that's what I wanted to know, thank you.
    Also, which one is faster?
    I know that DNS poisoning blocks more stuff because it blocks domain instead of hosts (that's why we can remove more hosts with the DNS poisoning method if we block the domain), but we don't really know which one is faster.
     
  70. srouquette

    srouquette Network Guru Member

    Adblock 3.7b.
    Code:
    ## ALL-U-NEED AdBlocking By YAQUI 5/1/09
    ## updated by ~nephelim~, Syl, jochen, groosh, ng12345, ray123, mstombs
    ## base64 decoder by Danny Chouinard's
    ## Dnsmasq Ed. v3.7b
    
    sleep 10
    
    ## Auto Update?
    AUPD="Y"
    
    ADB="/tmp/ADBLOCK.sh"
    {
    cat <<'ENDF' >$ADB
    #!/bin/sh
    
    ## EDITABLE VARIABLES
    OPTIMISE="N"
    USEHOSTS="N"
    
    USEWHITELIST="Y" # N/Y/R for remote
    WURL="http://example.com/whitelist.txt"
    WHITE="intel.com"
    BLACK=""
    
    USEPIXELSERV="N"
    PXL_ADR=192.168.1.2
    PXL_URL="http://example.com/pixelserv"
    PXL_EXE="/tmp/pixelserv"
    
    GETS="1 2 3 4"
    TRIM_BEGIN=3
    S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
    S2="http://mirror1.malwaredomains.com/files/justdomains"   #189K
    S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"  #97K
    S4="http://www.mvps.org/winhelp2002/hosts.txt"             #620K
    S5="http://hosts-file.net/hphosts-partial.asp"             #283K
    S6="http://hostsfile.mine.nu/Hosts"                        #2641K
    S7="http://support.it-mate.co.uk/downloads/hosts.txt"      #3851K
    ENDF
    }
    
    AUP() {
    if [[ "$AUPD" == "Y" ]] ; then
    if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then
    ## cru (a)dd <name> "min hr day mo wkday <cmd>" 
    ## min=0-59 hour=0-23 day=1-31 month=1-12 sun=0 *=all
    cru a AdUpd "0 4 * * * $ADB"
    fi
    fi
    }
    
    #### DO NOT EDIT BELOW ####
    
    b64="openssl enc -base64 -d"
    [[ "$(echo WQ==|$b64)" != "Y" ]] && b64="b64" 
    
    b64(){
    awk 'BEGIN{b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
    while(getline<"/dev/stdin"){for(i=1;i<=length($0);i++){c=index(b64,substr($0,i,1));
    if(c--){for(b=0;b<6;b++){o=o*2+int(c/32);c=(c*2)%64;if(++obc==8){if(o)
    {printf"%c",o}else{system("echo -en \"\\0\"")}obc=0;o=0;}}}}}}'
    }
    
    {
    cat <<'ENDF'| $b64 |gunzip >>$ADB
    H4sIAJDbz0wCA6VWbW/aSBD+7l8x3fhi6NU2VO3dCepIeTEpKoEI00tPpa1ce4FV
    waa2SdIL+e83O7sGh5Iq0ilELDs7+8zLMzN7PDwPPPPA6HcvPdZw6I8Z537fY26x
    WLpTnjBjdHGpfxZ8sWTG6aDf0RtRmkyY8XYQjAK9M0vzImfGVafb8/XWzUwUnBnG
    5Yfel8Afvb+s1eHOEBP4+BGY+T7wL7sf/F7gD/9m4HnA/mHw6RO0oZjxRB97BvYt
    mPIC/4Nfkd5MeQH2YCuixfthz4hmizSG37daxkTgZdJgMYWvWaPVUKLjs+HGltpS
    xOkEluKWz3OeXdeVQRV7dpBQuQ35nPMlNPEEQTz1KmOeTqc8g+Ozk97g9B34w+Fg
    2IIoTJK0gLwIs2Krbsgvg1+Hc2AyWSU6k27h594wTntDCmy2AHsCJiaxXGICy6XM
    XbmmHEnN0bB7Qao5j8EWHCwjd8cHznPTdXH1ufnyz7HToE/zI4yLT89pXy31mVar
    KpqnUTiXVJBS97PpxpYyA9HOAsKapBkIEIm0dBS0IU6Ve6ts7rGxGZiCyWBShk3c
    lGm2YQ1FBnYMbJwxOKI7HzLFFGBPOQrQpy8n/nm3X0me3FRpisJCKR8dqVA9TEaL
    IFXQf5mm0jqZgzhNOIUzGNXqJcHNYywxeOZBQ9qxXpfsyNGVaYa8sTHcti15aZVb
    VpJ+TeMfVoUz2sdtbp1F/FqCXpy99swa/shXCxKso1VBEcI/e9KsazsqatKQw0My
    DrUlhFmjeGh5nfD2cbQFebjggFovIP8mlkuRTGG1jMOCO5J/beC3otBcjWYpARxV
    zcXw9K7ejnY7wNXb7sjvdYPRng4giZJjByGu0EGMtKYqMNesKSR5ZC13LeSu447x
    f2rV3Zip/FJy+PxxzGG1yrfMu8Jmsp96VD77a7kF1PPmItc3lObKZOvKgk2JWWpb
    V4mqygdOq+yQoP4/fH8Kl3ftVt2FUJh50js+fcegRNL1Pk4QTO1pH8qaoqbUp1z/
    xKM0KyR5aFw4jlMlA42TPUSQOmCvwE5V6RIGuRXefAO7A5ZjgUU1fzcY4kBi93cY
    x5rw+p22OGq0hW3Xl5lI0DnBDlhbrdk4YfeWqh0Jsd5cd7BzXTsGtOk3vFZMaiKJ
    +W3NbLyI68+8Zv0u9syGvtG0bUQ0FDatH6Izh5miio7wqpfJYO+GikJEAeOxYyyu
    y5a+jXFf1dNoMMJWcBOBPYc3JK3/3NbwEPCkyATPpW7nrB9c7FbjYwmIlmUbAZPO
    6Cn7x6tX5UaF5/n689oycVhhX7P2yJsC5wo9OZqwmRebg5RXfVpehZWyDuM443nu
    uYeuunit0lY2ltPO+RNdARoALi8iN07yRZh/d2QHJubSiJTi8ge8eeMPOgaiJzYl
    w9u+dAwpIVsfv3Br4eDyYefD392LbuDvM/ApTqj3ALrtkaGs+kqgXYSmF8LGnc65
    9iaeRUs7XBWzNBNFWIhr6UE043Yu/uXey8arvyR37DD/kUTea3JTR/mk904/FrJr
    EXHQ/uKLJV0+yepSQU09eyLmXNmvzN8r3sawQ88Hik9liPrAtB6r7qlByjbQu+VQ
    YokcslWSYEf6VZ9E8slXmexbZZKxc+mlNm/zyJXzEN868jVAI0+WqWyH9E1lR4SV
    nKCYomOkYvwHxgWvWZQLAAA=
    ENDF
    }
    
    chmod 775 $ADB
    $ADB
    AUP
    
    FYI, gzip block:
    Code:
    #!/bin/sh
    {
    cat <<'ENDF'|gzip|openssl enc -base64
    ARGS=$#
    NIP="0.0.0.0"
    GEN="/tmp/gen"
    TMP="/tmp/temp"
    CONF="/tmp/conf"
    HOSTS="/tmp/hosts"
    WFILE="/tmp/white"
    
    PXL_SETUP() {
    if [[ "$USEPIXELSERV" == "Y" ]] ; then
    if [[ ! -x $PXL_EXE ]] ; then
    wget -O $PXL_EXE $PXL_URL
    chmod +x $PXL_EXE
    fi
    ifconfig br0:0 $PXL_ADR
    if [[ "$(pidof pixelserv)" == "" ]] ; then $PXL_EXE $PXL_ADR; sleep 1 ; fi
    if [[ "$(pidof pixelserv)" == "" ]] ; then
    logger ADBLOCK ERROR: cannot start pixelserv
    else
    eval "NIP=$PXL_ADR"
    fi
    fi
    }
    
    CLR() {
    rm -f $GEN
    rm -f $TMP
    rm -f $CONF
    rm -f $WFILE
    }
    
    TRIM() {
    sed -ie '
    s/\#.*$//
    s/^127\.0\.0\.1[ \t]*//
    s/[ \t]*$//
    s/^::1[ \t]*//
    s/localhost$//
    /^$/d' $TMP
    }
    
    DS() {
    for i in $GETS; do
    eval url="\$S$i"
    if wget $url -O - | tr -d "\r" > $TMP ; then
    if [[ $i -ge $TRIM_BEGIN ]] ; then TRIM ; fi
    cat $TMP >> $GEN
    logger ADBLOCK: $url
    else
    logger ADBLOCK ERROR: cannot get $url
    fi
    done
    }
    
    TST(){
    if [[ $ARGS != 0 ]] || [[ "$(ps | grep -e '--conf' | grep 'nobody')" == "" ]]; then
    rm -f $GEN.md5
    fi
    MD5=$(md5sum $GEN|cut -d " " -f1)
    if [[ -f $GEN.md5 ]] && [[ $MD5 == $(cat $GEN.md5) ]];  then
    logger ADBLOCK: same MD5, skipping update.
    CLR; exit
    else
    echo $MD5>$GEN.md5
    fi
    }
    
    LWHT() {
    if [[ "$USEWHITELIST" == "Y" ]] ; then
    for site in $WHITE
    do
    sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN
    done
    elif [[ "$USEWHITELIST" == "R" ]] ; then 
    if wget $WURL -O - | tr -d "\r" > $WFILE ; then
    logger ADBLOCK: whitelist $WURL
    sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE
    for site in $(cat $WFILE)
    do
    sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN
    done
    else
    logger ADBLOCK ERROR: cannot get whitelist $WURL
    fi
    fi
    echo "$BLACK" |sed 's/[ \t]*/\n/g'|sed '/^$/d' >> $GEN
    }
    
    CLN() {
    logger ADBLOCK: sorting hosts...
    if [[ "$USEHOSTS" == "Y" ]] ; then
    sort -u -o $TMP $GEN
    else
    awk -F '.' 'BEGIN{ORS=""}{for(i=NF;i>0;i--)print $i"#";print "\n"}' $GEN|sort|
    awk -F '#' 'BEGIN{ORS="";d = "%"}{if(index($0,d)!=1){d=$0;print $--NF;
    for(i=--NF;i>0;i--)print "."$i;print "\n"}}' > $TMP
    fi
    logger ADBLOCK: hosts sorted.
    mv -f $TMP $GEN
    }
    
    CNT() {
    TOT=$(wc -l < $GEN)
    logger ADBLOCK: $TOT entries
    }
    
    FDNSM() {
    if [[ "$USEHOSTS" == "Y" ]] ; then
    cp -f $GEN $HOSTS
    chmod 644 $HOSTS
    sed -i -e 's|^|'$NIP' |' $HOSTS
    sed -i -e '1i127.0.0.1 localhost' $HOSTS
    else
    sed -i 's|^.*$|address=/&/'$NIP'|' $GEN
    fi
    }
    
    LCFG() {
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    cat /etc/dnsmasq.conf >> $CONF
    cat >> $CONF <<EOF
    addn-hosts=/tmp/hosts
    EOF
    else
    cat /etc/dnsmasq.conf >> $GEN
    fi
    }
    
    OPT() {
    if [[ "$OPTIMISE" == "Y" ]] ; then 
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    eval "CFG=$CONF"
    else
    eval "CFG=$GEN"
    fi
    cat >> $CFG <<EOF
    dhcp-authoritative
    cache-size=2048
    log-async=5
    EOF
    fi
    }
    
    LBLK() {
    service dnsmasq stop
    if [[ "$USEHOSTS" == "Y" ]] ; then 
    dnsmasq --conf-file=$CONF
    else
    dnsmasq --conf-file=$GEN
    fi
    }
    
    FS() { 
    if ps | grep -E "dnsmasq" | grep -E "nobody" ; then 
    logger ADBLOCK: dnsmasq is running
    else
    logger ADBLOCK ERROR: restarting dnsmasq...
    dnsmasq
    fi
    }
    
    PXL_SETUP
    CLR
    DS
    TST
    LWHT
    CNT
    CLN
    CNT
    FDNSM
    LCFG
    OPT
    LBLK
    FS
    CLR
    
    ENDF
    }
    
    edit (a): moved some options for pixelserv

    edit (b): updated base64 test.
     
  71. mrplow

    mrplow Addicted to LI Member

    I'm using build 52 of tomatousb mod and I was wondering how I would change the script to load the whitelist and pixelserv from my usb drive which is automounted to /tmp/mnt/linksys at boot. I've changed the webserver's port to 800 to free up port 80 for pixelserv. Its been said that tomatousb already changes some dnsmask stuff so optimize doesn't work right. So should the custom dns options be different if using tomatousb? I'm testing build 3.7 in a few minutes.
     
  72. mrplow

    mrplow Addicted to LI Member

    works great, I just had to change the DNS Cache Size to 2048 and it worked well, I'd still like to get pixelserv to work on the router though
     
  73. QSxx

    QSxx LI Guru Member

    Sure, as soon as i get home (in 4-5 hours time) i'll load it on both

    Tomato 1.28.1816 [WRT54G v2] RUNS
    Tomato RAF ND 1.28.8515 [WRT54G-TM] RUNS
    and report results here (btw, again - that was FAST)
     
  74. ~nephelim~

    ~nephelim~ LI Guru Member

    I had just given up but luckily there was a workaround.

    it might be possible to add uudecode support if

    Code:
    if type uudecode ; then
    echo YmFzZTY0IGRlY29kaW5nIHN1cHBvcnRlZA0K |uudecode
    fi
    
    is confirmed to work on some tomato version (assuming uudecode support pipes)

    the "type" command is a shell builtin and hopefully would be available on all tomato versions.
     
  75. ~nephelim~

    ~nephelim~ LI Guru Member

    tomatousb build 52 includes those optimize options automatically thus none of them can be used in custom dns options or dnsmasq will not start (will stop after triggering an error when it finds such options twice)
     
  76. srouquette

    srouquette Network Guru Member

    @mrplow: for the moment you have to download pixelserv from a server, but I'll try to do something about that.

    @nephelim: I don't have uudecode on TomatoUSB K24 NoUSB VPN build 52.

    edit:
    @mrplow: ok, you can try it now.
    I moved some option outside the zip if you want to customize them.
    PXL_PORT to specify the port, and PXL_EXE to specify when pixelserv is located. If pixelserv already exists (and is executable), it won't be downloaded.

    mmmh... after thinking, I don't think the port option will help, because the hosts file is formatted without a port.
    Is it possible to add a port in a host name?

    edit2:
    yep, I searched a little bit and the port option is useless here, so I removed it.
     
  77. ~nephelim~

    ~nephelim~ LI Guru Member

    Then only vanilla tomato and Victek builds are left.

    @QSxx please confirm uudecode availability on RAF and vanilla tomato.

    edit: maybe it would be safer to add md5sum and nc as well.


    Code:
    nvram get os_version
    
    if type uudecode  >> /dev/null 2>&1 ; then
    echo YmFzZTY0IGRlY29kaW5nIHN1cHBvcnRlZA0K |uudecode
    else
    echo uudecode not available
    fi
    if type md5sum >> /dev/null 2>&1 ; then
    echo md5sum available
    else
    echo md5sum not available
    fi
    if type nc >> /dev/null 2>&1 ; then
    echo nc available
    else
    echo nc not available
    fi
    if [[ "$(echo|openssl enc -base64)" == "Cg==" ]]; then
    echo openssl encoding supported
    else
    echo openssl encoding unsupported
    fi
    
     
  78. mrplow

    mrplow Addicted to LI Member

    Code:
    Nov  2 07:15:20 tomato-router daemon.info pixerlserv[857]: /tmp/mnt/linksys/pixelserv V16 compiled: Jul  8 2010 22:50:18 from pixelserv16.c
    Nov  2 07:15:20 tomato-router daemon.notice pixerlserv[859]: Listening on 192.168.1.1:80
    Nov  2 07:15:23 tomato-router user.notice root: ADBLOCK: http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Nov  2 07:15:24 tomato-router user.notice root: ADBLOCK: http://mirror1.malwaredomains.com/files/justdomains
    Nov  2 07:15:25 tomato-router user.notice root: ADBLOCK: http://www.malwaredomainlist.com/hostslist/hosts.txt
    Nov  2 07:15:35 tomato-router user.notice root: ADBLOCK: http://www.mvps.org/winhelp2002/hosts.txt
    Nov  2 07:15:36 tomato-router user.notice root: ADBLOCK: 32704 entries
    Nov  2 07:15:36 tomato-router user.notice root: ADBLOCK: sorting hosts...
    Nov  2 07:17:21 tomato-router user.notice root: ADBLOCK: hosts sorted.
    Nov  2 07:17:21 tomato-router user.notice root: ADBLOCK: 24497 entries
    Nov  2 07:17:22 tomato-router daemon.info dnsmasq[538]: exiting on receipt of SIGTERM
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: started, version 2.55 cachesize 2048
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-I18N DHCP no-scripts TFTP
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: asynchronous logging enabled, queue limit is 5 messages
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq-dhcp[942]: DHCP, IP range 192.168.1.100 -- 192.168.1.149, lease time 1d
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: reading /etc/resolv.dnsmasq
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: using nameserver 64.59.161.68#53
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: using nameserver 64.59.160.15#53
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: using nameserver 64.59.160.13#53
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: read /etc/hosts - 2 addresses
    Nov  2 07:17:23 tomato-router daemon.info dnsmasq[942]: read /etc/hosts.dnsmasq - 5 addresses
    Nov  2 07:17:23 tomato-router user.notice root: ADBLOCK: dnsmasq is running
    dnsmasq is segfaulting now, although it does appear to be working fine and serving pixelserv for ads
     
  79. ~nephelim~

    ~nephelim~ LI Guru Member

    Found out a way to simplify the Null byte workaround to use only a properly escaped echo command.


    Code:
    b64="openssl enc -base64 -d"
    [[ "$(echo WQ==|$b64)" != "Y" ]] && b64="b64" 
    
    b64(){
    awk 'BEGIN{b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
    while(getline<"/dev/stdin"){for(i=1;i<=length($0);i++){c=index(b64,substr($0,i,1));
    if(c--){for(b=0;b<6;b++){o=o*2+int(c/32);c=(c*2)%64;if(++obc==8){if(o)
    {printf"%c",o}else{system("echo -en \"\\0\"")}obc=0;o=0;}}}}}}'
    }
    
     
  80. QSxx

    QSxx LI Guru Member

    3.7a runs on Tomato vanilla and RAF ND 8515 :) :)

    uudecode command fails when run from shell on both vanilla and RAF :/

    I'll report any anomalies - so far so good...
     
  81. srouquette

    srouquette Network Guru Member

    @nephelim: ok, I updated it in 3.7b :)

    @QSxx: that's good to hear ^^
     
  82. mrplow

    mrplow Addicted to LI Member

    Code:
    Dec 31 16:00:55 tomato-router user.warn kernel: EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
    Dec 31 16:00:55 tomato-router user.info hotplug[511]: USB ext2 fs at /dev/sda1 mounted on /tmp/mnt/linksys
    Dec 31 16:00:55 tomato-router daemon.info dnsmasq[489]: exiting on receipt of SIGTERM
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: started, version 2.55 cachesize 2048
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-I18N DHCP no-scripts TFTP
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: asynchronous logging enabled, queue limit is 5 messages
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq-dhcp[538]: DHCP, IP range 192.168.1.100 -- 192.168.1.149, lease time 1d
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: reading /etc/resolv.dnsmasq
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: using nameserver 64.59.161.68#53
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: using nameserver 64.59.160.15#53
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: using nameserver 64.59.160.13#53
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: read /etc/hosts - 2 addresses
    Dec 31 16:00:56 tomato-router daemon.info dnsmasq[538]: read /etc/hosts.dnsmasq - 5 addresses
    Nov  2 12:18:01 tomato-router user.info syslog: SNet version started
    Nov  2 12:18:01 tomato-router daemon.notice miniupnpd[717]: HTTP listening on port 28407
    Nov  2 12:18:01 tomato-router daemon.notice miniupnpd[717]: Listening for NAT-PMP traffic on port 5351
    Nov  2 12:18:02 tomato-router cron.err crond[494]: time disparity of 21478757 minutes detected
    Nov  2 12:18:11 tomato-router daemon.info pixerlserv[808]: /tmp/mnt/linksys/pixelserv V16 compiled: Jul  8 2010 22:50:18 from pixelserv16.c
    Nov  2 12:18:11 tomato-router daemon.notice pixerlserv[810]: Listening on 192.168.1.1:80
    Nov  2 12:18:13 tomato-router user.notice root: ADBLOCK: http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml
    Nov  2 12:18:15 tomato-router user.notice root: ADBLOCK: http://mirror1.malwaredomains.com/files/justdomains
    Nov  2 12:18:17 tomato-router user.notice root: ADBLOCK: http://www.malwaredomainlist.com/hostslist/hosts.txt
    Nov  2 12:18:25 tomato-router user.notice root: ADBLOCK: http://www.mvps.org/winhelp2002/hosts.txt
    Nov  2 12:18:26 tomato-router user.notice root: ADBLOCK: 32588 entries
    Nov  2 12:18:26 tomato-router user.notice root: ADBLOCK: sorting hosts...
    Nov  2 12:20:15 tomato-router user.notice root: ADBLOCK: hosts sorted.
    Nov  2 12:20:16 tomato-router user.notice root: ADBLOCK: 24414 entries
    Nov  2 12:20:17 tomato-router daemon.info dnsmasq[538]: exiting on receipt of SIGTERM
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: started, version 2.55 cachesize 2048
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: compile time options: IPv6 GNU-getopt no-RTC no-DBus no-I18N DHCP no-scripts TFTP
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: asynchronous logging enabled, queue limit is 5 messages
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq-dhcp[888]: DHCP, IP range 192.168.1.100 -- 192.168.1.149, lease time 1d
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: reading /etc/resolv.dnsmasq
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: using nameserver 64.59.161.68#53
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: using nameserver 64.59.160.15#53
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: using nameserver 64.59.160.13#53
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: read /etc/hosts - 2 addresses
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: read /etc/hosts.dnsmasq - 5 addresses
    Nov  2 12:20:18 tomato-router user.notice root: ADBLOCK: dnsmasq is running
    Testing 3.7b on tomatousb-52-VPN-MIPS1-extra on wl-500gp well now I don't see pixelserv in the logs at all, its probably that first segfault. dnsmasq still segfaults.

    **EDIT**
    Ok well I forgot to enable pixelserv with the new version, now pixelserv loads but the the SIGTERMs from dnsmasq are still there.

    just thought I'd add running this script
    Code:
    nvram get os_version
    
    if type uudecode  >> /dev/null 2>&1 ; then
    echo YmFzZTY0IGRlY29kaW5nIHN1cHBvcnRlZA0K |uudecode
    else
    echo uudecode not available
    fi
    if type md5sum >> /dev/null 2>&1 ; then
    echo md5sum available
    else
    echo md5sum not available
    fi
    if type nc >> /dev/null 2>&1 ; then
    echo nc available
    else
    echo nc not available
    fi
    if [[ "$(echo|openssl enc -base64)" == "Cg==" ]]; then
    echo openssl encoding supported
    else
    echo openssl encoding unsupported
    fi
    Outputs:
    1.28.9052 MIPSR1-beta23 K26 USB vpn3.6
    uudecode not available
    md5sum available
    nc available
    openssl encoding supported
     
  83. ~nephelim~

    ~nephelim~ LI Guru Member

    Those sigterm are fine (they are not segfaults). dnsmasq is stopped under normal script operation and also while tomato itself carry other tasks (eg WAN reconnect).

    Thanks for providing the output of that diagnostic script.
     
  84. mrplow

    mrplow Addicted to LI Member

    thanks, I just confused sigterms with segfaults, it was working as it should!
     
  85. srouquette

    srouquette Network Guru Member

    Pixelserv, from your log:
    Code:
    Nov  2 12:18:11 tomato-router daemon.info pixerlserv[808]: /tmp/mnt/linksys/pixelserv V16 compiled: Jul  8 2010 22:50:18 from pixelserv16.c
    
    Sigterm is when I call "service dnsmasq stop" to restart it with the new config file.
    From your log:
    Code:
    Nov  2 12:20:17 tomato-router daemon.info dnsmasq[538]: exiting on receipt of SIGTERM
    Nov  2 12:20:18 tomato-router daemon.info dnsmasq[888]: started, version 2.55 cachesize 2048
    
    :)
     
  86. QSxx

    QSxx LI Guru Member

    3.7b still works :), even with pixelserv17 (my Star Trek Online launcher (IE dependent) doesn't complain any more)
     
  87. Toink

    Toink Network Guru Member

    I want to try this but I can't seem to extract pixelserv17.zip from mstombs link.

    "Cannot open file it does not appear to be a valid archive"
     
  88. srouquette

    srouquette Network Guru Member

    Try to download it again, it works.
     
  89. Toink

    Toink Network Guru Member

    @srouqoutte

    Thanks! I used winrar to exract it instead :) ... Now would you (or anyone) please be kind enough to point me to the tutorial on how to install the pixelserv to my router? I was hoping there was a tutorial in the compressed file.. I tried going to http://www.linksysinfo.org/forums/showthread.php?t=62758 but I can't seem to find the post on how to do it :(
     
  90. srouquette

    srouquette Network Guru Member

    Do you have a USB drive connected to your router?
    If yes, you can specify the path to pixelserv with the variable PXL_EXE in the script.

    If no, you have to put pixelserv on an external server, adblock will download it from the url PXL_URL.
     
  91. damwill

    damwill Network Guru Member

    Can it be stored in JFFS?

     
  92. Toink

    Toink Network Guru Member

    Aha! All the while I thought I should telnet to my router to install :wacko: Ok will try to store in my Synology... Thank you!:)
     
  93. srouquette

    srouquette Network Guru Member

    I think so.
    If mrplow can use it on his usb drive, it should be possible on JFFS :)
     
  94. dkirk

    dkirk Network Guru Member

    If you have pixelserv running localling from the USB or JFFS drive then the URL entry is ignored, is that correct?

    The IP address entry must match up with the one displayed by pixelserv (192.168.1.2) when it starts, correct?
     
  95. srouquette

    srouquette Network Guru Member

    Yes, the script download the file only if PXL_EXE doesn't exist or isn't an executable.

    If you type "ps" in your shell, you should see pixelserv running with the IP specified in the script, yes. The script runs pixelserv with this IP, and this IP is also replaced in the hosts file.

    So... If the file doesn't exists, it download it and try to run it. If it started, then the IP is replaced in the hosts file, if it isn't, the IP in the hosts file is 0.0.0.0 (fastest 404 error).
     
  96. dkirk

    dkirk Network Guru Member

    I see that in PS, so I know it's running, and it responds to the browser. I don't see in the code where the "ip is also replaced in the hosts file", just out of curiosity, I really want to understand this script.

    I would like to convey my thanks for you folks keeping this thread alive and producing some incredibly elegant code. I appreciate the opportunity to learn from some very talented coders, again, thank you!
     
  97. srouquette

    srouquette Network Guru Member

    Search every occurence of NIP.
    First, we format every hosts file by removing the IP, to keep only the hosts/domains.
    If pixelserv is running, we replace NIP (0.0.0.0) by PXL_ADR.
    Then we format the file if it's for the hosts method (0.0.0.0 example.com) or dns poisoning (address=/example.com/0.0.0.0).
     
  98. dkirk

    dkirk Network Guru Member

    eval "NIP=$PXL_ADR" ... I was missing that part up towards the top ... again, this code is truly a beautiful piece of work!
     
  99. Toink

    Toink Network Guru Member

    Pixelserv wont work on my WRT610N.

    I placed the file pixelserv in my Kingston USB. I enabled USB on my WRT610N. It shows in the router as:

    Code:
    Partition 'KINGSTON' vfat (1,906.91 MB) is mounted on /tmp/mnt/KINGSTON
    I then change the adblock script to:

    Code:
    USEPIXELSERV="Y"
    PXL_ADR=192.168.1.2
    PXL_URL="http://example.com/pixelserv"
    PXL_EXE="/tmp/mnt/KINGSTON/Pixelserv"
    
    Rebooted the router and checked the logs. I keep getting this:

    Code:
    user.notice root: ADBLOCK ERROR: cannot start pixelserv
    I wonder what I'm doing wrong....
     
  100. damwill

    damwill Network Guru Member

    I missed a step... How can I run pixelserv on jffs?
     

Share This Page