1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HTTP SPAM Blocking - How To

Discussion in 'Tomato Firmware' started by mikester, Apr 18, 2007.

?

Was this useful?

Poll closed Apr 22, 2007.
  1. Yes

    8 vote(s)
    57.1%
  2. No

    3 vote(s)
    21.4%
  3. Maybe

    3 vote(s)
    21.4%
  1. mikester

    mikester Network Guru Member

    Do you hate wasting bandwidth and time downloading pop-ups, pop-unders, video ads, and irritating mouseovers? I do and have a pretty reliable way for Tomato to drop all HTTP requests to the spam ads.

    The Benefits:
    * the following setup kills about 99% of the unwanted spam on my network
    * web surfing is now a LOT faster because it blocks all outgoing requests for spam as opposed to blocking incomming data that has already been transfered to your computer
    * no need for you to sell your surfing habits to 3rd party proxies
    * no need to install Squid/Proxy software on your Tomato
    * blocks Torrents! If you want to allow torrents then delete the first two lines in List #1 containing "announce" and "tracker"

    It's a work in progress. If there is interest I can periodically post my updates. Please forward any suggestions and inclusions.

    Enjoy!

    HOW TO
    -----------------------------------
    * Under http://[Your Tomato IP]/restrict.asp
    * Create a new rule - I use names "Keyword Blocking X"
    * Use the settings in the attached thumbnail
    * Using the "All Accept" rule will allow you to bypass the filter by MAC address
    * The lists have a maximum length - right now I have two lists/rules

    Rule List #1
    -----------
    .com.com
    2o7
    2o7.net
    a.answers.com
    adbrite.com
    adcentriconline.com
    adclick.php
    addynamix
    ad-flow
    adhost.com
    adhostingsolutions.com
    adknowledge.com
    adlog.com
    adonspot
    adrants.com
    adrevolver.com
    ads.adsonar.com
    adstream_mjx.ads
    adtech.de
    adventnet.com
    advertising.com
    announce
    anton.lr2.com
    anytimeproducts.com
    aspalliance.com
    atdmt
    auctionads.com
    axilsearch
    badges.del.icio.us
    banner.com
    betamarker.com
    bigboobpass.com
    bigmouthmedia.com
    boubleclick
    casalemedia
    click_lx.ads
    clickability
    clicksor
    cliksor
    clobo
    cm2.zonelabs.com
    csmonitor.com
    dada.it
    digitalmedianet.com
    domainsponsor.com
    doubleclick.com
    doubleclick.net
    downloads.walmart.com
    dw.com
    elsitio.com
    entertainsite
    euroclick.net
    fastclick
    filewire
    firstadsolution
    fuc* <----the "F" word
    gamespot
    gamespy
    google-analytics.com
    googlesyndication
    guarduptodate
    hbmediapro
    hitbox.com
    hotpocketsdojo.com
    hs2.zonelabs.com
    hypertor
    i.i.com
    iframeurl
    image.com
    intellitxt.com
    jsmaster
    kontera.com
    layer-ads.de
    llnw.net
    mcafee.com
    mdcanada.ca
    media.com
    mediaplex.com
    media-servers
    meegos
    musicgiants.gbis.com
    mygamercard
    napster.com
    netregistry.net
    netwealthpartner.com
    offermatica.com
    onlinesecurityhelp
    opticaljungle.com
    oversee.net
    paypopup
    phl-te.tacoda.net
    pointroll.com
    popunder
    precisionclick
    promo.tigeronline.com
    questionmarket
    realcastmedia
    realmedia
    revenue.net
    revsci.net
    screensavers.com
    servedby
    serving-sys.com
    specificclick
    sportsinteraction
    spyfalcon
    spylog
    statcounter
    tagworld
    tdnetworks
    tennpac.com
    toolbar
    tracker
    traff4ppc
    trafficmonkey.com
    trafficmp
    tribalfusion
    unicast.com
    usenext.de
    userbars
    valueclick
    video.msn.com
    x2Fnoactivex.html
    xiti.com
    yieldmanager
    yournetaccess
    youronlinesecurity
    ypn-js.overture.com

    List 2 - last updated 2008-08-11
    --------
    lakequincy.com
    orcsweb.com
    cpaclicks.com
    information.com
    qarchive.org
    webtrendslive.com
    hosting.de
    blogads.com
    arcweb.com
    allmusic.com
    zdmcirc.com
    op-0ut.get2.us
    show_ads.js
    inMail24.com
    rlink.org
    wholeprofits.com
    itrack.it
    passion.com
    ads.monster.com
    advertserve.com
    bluestreak.com
    ecnext.com
    .search.com
    pochta.ru
    digits.com
    nm.ru
    webtrenslive.com
    alternateurl.com
    madskills.com
    srad.js
    afy11.net
    text-link-ads.com
    snap.com
    gmpg.org
    adclick.php
    cpxinteractive.com
    member.dnsstuff.com/js
    adbureau.net
    graphics.dnsstuff.com/images
    member.dnsstuff.com/amember
    x-ratedclips.com
    porn.com
    player.php
    mpegs.com
    nudity.com
    adengage.com
    eprize.net
    101com.com
    domainroundtable.com
    auctions.domaintools.com
    ustream.tv
    earthwebhardware.com
    click_lx.ads
    hail.com
    adultfriendfinder.com
    livejasmin.com
    myspaceJS037.js
    myspacetv.com
    experts-exchange.com
    hitslink.com
    arrivenet.com
    2o7.net
    tfges.cn
    webtrendslive.com
    adsenseOpt.js
    subscription.js
    64.46.39.165
    google_ads
    valencemedia.com
    collegehumor.com
    clickhype.com
    zango.com
    zangocash.com
    licenseacquisition.org
    clearspring.com
    adultfriendfinder.com
    streamray.com
    alt.com
    bondage.com
    utarget.co.uk
    atwola.com
    hosting-24.cn
    .cn
    porn
    secureserver.net
    208.109.167.144
    popunder.htm


    A good list of black listed websites can be retrieved from mvps.org -> serach under host file blocking. Here is a link (700kb text file)

    mvps.org/winhelp2002/hosts.txt

    An easier solution is to use an opendns server
     

    Attached Files:

  2. roadkill

    roadkill Super Moderator Staff Member Member

    Thanks
    I think it would be a cool feature to add..
    like AdBlock in Firefox
     
  3. Hypernova

    Hypernova LI Guru Member

    You have a few repeats in that list especially doubleclick and a few others. I think follow the examples giving in the F/W and then use the list provided in AdBlock is a better solution.
     
  4. StevenG

    StevenG LI Guru Member

    I gave this a shot tonight, and couldn't get it to work. I tried blocking just doubleclick as a keyword, but it still loaded it. Hmmm...

    One thing I liked with the actiontec router that came with my Fios install was that you could set up include lists, not just exclude. When it came to locking down my kids PC, it was much easier to say which sites I wanted to let them have access to vs. not have access to.
     
  5. GeeTek

    GeeTek Guest

    You have obviously never tried to block double click. It is impossible to have too many variations and entries to block them. They always seem to find a way around the filters. Now that google owns them you need to triple the double click entries in every single malware filter that you can.
     
  6. Hypernova

    Hypernova LI Guru Member

    doubleclick. <-All I used to block them and it works.
     
  7. GeeTek

    GeeTek Guest

    Thanks so much for the clarification. I'll drop all of my duplictes immediately.
     
  8. mikester

    mikester Network Guru Member

    Duplicates are easier to find especially when they are printed on a full screen editor!

    About the doubleclick ads comming through, I've found that a lot of ads come from javascripts through includes. You really need to check out the web page source code to find the bugger loading the stuff up.

    Watch out that some spam sites use keywords used in normal web programming.
    i.e. if you block the word "banner"
    http://xys/bannerfourums/newreply.php?do=click4
    would get the web page blocked even though no banner.com web content was being loaded.

    Amazon and Walmart are big offenders for spam ads but if you like to look for price comparisons online then a LOT of web pages suddenly disappear! ;-)

    Post the offending web page links and I'll have a look. My list is by no means all encompassing, just blocks the websites I hit.

    I cleaned up and sorted my list in the original post and saved the changes. Now fits on one list! Keep in mind there's a maximum number of characters allowed in the list.
     
  9. Int15

    Int15 Network Guru Member

    Good job, thanks!
    However, why are you blocking aspalliance.com?

    Thanks again,
    Int15
     
  10. mikester

    mikester Network Guru Member

    I use the COB rule ;-)

    Basically block sites/content if they have "popped up" somehow/someway that is irrelevant to my "internet experience".

    I find aspalliance serves/posts too much ad&spam content on other scripting/programming websites.

    Change it to "ads.aspalliance.com" and it will block some of the spam content but still show you the basic web site.
     
  11. Int15

    Int15 Network Guru Member

    Great, thanks again!

    -Int15
     
  12. lwf-

    lwf- Network Guru Member

  13. yaqui

    yaqui LI Guru Member

  14. mikester

    mikester Network Guru Member

    All good links but I think I prefer the "small is beautiful" philosophy of Tomato. Personally I don't want to install and maintain spam buster software on every computer on the network...just call me lazy
     
  15. larsrya8

    larsrya8 LI Guru Member

    Adblock Plus for Firefox maintains itself.
     
  16. yaqui

    yaqui LI Guru Member

  17. paped

    paped LI Guru Member

    The list seems to work great.... thanks very much....
     
  18. yaqui

    yaqui LI Guru Member

    How much load

    Has anyone studied how much load using lists like this puts on the router?

    Wondering if it is better to do it on a 'per machine' basis.
     
  19. mikester

    mikester Network Guru Member

    Here's some stats for you:

    WRT54GL running Tomato v.1.05.0977
    CPU Load (1 / 5 / 15 mins) 3.91 / 4.49 / 4.29 (154 connections, mostly web surfing)
    Total / Free Memory 14.20 MB / 1,432.00 KB (9.85%)
    Uptime 16 days, 22:38:50
     
  20. mikester

    mikester Network Guru Member

    Just for fun I chose a website I like to view but HATE the spam content.

    I ran a test using IE6. Before each test I went to "Tools" - "Internet Options", "deleted cookies", "delete files + all offline content", "clear history". I didn't bother timing the differences as I think the results are self explainatory.

    URL: www.entrepeneur.com

    Data Transfer Stats:
    Tomato Spam Filter OFF
    UL: 84014 bytes
    DL: 777609 bytes

    Tomato Spam Filter ON
    UL: 49281 bytes
    DL: 442139 bytes

    Thats roughly a 44% reduction in data transfered. It's like getting an extra 44% of bandwidth!

    Hey Yaqui, how about you show some comparison stats along with some time comparisons?

    Flame away!
     
  21. larsrya8

    larsrya8 LI Guru Member

    Those load numbers are really high... I thought it wasn't supposed to go over 1.00?
     
  22. yaqui

    yaqui LI Guru Member

    Was this just using one machine and having 154 simultaneous connections going?

    I would like to know what happens with you have multiple client machines (like more than 10) doing many http requests.

    I think the load distributed with each machine doing the filtering would be better, rather than the router doing all the work for all 10 machines.

    Maybe this is wrong thinking, because the router sees all the requests the same as it would with just one client machine?
     
  23. mikester

    mikester Network Guru Member

    1 wireless and 2 wired machines were connected at the time.

    I don't think I've ever seen load numbers below 1.00. Can you post yours for comparison?
     
  24. yaqui

    yaqui LI Guru Member

    How do I isolate just measuring the filtering process? The load is taking into account all the routing functions too... isn't it??

    I'm seeing higher numbers with each additional machine I connect.
    I don't know if there is anyway to just isolate the measurement like I said.... maybe there is some way with the 'top' command?
     
  25. larsrya8

    larsrya8 LI Guru Member

    Uptime 11 days, 21:51:08
    CPU Load (1 / 5 / 15 mins) 0.00 / 0.01 / 0.00
    Total / Free Memory 14.20/3,320.00 (22.83%)

    This is with Conntrack reporting ~370 connections. Six computers connected (5 wired, 1 wireless). Bittorrent running on at least one.

    I have QOS, but I'm only using ports to classify.. no L7. Also using WPA on the wireless.
     
  26. mikester

    mikester Network Guru Member

    Weird...my cpu load is pretty consistent with or without the keyword blocking running. Maybe because I'm also running a WDS link, QOS, cifs file shares, remote syslog and SNMP?

    My other router load is
    CPU Load (1 / 5 / 15 mins) 0.12 / 0.03 / 0.02
     
  27. tunasashimi

    tunasashimi LI Guru Member

    If you're not seeing lag with those load averages, then..... you must be using dialup :D

    Does anybody have some latency measurements, or does tomato have some really cool hardcoded nice values for processes?
     
  28. snwbdr

    snwbdr Network Guru Member

    You had mcafee.com up there. I use it because it is free with comcast. I'm cheap. Anyway it blocks updates to the antivirus and firewall. I should have read the list first, but how is mcafee spam? Anyway I just deleted it from the list everything else works good, but not sure what everything on there is. Anyone try adblock plus's list? Does it work well or not? thanks
     
  29. yaqui

    yaqui LI Guru Member

  30. dangdonkey

    dangdonkey Network Guru Member

    To your network yes but it still makes it to the router where it's dropped.
     
  31. larsrya8

    larsrya8 LI Guru Member

    Are you sure it doesn't block the outgoing connections to those sites? You may be confusing this with incoming QOS.
     
  32. jochen

    jochen LI Guru Member

    I'm looking for a similar solution for parental control for the kids. Any ideas how to do this? I think it is no so easy to do this based on URLs.
    My understanding of tomatos http blocking is, that it is "blacklist" based (all URLs matching the keywords are blocked). I would prefer a "whitelist" based solution.
     
  33. yaqui

    yaqui LI Guru Member


    You can try FoxFilter

    It will filter keywords in the url.

    Edit: Sorry, I see that you want a whitelist. Maybe there can be some way to use wildcards in access restriction to block all sites... then set up another access restriction with a listing of only they sites you want to allow. (something like: ^*.*.*$ or simply ^$ for all sites?)

    Or, there may be a way in FoxFilter to use wildcards since that will allow URL Exceptions, I'm not sure yet if\how you can wildcard in that.

    I don't see "allow URL exceptions" yet in Tomato. Maybe the author(s) can add these features?
     
  34. Hi,
    I am using your HTTP SPAM Blocking Rule List #1, which is great. Thank you.
    You say in your post that right now you use two lists/rules. Could you please send or post the other one for me.
    Thanks
     
  35. wycf

    wycf Network Guru Member

  36. mikester

    mikester Network Guru Member

    try using

    ads.php
    edge.quantserve.com
    quant.js
    ad_func.js

    in your list

    Right now only one list is needed - I eliminated duplicates
     
  37. Talon88

    Talon88 LI Guru Member

  38. wycf

    wycf Network Guru Member


    You are kidding, right?
     
  39. wycf

    wycf Network Guru Member

    ok, that works, partially.

    how can I block those gif images under
    http://pub.creaders.net/html/site_ad_images/
    they are annoying.
     
  40. Talon88

    Talon88 LI Guru Member

    :::

    Oh, I made a mistake. I miss read you want to
    block that site....

    :::


     
  41. wycf

    wycf Network Guru Member

    The FAQ says:
    "Some limitations: Hostname is a separate string from path?query (path and query are considered as one string), so you can't use "domain.com/path". Others, like the POST data, or the content of the requested pages are not checked. Escaped characters are not decoded."

    I am not understand this very clear.

    So if I want block juat part of a website, for example, all the pages under http://somedomain.com/ads/, what should I do?
     
  42. yaqui

    yaqui LI Guru Member

    If you are blocking the advertising company's domain name (ie. doubleclick)... then all ads/web spam associated with that domain will be blocked regardless of any sub-path/url you may be visiting.
     

Share This Page