Adblocking megalist = dnsmasq crash

Discussion in 'Tomato Firmware' started by Bunsen, Mar 10, 2019.

  1. Bunsen

    Bunsen Reformed Router Member

    I've been trying to get my router [R7000 running FreshTomato] to block as many ads and malicious sites as possible. Sounds reasonable, right?

    I modified the default adblocking script to get it's "list of lists" from https://v.firebog.net/hosts/lists.php?type=all, but it seems like the result is just too much for dnsmasq to handle.. at least with the default settings.

    How can I change the dsnmasq settings to allow it to use more memory, or is it really at the limit of what the router can handle? Any other creative ideas out there, or do I really to exclude some?
     
  2. Mr9v9

    Mr9v9 Serious Server Member

    Aside from increasing the cache size there is not much you can do if you run out of resources.
    You could turn on logging and see what is happening with the queries, maybe try some other blocklists.

    I tested various configs and determined for my setup a VM running Pi-Hole and uBlock in a browser both handles and far exceeds what my R7000 could ever do. If you go this route you need to make sure that you don't use the router to handle the queries anymore.
     
  3. Bunsen

    Bunsen Reformed Router Member

    Thanks for the reply.
    I'll probably play with Pi-Hole at some point soon.

    By the way - do you know if it's more efficient to get the block list to dnsmasq by a conf file using the "address=/<domain>/0.0.0.0" syntax VS. a hosts file with "0.0.0.0 <domain>" format?

    Or is it exactly the same since it gets loaded as a hash value either way?
     
  4. Mr9v9

    Mr9v9 Serious Server Member

    I'm pretty sure it does not make a difference, and it is less messy in the hosts file, just make sure you restart the service or reboot to make sure all your changes are going into effect.

    Again log some of the queries with the debug option and go to some test pages to see if it is making a dent.
     
  5. jerrm

    jerrm Network Guru Member

    @Bunsen

    The cache-size directive should be irrelevant to list size. It only has to do with concurrent queries.

    For the same number of entries, a hosts file will be more memory efficient, but using the address directive will block more sites. The address directive blocks entire domains instead of individual hosts.

    Comparing "address=/badadsite.com/0.0.0.0/" to "0.0.0.0 badadsite.com" in a hosts file:

    • Both will block "http://badadsite.com."
    • The hosts entry will only block the exact address badadsite.com.
    • The address directive will also block ad1.badadsite.com, ad2.badadsite.com, todayshost.badadsite.com, bluepill.ad42.badadsite.com, etc.

    If the list is properly stripped of subdomains, then address may be more efficient, but the tomato script doesn't do so last I checked. In the above example the downloaded lists may contain all of the entries - badadsite.com, ad1.badadsite.com, ad2.badadsite.com, todayshost.badadsite.com, bluepill.ad42.badadsite.com. If using the address directive only badadsite.com is needed, the other entries are a waste.

    If everything downloaded for me, your "list of lists" cleans to 828,898 lines using the tomato script. Stripped of subdomains, it reduces to 528,710.
     
    Last edited: Mar 11, 2019
    Bunsen and Mr9v9 like this.
  6. Bunsen

    Bunsen Reformed Router Member

    @Mr9v9: It's definitely making a dent! Thank you
    @jerrm: great point about subdomains. I hadn't considered that.
    If "address=/badadsite.com/0.0.0.0/" and "address=/ad42.badadsite.com/0.0.0.0/" are in the config - do they both get loaded or is it smart enough to know it's already covered? [basically I'm asking if I should bother stripping them]

    How did you go about stripping them off?
    After a bit of thinking...my solution is: sed 's/.*\.\(.*\..*\..*\)/\1/'

    I have no doubt you can do better..there are several things this won't work catch.
     
    Last edited: Mar 12, 2019
  7. jerrm

    jerrm Network Guru Member

    @Bunsen - unless it has changed recently, they both get loaded and eat memory.

    My solution was to find the highest level name for any domain present and strip out any redundancies, not assume a certain number of levels.

    See the optimize mode in Script: Adblock - not so lean, Probably not the speediest method if you have optware and php/python/perl/etc available, but was the best I came up with using tomato's base shell tools.
     
    Bunsen likes this.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice