1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How To Add Local blacklist.txt to Adblock? (Samba URL?)

Discussion in 'Tomato Firmware' started by terrelsa13, Sep 9, 2017.

  1. terrelsa13

    terrelsa13 Connected Client Member

    I am trying to add my own local blacklist file to the Adblock page.

    It looks like I should be able to add a new entry using a samba url (last row).
    [​IMG]

    But the logs show an error finding this URL.
    Code:
    Sep  8 23:38:37 CYBERTRON user.info adblock: [6] downloading blacklist - https://raw.githubusercontent.com/WindowsLies/BlockWindows/master/hosts
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: query[AAAA] raw.githubusercontent.com from 127.0.0.1
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: forwarded raw.githubusercontent.com to 127.0.0.1
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: validation result is INSECURE
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: reply raw.githubusercontent.com is <CNAME>
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: reply github.map.fastly.net is NODATA-IPv6
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: query[AAAA] github.map.fastly.net from 127.0.0.1
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: cached github.map.fastly.net is NODATA-IPv6
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: query[A] raw.githubusercontent.com from 127.0.0.1
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: cached raw.githubusercontent.com is <CNAME>
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: forwarded raw.githubusercontent.com to 127.0.0.1
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: validation result is INSECURE
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: reply raw.githubusercontent.com is <CNAME>
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: reply github.map.fastly.net is 151.101.112.133
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: query[A] raw.githubusercontent.com from 127.0.0.1
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: cached raw.githubusercontent.com is <CNAME>
    Sep  8 23:38:37 CYBERTRON daemon.info dnsmasq[11054]: cached github.map.fastly.net is 151.101.112.133
    Sep  8 23:38:38 CYBERTRON user.info adblock: ... [6] found 109 entries
    Sep  8 23:38:38 CYBERTRON user.info adblock: skip disabled blacklist - https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/social/hosts
    Sep  8 23:38:38 CYBERTRON user.info adblock: [7] downloading blacklist - smb://hieroglyph/public/adblock/blacklist.txt
    Sep  8 23:38:38 CYBERTRON user.info adblock: ... [7] download error! Please check URL
    I do not think it is a permissions issue as I use this particular folder as the guest "storage space". And it is accessible from a different machine.
    [​IMG]
    [​IMG]
    [​IMG]

    I can confirm the domains in this blacklist.txt file can still be reached after I get the above error.

    I also tried putting the blacklist.txt file in the /opt/ folder on the usb stick plugged into the router. And putting the path to the blacklist.txt file a couple different ways. They both return the same download error.
    [​IMG]
    [​IMG]

    File path on router in /opt/ folder.
    [​IMG]


    Info: I know I can add the domains to the "Blacklisted Domains" text box. But I would like to move these into a text file.

    This is the contents of my blacklist file for now. More will be added to it once I can get this simple one working.
    Code:
    #########################################
    # Custom Adblock List
    127.0.0.1 localhost #IPv4 localhost
    ::1 localhost #IPv6 localhost
    #########################################
    127.0.0.1       collegehumor.com
    127.0.0.1       funnyordie.com
    127.0.0.1       textsfromlastnight.com

    Is there a way to add your own local blacklist file? What am I missing?
     
  2. Sean B.

    Sean B. LI Guru Member

    I don't use adblock, but the error in the log you posted implies either the URL format is invalid, or that connecting to smb://yoursite as a website failed.. sense the error response still called it a URL. Either case would mean having smb: instead of http: is not a recognized syntax by adblock to indicate a network share location rather than a web URL. IE: using the wrong ports. May be worth trying file:// instead.
     
    Last edited: Sep 10, 2017
  3. terrelsa13

    terrelsa13 Connected Client Member

    Thank you Sean B. I was hoping I was doing something wrong.

    But this still leaves my original question unanswered:
     
  4. Sean B.

    Sean B. LI Guru Member

    Sorry, I forgot to put in my suggestion. I edited the post to include it, which was to try file:// instead.
     
  5. terrelsa13

    terrelsa13 Connected Client Member

    Unfortunately, I was not able to get file:// to work for me. All of my attempts returned the same error as before.

    Original Paths
    Router File Path: /opt/adblock/blacklist.txt
    Server File Path: /media/Sphinx/Public/adblock/blacklist.txt
    Server SMB URL: smb://172.31.13.123/Public/adblock/blacklist.txt

    Unsuccessful File:// URL Attempts
    file://opt/adblock/blacklist.txt
    file:///opt/adblock/blacklist.txt
    file://172.31.13.10/opt/adblock/blacklist.txt
    file:///172.31.13.10/opt/adblock/blacklist.txt
    file://172.31.13.123/media/Sphinx/Public/adblock/blacklist.txt
    file:///172.31.13.123/media/Sphinx/Public/adblock/blacklist.txt
    file://172.31.13.123/Public/adblock/blacklist.txt
    file:///172.31.13.123/Public/adblock/blacklist.txt

    • Router IP 172.31.13.10
    • Server IP 172.31.13.123
    • Router hostname cybertron
    • Server hostname hieroglyph
     
  6. Sean B.

    Sean B. LI Guru Member

    None of those are the same as the URL in this screenshot you posted to show the file is accessible from other machines:

    [​IMG]

    So, if that screenshot indeed shows that the file and its location are accessible from other clients on the network.. the line to correctly try file:// with would be:

    file://hieroglyph/public/adblock/blacklist.txt

    as the URL needs to correctly represent the hostname and path of the file as seen by the network. Not a local /opt path etc. File:// may indeed not work, but don't chase your tail by not using a known working network path.
     
  7. terrelsa13

    terrelsa13 Connected Client Member

    Are you saying I cannot use the IP address of the machine and must use the hostname?

    I show at the bottom of my last post, the server hostname hieroglyph has the IP 172.31.13.123.
     
  8. Sean B.

    Sean B. LI Guru Member

    That's exactly what I'm saying. The network share path is not the same as a network IP or hostname, more specifically it is a designation made within the SMB protocol.. and using an IP in place of the share path will not work. Remember, you're not using a network browser, you're using a web browser. Try it using the IP in the same way you made the screenshot. I think you'll find it fails.
     
  9. terrelsa13

    terrelsa13 Connected Client Member

    I've never had that problem before. It works the same using the hostname or the IP for me.
    Ignore the script I added to the folder.
    [​IMG]

    But just to be sure. I tried using the hostname and get the same error.
     
  10. Sean B.

    Sean B. LI Guru Member

    That is not a web browser, and therefor not a URL, hence not the same way you tried it in the screenshot. If you wish to see what I mean, try it in a web browser as you were accessing it before. Changing your process of doing things as you're trying to track a problem is very bad practice in terms of diagnostics. But, a moot point if you did in fact try file:// with the network path as shown in the screenshot I referenced and it didn't work. I'll see if I can track down any other possibilities for you.
     
  11. terrelsa13

    terrelsa13 Connected Client Member

    You are right. And I have learned something: Neither smb url will work in the web browser until it is opened in the file manager. This may be a clue as to why adblock in the router is returning a "download error".

    So when accessing smb://172.31.13.123/Public/adblock/ in the file manager I can also open the blacklist.txt file it in a web browser.
    Same for when accessing smb://hieroglyph/Public/adblock/ in the file manager, only then can blacklist.txt be opened in the web browser.
    [​IMG]

    To be more clear here. I have to open the smb url in the file manager before I can open it in the web browser.

    Thanks!
     
  12. Sean B.

    Sean B. LI Guru Member

    I'd say something isn't quite right with either the share permissions or samba config. You should be able to access the file by using the full and correctly formatted share path via a web browser without having to establishing a connection with file manager first.
     
  13. terrelsa13

    terrelsa13 Connected Client Member

    I am using the Samba graphical configuration tool. Here is my samba config:
    https://pastebin.com/xHZTev8F

    The Public folder of this samba share has recursively been set with 777 permissions and nobody:nogroup ownership.
     
  14. Sean B.

    Sean B. LI Guru Member

    That is your actual samba config file, and not still the default example config?


    And what is your network configuration?
    IE:

    Is there a domain or active directory for the network?

    Is there a WINS server on the network?

    Any other routers or relays in between clients/server and the router running samba?
     
  15. Sean B.

    Sean B. LI Guru Member

    I don't see any shares in that config which would correspond to the hieroglyph URL that has been used in the screenshots.. so I'm going to assume this is the one that would be used as it's the only Public folder path.

    Code:
    [Public]
        comment = Public Share
        path = /media/Sphinx/Public
        writeable = yes
    ;   browseable = yes
        guest ok = yes
    Remove the ; from the "browseable = yes" line. If there's not already a WINS server on the network, and as long it wouldnt be a conflict/compatibility issue with your network configuration, enable nmbd to be a WINS server on the network by removing the # from this line and changing no to yes:

    Code:
    #   wins support = no
    Then, in the routers web interface goto Basic->Network and put the IP address of that machine in the box for "WINS (for DHCP)". Make sure all the clients renew their DHCP leases, either manually or via reboot.
     
    Last edited: Sep 10, 2017
  16. terrelsa13

    terrelsa13 Connected Client Member

    I am not sure what these are or what they are used for. So I have not set anything like this up.
    If they come turned on by default in Linux Mint 17.3 or Shibby v140, they still have the default settings.

    I do have routers setup as access points for wifi. They do not act as dhcp servers. I used this link to configure them as APs.
    http://www.linksysinfo.org/index.ph...router-a-dumb-access-point.37403/#post-182212

    Router is not running samba. The samba share is running on a desktop (hieroglyph) running Linux Mint 17.3.
    The desktop is directly connected to a switch which is connected to the router (cybertron).

    The blacklist.txt file is in two places.
    First place is the router located here: /opt/adblock/blacklist.txt
    Second place is the desktop located here: /media/Sphinx/Public/adblock/blacklist.txt

    It's really late here. I'll have to continue tomorrow.
    I will reply when I remove the semi colon from in front of browseable=yes in the samba config file and give it a try.
     
  17. terrelsa13

    terrelsa13 Connected Client Member

    Thanks for the help so far Sean B.
     
  18. Sean B.

    Sean B. LI Guru Member

    You're quite welcome. Also note I added instructions on how to enable the WINS server on that linux box to my previous post.
     
  19. Sean B.

    Sean B. LI Guru Member

    Should have verified this before, but when you access the share from file manager it's not asking for a login/password correct? And a login/password hasn't been "remembered" by file manager that you forgot about?
     
  20. remlei

    remlei Networkin' Nut Member

    adblock fetch its database using wget which only support http/s and ftp (not sure if wget busybox supports ftp) and there's no easy way to do it.

    so there's 2 way to have your own blacklist, which 1 add it on dnsmasq, or host it using tomato's builtin apache server or maybe using the builtin httpd (which what tomato uses for its tomato web admin page).
     
    terrelsa13 likes this.
  21. terrelsa13

    terrelsa13 Connected Client Member

    Made both of these changes to the smb.conf file. Then rebooted all my devices.

    Then gave these a try in adblock:
    file://172.31.13.123/Public/adblock/blacklist.txt
    file:///172.31.13.123/Public/adblock/blacklist.txt
    file://hieroglyph/Public/adblock/blacklist.txt
    file:///hieroglyph/Public/adblock/blacklist.txt

    Same error as before.

    When I access my share from the file manager the login/password prompt does appear. I leave it at "Connect As: Anonymous" and press Connect because there is no username/password.

    [​IMG]
     
  22. Sean B.

    Sean B. LI Guru Member

    Anonymous is still a login. The guest account is requiring the user "anonymous" and a blank password. And from the post a few above it appears SMB is not functional with adblock anyway sense it uses wget.

    Is there a reason you don't simply run httpd on hieroglyph? Point it to a directory, make a file called list.html which contains:

    Code:
    <html>
      <head>
        <title>List</title>
      </head>
      <body>
        TYPE YOUR PLAIN TEXT LIST OF SITES HERE
      </body>
    </html>
    And there you have it. Use http://172.31.13.123/directoryyoumade/list.html as a URL in adblock.
     
  23. terrelsa13

    terrelsa13 Connected Client Member

    The steps below are not the best way to add a custom list to adblock.
    A better way can be found here.

    For me, looks like the dnsmasq route is the easiest.

    Here's what I did:
    1) Create (or in my case edited) a blacklist file which will contain the sites you want to block.
    In my case I ssh into the router and use a usb stick mounted as the /opt folder to hold any persistent data. Your may create the blacklist file some other way or in some other location. Just make sure you use the correct location in step #4.

    ssh into the router and do the following:
    Code:
    # mkdir /opt/adblock/
    # nano /opt/adblock/blacklist.txt

    2) Put the following inside of the blacklist.txt file.
    Code:
    address=/textsfromlastnight.com/0.0.0.0
    address=/funnyordie.com/0.0.0.0
    address=/collegehumor.com/0.0.0.0
    *Most likely you want to block different or additional websites. You can add them to this blacklist file now or later.
    Save the blacklist.txt file.

    3) Log into your routers webpage.
    Navigate To: Advanced Settings > DHCP/DNS > Dnsmasq Custom Configuraiton

    4) In the Dnsmaq Custom Configuration text box enter:
    Code:
    conf-file=/opt/adblock/blacklist.txt
    Save these changes.

    5) Once the router is done saving, you should not be able to access the websites you added to the blacklist.txt file in step #2.
    *There does not appear to be any issue using the routers built-in adblock along with this additional blacklist.txt file.


    Thanks to @remlei for the suggestion to use dnsmasq.

    @Sean B. I was wrapping up this reply when you sent the suggestion to use httpd on hieroglyph. I like the idea of having this file live on the router. That way if hieroglyph is off the blacklist file is still accessible. Thank you for all the help!
     
    Last edited: Sep 12, 2017
  24. Sean B.

    Sean B. LI Guru Member

    I assumed ( my bad, I should know better ) that you specifically wanted to have the files on a remote server, as doing so locally on the router would be much easier. For instance, simply use the wwwext/cgi-bin directory and making the same file I described in my last post.. you then use the routers IP as the URL preceeded by the username and password you use to access the routers GUI..

    http://user:password@ROUTER-IP-ADDRESS/ext/cgi-bin/list.html

    The file would not survive a reboot.. but a few lines put into the init script page would rebuild it every time.

    Few things about the solution you're going with:

    A: Using the conf-file option bypasses /etc/dnsmasq.conf in tomato and uses only your blacklist.txt file as its conf file.. as when dnsmasq hits that conf-file line in /etc/dnsmasq.conf ( where the custom config box puts stuff in addition to the defaults ) it will restart and access only that .txt file. In other words, you just killed many options that need to be set. I strongly suggest you do not do this.

    B: There is absolutely no reason to make the .txt file in the first place. All those address lines you made the txt file for can simply go into the custom config box in DHCP/dns, and they will be added to all the other config lines that tomato needs in dnsmasq by default. The suggestion of doing it the way you were told is.. well.. shouldn't exist.

    C: I'd suggest using 127.0.0.1 on the end of those address lines instead of 0.0.0.0

    D: I had the impression adblock added stronger layers of protection to the sites listed being accessed.. as the address= lines in dnsmasq are not terribly reliable, could even been circumvented on accident really.

    Just some food for thought.
     
  25. terrelsa13

    terrelsa13 Connected Client Member

    I may not understand what you are saying:
    Using the Dnsmaq Custom Configuration text box adds to the already existing dnsmasq.conf file. It doesn't appear to overwrite it.
    All the sites downloaded from the adblock webpage and my blacklist.txt file seem to be blocked.

    This is my router "auto-generated" /etc/dnsmasq.conf file: https://pastebin.com/ftzVEcd2

    You can find my added option (conf-file=/opt/adblock/blacklist.txt) is the last line. However you can see at lines 47-48 the router "auto-generates" these lines according to my router setup. And then adds the last line according to what is entered into the Dnsmaq Custom Configuration text box.

    Is it not ok to use the conf-file option more than once when it is inside of the /etc/dnsmasq.config file???
    Code:
    conf-file=/etc/dnsmasq.adblock
    conf-file=/etc/trust-anchors.conf
    Does this mean the trust-anchors.conf is is killing the dnsmasq.adblock.conf?
    And then my newly added blacklist.txt is killing the trust-anchors.conf?

    From dnsmasq manpage:
    Code:
           -C, --conf-file=<file>
                  Specify  a different configuration file. The conf-file option is also allowed in configuration files,
                  to include multiple configuration files. A filename of "-" causes dnsmasq to read configuration  from
                  stdin.
    You are correct. But eventually as enough websites get added wouldn't this exhaust my NVRAM?
    But yes, for all practical purposes idk how many site entries it would take to use up "too much" NVRAM.

    I will give this change a try.

    Yep, I understand there is not going to be a 100% effective way to block sites (switching to 4G or using a VPN bypasses adblock). Not to mention I understand there's no way to block every website with a particular content (i.e. porn) as there are a virtually unlimited amount of them.

    If someone wants to access a blocked site bad enough and they figure out how to defeat adblock, then bravo to them, because they probably just learned something. :)
     
    Last edited: Sep 10, 2017
  26. terrelsa13

    terrelsa13 Connected Client Member

    Actually I have a question about this:
    The /etc/dnsmasq.adblock file (created by the router adblock webpage) uses 0.0.0.0 for all of its entries.
    What are the pros/cons of using 0.0.0.0 vs 127.0.0.1?

    To be more specific. I kind of get the difference between the two. One is the loop back address and the other is saying something like, "listen to all interfaces on this device".

    I have 4 VLANs. So my question is specific to the pros/cons for use in this dnsmasq adblock scenario when using multiple VLANs. In this case does 127.0.0.1 still make sense?
     
  27. terrelsa13

    terrelsa13 Connected Client Member

    As @Sean B. suggested, "simply use the wwwext/cgi-bin directory". Putting the list of sites you want to block into the wwwext/cgi-bin/ folder is the better way to do this.

    Sym Link: /www/ext/cgi-bin/
    Path: /tmp/var/wwwext/cgi-bin/

    *My previous post should not be followed. I suggest removing any changes made from the previous post.*

    Here's what I did:
    I am making two assumptions:
    -You are able to ssh into your router.
    -You have a usb drive mounted as the /opt folder. (I used these instructions to mount my usb as the /opt folder.)

    1) ssh into the router and enter the following:
    Code:
    # mkdir /opt/adblock/
    # nano /opt/adblock/list.html
    2) Put the following inside of the list.html file.
    Code:
    <html>
      <head>
        <title>list.html</title>
      </head>
      <body>
    127.0.0.1 textsfromlastnight.com
    127.0.0.1 funnyordie.com
    127.0.0.1 collegehumor.com
      </body>
    </html>
    Most likely you want to block different or additional websites. You can add them to this list.html file now or later.

    Now save the list.html file.

    3) Log into your router's webpage.

    4) Navigate To: USB & NAS > USB Support > Run before unmounting

    You will already have a line similar to this (/opt/etc/init.d/rc.unslung start) in text box after you have followed the instructions to mount a usb drive as the /opt folder.

    Put the following underneath the rc.unslung start line.
    Code:
    cp /opt/adblock/list.html /tmp/var/wwwext/cgi-bin/list.html
    Save these changes.

    5) Navigate To: Advanced Settings > Adblock

    In the Blacklists section add a new entry and "use the routers IP as the URL preceeded by the username and password you use to access the routers GUI.."

    http://username:password@ROUTER-IP-ADDRESS/ext/cgi-bin/list.html
    It is not a typo. After the ip address it should say /ext and not /www/ext.

    Save these changes.

    7) You will have to either:
    -Unmount your usb drive, mount your usb drive, and save the adblock page again.
    Or...
    -Reboot

    8) Now, you should not be able to access the websites in the /opt/adblock/list.html file.

    Anytime the /opt/adblock/list.html file is changed you will need to repeat step #7.
     
    Sean B. likes this.
  28. Sean B.

    Sean B. LI Guru Member

    My mistake, multiple instances of conf-file can be used. Koitsu and myself had to develop a patch due to Tomato's Samba config creating a default interfaces line, and if you added or changed the interface with your own line in the custom box the 2 would conflict and Samba would go nuts. For some reason I was thinking of that when I answered.

    Using 0.0.0.0 can end up with different results depending on how the computer/network are configured. If the computer has multiple interfaces with different IP's it could result in multiple connection attempts on different interfaces. At the very least it will likely result in a delay and then a DNS error as I don't think a host will take 0.0.0.0 as a valid response ( still blocks the site, but with a more noticeable cause ). Where as 127.0.0.1 is a local-only fake interface.. nothing about the request will leave the computer itself ( aside from the initial DNS query for the redirected domain ), and will fail with a connection failed or refused status. Another option would be to redirect to a plain text page on your local server or another file on the router in the wwwext directory.. to notify the user that this website is disallowed on your network.
     

Share This Page