1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

pixelserv compiled to run on router WRT54G

Discussion in 'Tomato Firmware' started by Jedis, Sep 5, 2009.

  1. HunterZ

    HunterZ LI Guru Member

    @koitsu: Thanks. The command there has a typo, though (as mentioned in the comments below it), and should be:
    Code:
    CFLAGS="-static -Wl,-Bdynamic,-lgcc_s,-Bstatic"
    This gives me a 105KB binary instead of a 15KB one. I'll attach it here and see if it works for leandroong or others.

    Also, I did find that libgcc_s.so library in the entware toolchain (which I have a secondary toolchain), but its linker couldn't find it for some reason. It looks like entware's mipsel-uclibc-ldconf is hard-coded to look for cache files somewhere under /opt, and I didn't want to mess with trying to untangle that.
     

    Attached Files:

    Goggy, jerrm and M0g13r like this.
  2. leandroong

    leandroong Addicted to LI Member

    @HunterZ, result of my test

    I rename pixelserv.static to pixelserv

    Code:
    /media/optware/adblock # ./pixelserv -help
    Usage:./pixelserv [IP No/hostname (all)] [-n i/f (all)] [-o select_timeout (10 seconds)] [-p port (80) & (443)] [-r (enables redirect to encoded path in tracker links)] [-s /relative_stats_html_URL (/servstats) [-t /relative_stats_txt_URL (/servstats.txt) [-u user ("nobody")]
    /media/optware/adblock # readelf -d ./pixelserv
    
    Dynamic section at offset 0x12c contains 22 entries:
      Tag        Type                         Name/Value
    0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
    0x0000000c (INIT)                       0x403dd8
    0x0000000d (FINI)                       0x416ef0
    0x00000004 (HASH)                       0x400204
    0x00000005 (STRTAB)                     0x4029a0
    0x00000006 (SYMTAB)                     0x400d40
    0x0000000a (STRSZ)                      4234 (bytes)
    0x0000000b (SYMENT)                     16 (bytes)
    0x70000016 (MIPS_RLD_MAP)               0x429950
    0x00000015 (DEBUG)                      0x0
    0x00000003 (PLTGOT)                     0x429960
    0x70000001 (MIPS_RLD_VERSION)           1
    0x70000005 (MIPS_FLAGS)                 NOTPOT
    0x70000006 (MIPS_BASE_ADDRESS)          0x400000
    0x7000000a (MIPS_LOCAL_GOTNO)           50
    0x70000011 (MIPS_SYMTABNO)              454
    0x70000012 (MIPS_UNREFEXTNO)            25
    0x70000013 (MIPS_GOTSYM)                0xb1
    0x6ffffffe (VERNEED)                    0x403db8
    0x6fffffff (VERNEEDNUM)                 1
    0x6ffffff0 (VERSYM)                     0x403a2a
    0x00000000 (NULL)                       0x0
    /media/optware/adblock #
    
    /media/optware/adblock # sh adblock.sh
    ADBLOCK: Download starting
    ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Fri, 29 Aug 2014 08:52:11 GMT)
    ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Mon, 25 Aug 2014 12:50:23 GMT)
    read(net): Connection reset by peer
    ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Wed, 20 Aug 2014 18:15:38 GMT)
    ADBLOCK: Filters unchanged
    ADBLOCK: Exiting
    /media/optware/adblock #
    
    I have libgcc_s.so.1 in /opt/lib.
    Code:
    /media/optware/adblock # wget -S -O - http://10.0.1.254:/servstats
    --2014-09-01 18:47:56--  http://10.0.1.254/servstats
    Connecting to 10.0.1.254:80... connected.
    HTTP request sent, awaiting response...
      HTTP/1.0 404 Not Found
      Server: httpd
      Date: Mon, 01 Sep 2014 10:47:56 GMT
      Content-Type: text/html
      Connection: close
    2014-09-01 18:47:56 ERROR 404: Not Found.
    
     

    Attached Files:

    Last edited: Sep 1, 2014
  3. M0g13r

    M0g13r LI Guru Member

    nice :)

    409 req, 0 err, 0 tmo, 90 cls, 0 nou, 0 pth, 96 nfe, 149 ufe, 1 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 2 ico, 42 ssl, 7 sta, 0 stt, 22 rdr
     
  4. HunterZ

    HunterZ LI Guru Member

    leandroong, what happens if you run pixelserv manually like this:
    pixelserv 10.0.1.254 -r

    Does it then show up if you run:
    ps w
     
  5. HunterZ

    HunterZ LI Guru Member

    I wonder what causes the closed connections (cls). Port scanning maybe?
     
  6. leandroong

    leandroong Addicted to LI Member

    /media/optware/adblock # ./pixelserv 10.0.1.254 -r
    pixelserv[1821]: ./pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    /media/optware/adblock #

    /media/optware/adblock # ps w
    PID USER VSZ STAT COMMAND
    1 root 1556 S /sbin/init
    2 root 0 SW [kthreadd]
    3 root 0 SW [ksoftirqd/0]
    5 root 0 SW [kworker/u:0]
    6 root 0 SW< [khelper]
    7 root 0 SW [kworker/u:1]
    81 root 0 SW [sync_supers]
    83 root 0 SW [bdi-default]
    84 root 0 SW< [crypto]
    85 root 0 SW< [kblockd]
    94 root 0 SW [khubd]
    120 root 0 SW [kswapd0]
    121 root 0 SW [fsnotify_mark]
    159 root 0 SW [mtdblock0]
    164 root 0 SW [mtdblock1]
    169 root 0 SW [mtdblock2]
    174 root 0 SW [mtdblock3]
    179 root 0 SW [mtdblock4]
    184 root 0 SW [mtdblock5]
    189 root 0 SW [mtdblock6]
    205 root 0 SW [kworker/0:1]
    212 root 0 SW< [deferwq]
    329 root 1540 S /sbin/syslogd -s512 -b0 -S -D -O /tmp/syslog.log
    332 root 0 SW [scsi_eh_0]
    333 root 0 SW [usb-storage]
    338 root 1540 S /sbin/klogd
    382 root 0 SW [RtmpCmdQTask]
    407 root 0 SW [kworker/0:2]
    478 root 0 SW [jbd2/sda2-8]
    479 root 0 SW< [ext4-dio-unwrit]
    486 root 1544 S /sbin/detect_link
    488 root 1544 S /sbin/detect_internet
    491 nobody 1052 S /usr/sbin/dnsmasq
    574 root 1552 S /sbin/udhcpc -i eth3 -s /tmp/udhcpc.script -p /var/run/udhcpc0.pid -t4 -T4 -d -O26 -O33 -O121 -O249
    586 root 3016 S /usr/sbin/httpd -p 80
    588 root 1540 S telnetd
    590 root 2444 S /bin/inadyn --config /etc/inadyn.conf
    593 root 1544 S /sbin/watchdog
    600 root 1364 S /usr/sbin/dropbear -x -4 -s
    602 root 824 S /usr/sbin/infosvr br0
    604 root 812 S /usr/bin/miniupnpd
    606 root 1544 S /sbin/rstats
    610 root 948 S /bin/lld2d br0
    611 root 848 S /usr/sbin/networkmap -w
    639 root 1556 S /bin/sh
    648 root 3048 S /sbin/nmbd -D -s /etc/smb.conf
    650 root 3120 S N /sbin/smbd -D -s /etc/smb.conf
    654 nobody 720 S /sbin/portmap
    667 root 0 SW< [rpciod]
    675 root 0 SW [lockd]
    676 root 0 SW [nfsd]
    678 root 868 S /sbin/rpc.mountd
    692 root 776 S dnscrypt-proxy --local-address=127.0.0.1:2053 --daemonize --provider-name=2.dnscrypt-cert.ns4.ca.dns.open
    715 root 4080 S lighttpd -f /opt/etc/lighttpd/lighttpd.conf
    716 root 8108 S transmission-daemon -g /opt/etc/transmission
    718 root 9188 S aria2c --conf-path=/opt/etc/aria2.conf
    735 root 16628 S motion
    1195 nobody 1044 S /usr/sbin/dnsmasq
    1546 nobody 1044 S /usr/sbin/dnsmasq
    1689 nobody 1048 S /usr/sbin/dnsmasq
    1789 root 3356 S N /sbin/smbd -D -s /etc/smb.conf
    1798 root 1432 S /usr/sbin/dropbear -x -4 -s
    1799 root 1556 S -sh
    1811 root 1432 S /usr/sbin/dropbear -x -4 -s
    1812 root 1556 S -sh
    1816 root 1432 R /usr/sbin/dropbear -x -4 -s
    1817 root 1568 S -sh
    1819 root 0 SW [flush-8:0]
    1824 root 1544 R ps w
    /media/optware/adblock #
     
  7. HunterZ

    HunterZ LI Guru Member

    Does 'ps w' show that process 1821 is still running?
     
  8. leandroong

    leandroong Addicted to LI Member

    @HunterZ, maybe there is conflict. I'm running lighttpd on 10.0.1.1:81 running webui-aria2 and http://10.0.1.1:9090/transmission/web/.
    Let me know what to do or test....

    note: addl info
    br0:1 Link encap:Ethernet HWaddr BC:AE:C5:EB:29:60
    inet addr:10.0.1.254 Bcast:10.255.255.255 Mask:255.0.0.0
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
     
  9. leandroong

    leandroong Addicted to LI Member

    no. Will restart router to see any changes and get back to you

    edit2: still no after router restart. ifconfig display
    br0:1 Link encap:Ethernet HWaddr BC:AE:C5:EB:29:60
    inet addr:10.0.1.254 Bcast:10.255.255.255 Mask:255.0.0.0
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
     
    Last edited: Sep 1, 2014
  10. HunterZ

    HunterZ LI Guru Member

    pixelserv is supposed to listen on its own IP address, on ports 80 and 443 by default
     
  11. leandroong

    leandroong Addicted to LI Member

    Possible conflict with dnscrypt-proxy?
    I think it conflict with my lighttpd, mini server.

    Edit2: lighttpd is using port 81, only router GUI is assign to port 80
     
  12. HunterZ

    HunterZ LI Guru Member

    Does your syslog contain any error messages?
     
  13. jerrm

    jerrm Network Guru Member

    I think you're right. Judging from this:
    Code:
    /media/optware/adblock # wget -S -O - http://10.0.1.254:/servstats
    --2014-09-01 18:47:56--  http://10.0.1.254/servstats
    Connecting to 10.0.1.254:80... connected.
    HTTP request sent, awaiting response...
      HTTP/1.0 404 Not Found
      Server: httpd
      Date: Mon, 01 Sep 2014 10:47:56 GMT
      Content-Type: text/html
      Connection: close
    2014-09-01 18:47:56 ERROR 404: Not Found.
    
    Something is already listening on port 80. Pixelserv would never return a 404 or include a server header as far as I know.

    Pixelserv does not throw any errors to the console if it can't bind the port, but check syslog, you should see something like:
    Code:
    Sep  1 11:08:33 RTN66U daemon.info pixelserv[9784]: pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    Sep  1 11:08:33 RTN66U daemon.err pixelserv[9786]: Abort: Address already in use - :192.168.0.1:80
    
     
  14. leandroong

    leandroong Addicted to LI Member

    53:11 pixelserv[881]: ./pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    Sep 1 22:53:11 pixelserv[883]: Abort: Address already in use - :10.0.1.254:80
    Sep 1 22:56:01 pixelserv[887]: ./1pixelserv version: V35.HZ6 compiled: Aug 31 2014 19:30:54 from pixelserv.c
    Sep 1 22:56:01 pixelserv[888]: Abort: Address already in use - 10.0.1.254:80
     
  15. leandroong

    leandroong Addicted to LI Member

    What remedy to change port?
     
  16. HunterZ

    HunterZ LI Guru Member

    You're going to have to figure out what's getting in pixelserv's way and fix it. I don't think there's another way to do it.

    Edit: Is there a way to restrict lighttpd to listen on only a single IP address, so that it doesn't respond on 10.0.1.254?
     
    Last edited: Sep 1, 2014
  17. jerrm

    jerrm Network Guru Member

    You can see what is binding the port with a full (non busybox) version of netstat (netstat -nlp) or with lsof.

    If you can't prevent the offending program from grabbing the port, you could run pixelserv on an alternate port and use iptables to redirect.
     
  18. leandroong

    leandroong Addicted to LI Member

    can I have your copy of adblock.sh for comparison?
    No conflict found on netstat -nlp.
    After running adblock.sh, I notice ifconfig generate, bro:1 with inet addr:10.0.1.254.

    Edit2: Searching padavan manual I found something interesting
    Connect to router`s WEB-interface using SSH-tunnel without Putty

    Thanks for the article on how to "Connect to router`s WEB-interface using SSH-tunnel". As I don't run Windows, but a Mac, I wish to connect to the web-interface without Putty, preferably just using a command line command.

    I have done this before using SSH tunneling, however my recipe for that doesn't seem to apply for Dropbear. I did SSH into the router (that part succeeds) and then try to execute the port tunneling command like:

    ssh -L 8888:localhost:80
    It turns out that localhost needs to be replaced with the internal IP address of the router (which can be derived using ifconfig br0 | grep inet, like:

    ssh -L 8888:192.168.41.254:80
     
  19. mstombs

    mstombs Network Guru Member

    Bit of history, tomato used to bind just the port No for all router IP addresses and prevent anything else use port 80
    but after pointing this out in post No 2 on this thread!
    http://www.linksysinfo.org/index.ph...ed-to-run-on-router-wrt54g.30509/#post-149258

    Rodney provided a one line change to router web gui source to make it only grab the router local lan IP.
    http://www.linksysinfo.org/index.php?threads/thoughts-on-httpd.31863/#post-158068

    dd-wrt also had (and guess still does) this restriction, so scripts needed to move the web gui or pixelserv IP via iptables, for example:-

    http://www.howtogeek.com/51477/how-to-remove-advertisements-with-pixelserv-on-dd-wrt/

    Rodney also used to build static versions of pixelserv and many other utilities, see for example

    http://multics.minidns.net/tomato/

    the build scripts will be in there as well.
     
  20. leandroong

    leandroong Addicted to LI Member

    Same result after temporarily disabling/killing lighttpd. This is source code issue, i think, I tried rodney binary and no conflict on log but unable to locate pid.

    Edit2: Will stick to firefox addon meantime...

    Edit3: sh adblock.sh stop remove bro1 from ifconfig. Enabling will create. Therefore, there is really no conflict. But when I enter url 10.0.1.254, I will get router gui, same as 10.0.1.1. With adblock stop, 10.0.1.254 not accessible

    Edit4: Although adblock not running as daemon, I can still say that it did manage to remove youtube starting ad when playing.
     
    Last edited: Sep 1, 2014
  21. HunterZ

    HunterZ LI Guru Member

    I think mstombs is saying that your router's built-in web gui is probably the one forcibly listening to port 80 on *all* IPs bound to the router's interfaces, and not lighttpd, and that this was fixed in Tomato but probably not in whatever firmware you're using.

    Could you configure your router's web gui to listen on a port other than 80?
     
  22. leandroong

    leandroong Addicted to LI Member

    I just configure it to listen on port 100 and it works. Re-running adblock and ps w, shows
    1049 nobody 1176 S /media/optware/adblock/pixelserv 10.0.1.254 -r

    router log:
    Sep 2 01:21:04 ADBLOCK: Download starting
    Sep 2 01:21:10 dnsmasq[923]: nameserver 121.1.3.89 refused to do a recursive query
    Sep 2 01:23:14 ADBLOCK: Downloaded
    Sep 2 01:23:14 ADBLOCK: Generating /var/lib/adblock/blocklist
    Sep 2 01:23:17 ADBLOCK: Config generated, 16816 unique hosts to block
    Sep 2 01:23:17 ADBLOCK: Setting up pixelserv on 10.0.1.254
    Sep 2 01:23:17 pixelserv[1046]: /media/optware/adblock/pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    Sep 2 01:23:17 pixelserv[1049]: Listening on :10.0.1.254:80
    Sep 2 01:23:17 pixelserv[1049]: Listening on :10.0.1.254:443
    Sep 2 01:23:17 ADBLOCK: Done, restarting dnsmasq

    coment: ads not block on drama site given and main youtube page. only effective on removing initial ad on youtube play.

    edit2: Conflicting pid
    1049 nobody 1176 S /media/optware/adblock/pixelserv 10.0.1.254 -r
    1052 nobody 1044 S /usr/sbin/dnsmasq
    1060 root 0 SW [flush-8:0]
    1064 root 1544 R ps w
    /media/optware/adblock # ./pixelserv 10.0.1.254 -r
    pixelserv[1065]: ./pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
     
    Last edited: Sep 1, 2014
  23. HunterZ

    HunterZ LI Guru Member

    The drama site ads are blocked for me. What sources are you using in your config? Here are mine (note that I only use ones that report last update time so that I can update frequently without re-downloading lists every time):
    Code:
    ### Sources (uncomment desired blocklists) [must be compatible to the hosts file format!] ###
    ## MVPS HOSTS (~600k) [default]:
    SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
    ## pgl.yoyo.org (~70k) [default]:
    SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
    ## Hosts File Project (~3M!):
    SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    ## The Cameleon Project (~600k):
    #SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
    ## AdAway mobile ads (~20k):
    #SOURCES="$SOURCES http://adaway.sufficientlysecure.org/hosts.txt"
    ## hpHosts ad/tracking servers (~400k):
    #SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
    SOURCES="$SOURCES http://hosts-file.net/ad_servers.txt"
    ## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list):
    #SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt http://hosts-file.net/hphosts-partial.asp"
    #SOURCES="$SOURCES http://www.montanamenagerie.org/hostsfile/hosts.txt"
    ## MalwareDomainList.com (~40k):
    SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
    #SOURCES="$SOURCES http://dl.dropbox.com/u/32428671/m.txt?dl=1"
    And here is my blacklist file:
    Code:
    dra.amazon-adsystem.com
    amazon-adsystem.com
    blastro.com
    ads.ign.com
    ad.doubleclick.net
    c.brightcove.com
    admin.brightcove.com
    cb.brightcove.net
    For me, hosts-file.net, winhelp2002.mvps.org, and pgl.yoyo.org all block rubiconproject.com, which is where one of the drama site ads is hosted.
     
  24. leandroong

    leandroong Addicted to LI Member

    I use default, 1,2 and malware (last). Running adblock in router memory.
    Did you notice the conflict pid?
    edit2: 1049 vs 1065
     
  25. HunterZ

    HunterZ LI Guru Member

    Which adblock script are you running? (i.e. which forum thread)
     
  26. leandroong

    leandroong Addicted to LI Member

    Script: Clean, Lean and Mean Adblocking with little path modification regarding dnsmasq restart and iptable location

    edit2: see attachment file
     

    Attached Files:

  27. HunterZ

    HunterZ LI Guru Member

    Have you tried rebooting your desktop/laptop after enabling adblock, to clear cached DNS entries from your OS and browser?
     
  28. leandroong

    leandroong Addicted to LI Member

    closing browser and ipconfig /flushdns
    edit2: restarting router log as follows:
    Sep 2 01:59:45 ADBLOCK: Downloaded
    Sep 2 01:59:45 ADBLOCK: Generating /var/lib/adblock/blocklist
    Sep 2 01:59:47 ADBLOCK: Config generated, 16816 unique hosts to block
    Sep 2 01:59:47 ADBLOCK: Setting up pixelserv on 10.0.1.254
    Sep 2 01:59:48 pixelserv[762]: /media/optware/adblock/pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    Sep 2 01:59:48 pixelserv[765]: Listening on :10.0.1.254:80
    Sep 2 01:59:48 pixelserv[765]: Listening on :10.0.1.254:443
    Sep 2 01:59:48 ADBLOCK: Done, restarting dnsmasq

    edit 3:
    ps w
    765 nobody 1176 S /media/optware/adblock/pixelserv 10.0.1.254 -r

    conflict pid
    /media/optware/adblock # ./pixelserv 10.0.1.254 -r
    pixelserv[879]: ./pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
     
    Last edited: Sep 1, 2014
  29. HunterZ

    HunterZ LI Guru Member

    In that case I'm stumped.
     
  30. mstombs

    mstombs Network Guru Member

    re pid Nos, when pixelserv daemonizes the main process pid increments a couple of times, this is normal.
    Also pixelserv spawns (forks) a new process for every reply, so you can have multiple entries seen in ps or top lists, the higher No one should be transient and short lived, maybe more visible with the default timeout increase recently. Real web servers have a fixed no of sub processes and inter-process communication to distribute the workload, without the OS overhead of creating and destroying processes.
     
  31. leandroong

    leandroong Addicted to LI Member

    change setting: ramlist=0 and same blacklist, rebooting router.
    1. router log
    Sep 2 02:12:54 ADBLOCK: Downloaded
    Sep 2 02:12:54 ADBLOCK: Generating /media/optware/adblock/blocklist
    Sep 2 02:12:57 ADBLOCK: Config generated, 16824 unique hosts to block
    Sep 2 02:12:57 ADBLOCK: Setting up pixelserv on 10.0.1.254
    Sep 2 02:12:58 Samba Server: daemon is started
    Sep 2 02:12:58 pixelserv[766]: /media/optware/adblock/pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    Sep 2 02:12:58 pixelserv[775]: Listening on :10.0.1.254:80
    Sep 2 02:12:58 pixelserv[775]: Listening on :10.0.1.254:443
    Sep 2 02:12:58 ADBLOCK: Done, restarting dnsmasq

    2. ps w
    775 nobody 1176 S /media/optware/adblock/pixelserv 10.0.1.254 -r

    3. conflicting pid also
    /media/optware/adblock # ./pixelserv 10.0.1.254 -r
    pixelserv[884]: ./pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c

    re-running ps w, does not show any adblock on 884, only on 775 (original)
     
    Last edited: Sep 1, 2014
  32. leandroong

    leandroong Addicted to LI Member

    There is still ads in drama. We need to figure out that little conflicting issue.

    edit 2: I don't have pid file, adblock.pid, on /var/run. This could be the reason?
     
    Last edited: Sep 1, 2014
  33. HunterZ

    HunterZ LI Guru Member

    Adblock doesn't have a pid except when it's starting/stopping. Adblock's job is to generate a blocklist and feed it to dnsmasq, which causes dnsmasq to direct blocked domain names to nullserv's IP address.

    Do you have dnsmasq configured to act as a DNS server, and do you have your PC configured to use your router as its sole DNS server?
     
  34. M0g13r

    M0g13r LI Guru Member

    nothing special it's my browser :)
     
  35. leandroong

    leandroong Addicted to LI Member

    my dns server as follow: 10.0.1.1 218.102.23.228 121.1.3.89

    Let me try changing PIXEL_IP to 253, will reboot router
    edit2:
    1. router log
    02:38:47 ADBLOCK: Config generated, 16824 unique hosts to block
    Sep 2 02:38:47 ADBLOCK: Setting up pixelserv on 10.0.1.253
    Sep 2 02:38:47 Samba Server: daemon is started
    Sep 2 02:38:47 pixelserv[766]: /media/optware/adblock/pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    Sep 2 02:38:47 ADBLOCK: Done, restarting dnsmasq
    Sep 2 02:38:47 pixelserv[775]: Listening on :10.0.1.253:80
    Sep 2 02:38:47 pixelserv[775]: Listening on :10.0.1.253:443

    2. ps w
    775 nobody 1176 S /media/optware/adblock/pixelserv 10.0.1.253

    3. still conflict pid
    /media/optware/adblock # ./pixelserv 10.0.1.253 -r
    pixelserv[880]: ./pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
     
    Last edited: Sep 1, 2014
  36. HunterZ

    HunterZ LI Guru Member

    You need to use only the router's DNS server, or else there's no guarantee that the OS will route DNS queries to the router instead of another server.

    If your router's web GUI has a DNS intercept option, you should enable that too.
     
  37. HunterZ

    HunterZ LI Guru Member

    I get those too, though. For all I know it could be the mobile devices in the house doing screwy things. If I wasn't feeling lazy I might attempt to implement logging of the offending IP addresses.

    Speaking of logging, I accidentally left a syslog call enabled that should have been changed back to a MYLOG call. If I end up making other changes I will be sure to fix that too.
     
  38. leandroong

    leandroong Addicted to LI Member

    just changed router to 1 dns server, 10.0.1.1, result still the same. Activating firefox addon for the meantime. Thanks
     
  39. HunterZ

    HunterZ LI Guru Member

    No, the router needs to use various WAN servers and not the router itself.

    LAN clients (PCs, etc.) are the ones that need to use only the router as their only DNS server.
     
  40. leandroong

    leandroong Addicted to LI Member

    correct, i'm talking pc dns point to 10.0.1.1 only
     
  41. AndreDVJ

    AndreDVJ Addicted to LI Member

    I believe the statically-linked build included isn't that static, because the library libgcc_s.so.1 won't go into the binary.

    Your static build:
    Code:
    root@WNR3500L:/tmp/mnt/storage/pixelserv-V35.HZ7/dist# readelf -d pixelserv.static
    Dynamic section at offset 0x12c contains 22 entries:
      Tag  Type  Name/Value
    0x00000001 (NEEDED)  Shared library: [libgcc_s.so.1]
    0x0000000c (INIT)  0x403dd8
    0x0000000d (FINI)  0x416ef0
    0x00000004 (HASH)  0x400204
    0x00000005 (STRTAB)  0x4029a0
    0x00000006 (SYMTAB)  0x400d40
    0x0000000a (STRSZ)  4234 (bytes)
    0x0000000b (SYMENT)  16 (bytes)
    0x70000016 (MIPS_RLD_MAP)  0x429950
    0x00000015 (DEBUG)  0x0
    0x00000003 (PLTGOT)  0x429960
    0x70000001 (MIPS_RLD_VERSION)  1
    0x70000005 (MIPS_FLAGS)  NOTPOT
    0x70000006 (MIPS_BASE_ADDRESS)  0x400000
    0x7000000a (MIPS_LOCAL_GOTNO)  50
    0x70000011 (MIPS_SYMTABNO)  454
    0x70000012 (MIPS_UNREFEXTNO)  25
    0x70000013 (MIPS_GOTSYM)  0xb1
    0x6ffffffe (VERNEED)  0x403db8
    0x6fffffff (VERNEEDNUM)  1
    0x6ffffff0 (VERSYM)  0x403a2a
    0x00000000 (NULL)  0x0
    Mine, compiled from router:
    Code:
    root@WNR3500L:/tmp/mnt/storage/pixelserv-V35.HZ7/dist_static# readelf -d pixelserv
    There is no dynamic section in this file.
    If I take strip off the script, build size increase from 311 KB (319.428 bytes) to 312 KB (319.508 bytes)! Just 1 kilobyte! LoL!

    The binary works, though the results are pretty much the same, some ads I found won't redirect.

    I am attaching my builds if anyone wants to check and use. I am compiling directly on the router, using Lancethepants' Tomatoware.
     

    Attached Files:

  42. HunterZ

    HunterZ LI Guru Member

    Yeah, like I said, my toolchain doesn't have libgcc_s.so.1 so I can't include it.

    I should probably set up Tomatoware.
     
  43. mstombs

    mstombs Network Guru Member

    Update re dynamic link version

    I still get lots of timeouts, with the new default 10s wait, lots of stat requests - because I could see Chrome or Chromium generating 3 requests for every key press, here's the progression over the last couple of days in household us

    Code:
    Sep 1 09:07:31 rtn66u daemon.info pixelserv[21808]: /mnt/usb4gb/pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    Sep 1 20:30:01 rtn66u daemon.info pixelserv[21810]: 3434 req, 0 err, 494 tmo, 3 cls, 0 nou, 0 pth, 161 nfe, 1460 ufe, 20 gif, 24 bad, 0 txt, 0 jpg, 0 png, 28 swf, 2 ico, 1074 ssl, 2 sta, 32 stt, 134 rdr
    Sep 2 20:30:01 rtn66u daemon.info pixelserv[21810]: 6986 req, 0 err, 992 tmo, 9 cls, 0 nou, 0 pth, 286 nfe, 3088 ufe, 43 gif, 25 bad, 0 txt, 10 jpg, 2 png, 43 swf, 4 ico, 2151 ssl, 21 sta, 34 stt, 278 rdr
    Sep 2 23:51:42 rtn66u daemon.info pixelserv[21810]: 7751 req, 0 err, 1082 tmo, 10 cls, 0 nou, 0 pth, 301 nfe, 3413 ufe, 52 gif, 25 bad, 0 txt, 10 jpg, 2 png, 43 swf, 5 ico, 2433 ssl, 41 sta, 35 stt, 299 rdr
    Now experimenting with 20s timeout using "-o 20", which may be the answer on my system, but probably only treating symptom not root cause!

    Code:
    /mnt/usb4gb/pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    133 req, 0 err, 0 tmo, 0 cls, 0 nou, 0 pth, 0 nfe, 17 ufe, 0 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 4 ssl, 100 sta, 12 stt, 0 rdr
    NB current code sends txt by default, so ufe = unknown file extension are all txt replies.
     
  44. jerrm

    jerrm Network Guru Member

    I still get some timeouts, but not nearly as many as you are:
    Code:
    /opt/bin/pixelserv version: V35.HZ7 compiled: Aug 31 2014 22:11:55 from pixelserv.c
    9073 req, 0 err, 166 tmo, 2 cls, 0 nou, 0 pth, 964 nfe, 4245 ufe, 26 gif, 182 bad, 0 txt, 4 jpg, 5 png, 0 swf, 0 ico, 2312 ssl, 25 sta, 17 stt, 1125 rdr
     
  45. HunterZ

    HunterZ LI Guru Member

    I haven't been getting timeouts at 10 seconds, while I did at 5 and whatever the old value was (2?). I'm glad I added a command line parameter, as it sounds like it may be different for everyone.

    Are timeouts caused by the browser taking too long to request something after connecting to pixelserv? 10 seconds sounds like an awful long time.

    Yeah. I wanted to break it all out, though, so that I could tell which code path was executed.

    I enabled some syslog output and saw that most of the ufe cases were .js (JavaScript) file requests. It may be worth adding .js as an explicit case for counting purposes.
     
  46. leandroong

    leandroong Addicted to LI Member

    I agree on that. FF site can't be block even added to blacklist

    taken from drama site, http://www.dramago.com/drama-updates
    http://ad.wingads.com/show/wing_bar.js
    http://www.dramago.com/ads/header_1.html
    http://www.dramago.com/ads/sidebar_lower_1.html
    http://www.dramago.com/ads/sidebar_upper_1.html
    http://www.google-analytics.com/analytics.js
    http://www.statcounter.com/counter/counter.js

    Youtube main page:
    Code:
    https://ad.doubleclick.net/N4061/adi/com.ythome/_default;sz=970x250;tile=1;ssl=1;dc_yt=1;kbsg=HPPH140903;kga=-1;kgg=-1;klg=en;kmyd=ad_creative_1;ytexp=930811,911305,901096,936932,946013,936120,931839;ord=5072827942750705?
    https://ad.doubleclick.net/N6762/adi/mkt.ythome_1x1/;sz=1x1;tile=3;ssl=1;dc_yt=1;kga=-1;kgg=-1;klg=en;kmyd=ad_creative_3;ytexp=930811,911305,901096,936932,946013,936120,931839;ord=6114001072988757?
    
    edit3: taken from adblock+ firefox addon blockable item lists

    edit4: modify code and I can test.
     
    Last edited: Sep 3, 2014
  47. mstombs

    mstombs Network Guru Member

    Thanks again for reviving the code development!

    In the history of this thread - the first versions following the perl code ignored the request and always sent a gif reply. This clearly didn't work for some we pages in IE - you would get script errors where the browser ignored the html header and tried to run the gif. So 'js*' were at one time specifically identified, and for counting purposes I agree would be interesting to isolate from 'truly unknown'.

    I do not understand the timeouts, I thought the browser issued the "GET" request and the OS would open the channel and use it in milliseconds, so it is surprising that 2, 10, 20 make a difference. I do remember that without the timeout processes could be stuck forever, and that attempts to handle all requests in one thread didn't work (browser slowdowns and errors) - it does take a couple of seconds to gracefully shutdown a connection. Behaviour is clearly different between browsers, I wonder if they don't like the 'connection close' command and are trying to re-use old connections?

    It's not Tomato specific, I can now see the problem in my Ubuntu VM between Firefox and Chromium just refreshing the stats page. I guess I need to use wireshark or tcpdump to see what is actually happening.

    In case it is relevant I never understood why the GNU macro TEMP_FAILURE_RETRY was needed on the first "select". I understand that the main thread gets interrupted to service the stats collection, but the signal flag SA_RESTART was supposed to resume things. We don't handle SIGPIPE which I understand could be generated by attempting to send over a broken connection, but I believe I experimented with ignoring that to no effect. Miniupnpd used to suffer from 'Broken Pipe' errors, not seen that message here!

    This may be relevant a similar issue elsewhere, maybe the select gets interrupted and subsequent comms broken?
    https://gitorious.org/rowboat/libcore/commit/3c6ca77845ce5f0e80c4dc49617601918023c1ed
     
    Last edited: Sep 3, 2014
  48. HunterZ

    HunterZ LI Guru Member

    It's probably your adblock configuration and not pixelserv that is at fault. When pixelserv is not running you will still get ad blocking, but it will be slow because your browser gets no response from the pixelserv IP.

    Therefore, if you're seeing ads, please ask for support in the adblock thread.

    No problem. I just didn't like that there were two forks that both had different good ideas.

    No idea. Any recommended code changes?

    Note that it's going to be harder for me to test for timeouts because I don't seem to have as much of a problem. I can try lowering my timeout window.
     
  49. jerrm

    jerrm Network Guru Member

    Agree with HunterZ. Pixelserv has no impact on whether the ad is blocked or not. The discussion doesn't belong here.

    The only possible exception is the redir code could ultimately redirect the browser to a site that is not in the blocklist, but I have seen no indication of this actually happening.

    Other than that one, very remote possibility, if the source for the ad is in the lists you use, and the router's dnsmasq is the only dns available, the ad should be blocked. Clear the browser cache and dns cache. Perform an nslookup on the domains in question, what IP do you get?
     
  50. leandroong

    leandroong Addicted to LI Member

    Maybe this is FW issue, i'm using padavan and not tomato. Pixelserv is running with all test that you have perform. On my android tab 2, Youtube ads is block and therefore working.
    Anyway, enough for the moment and will wait future versions. Happy running firefox adblock+ at the same time.
     
  51. leandroong

    leandroong Addicted to LI Member

    Bug, when edit router port forwarding, after save, iptables will restart and losing adblock registered by adblock.sh
    Remedy: re-run adblock.sh
     
  52. HunterZ

    HunterZ LI Guru Member

    Yes, this is a limitation of adblock.sh.
     
  53. jerrm

    jerrm Network Guru Member

    Not my version, but I have no idea if @leandroong 's padavan supports .fire event scripts (or how my rules would interact with padavan).
     
  54. HunterZ

    HunterZ LI Guru Member

    Sounds like I really need to update my installed adblock.sh, because I'm probably way out of date. I think I had made some custom tweaks, but I don't remember what they are at this point.
     
  55. jerrm

    jerrm Network Guru Member

    If you happy, probably not worth the effort.
     
  56. Beast

    Beast Network Guru Member

    Drama.png
    Hi

    I run adblock (jem's I think,,not the 4.5 by harp) and pixleserv H7. I dont see any adds on dramago.com
     
  57. leandroong

    leandroong Addicted to LI Member

    @Beast, thanks for the screen shot. I'm trying to figure out by reading source code. Something is not right on this padavan FW
     
  58. leandroong

    leandroong Addicted to LI Member

    ignore. It is correct
     
    Last edited: Sep 4, 2014
  59. mstombs

    mstombs Network Guru Member

    Well I have no idea about updates, but I'd like the syslog with "client closed connection without sending any data" disabled because all increasing the timeout has done is move errors from timeout to cls!

    Code:
    Sep 2 23:51:42 rtn66u daemon.info pixelserv[21810]: 7751 req, 0 err, 1082 tmo, 10 cls, 0 nou, 0 pth, 301 nfe, 3413 ufe, 52 gif, 25 bad, 0 txt, 10 jpg, 2 png, 43 swf, 5 ico, 2433 ssl, 41 sta, 35 stt, 299 rdr
    ...
    Sep 3 20:30:01 rtn66u daemon.info pixelserv[4775]: 1935 req, 0 err, 8 tmo, 226 cls, 0 nou, 0 pth, 112 nfe, 749 ufe, 12 gif, 8 bad, 0 txt, 0 jpg, 0 png, 11 swf, 1 ico, 653 ssl, 105 sta, 12 stt, 38 rdr
    Sep 4 20:30:02 rtn66u daemon.info pixelserv[4775]: 4660 req, 0 err, 8 tmo, 586 cls, 0 nou, 0 pth, 313 nfe, 1698 ufe, 48 gif, 72 bad, 0 txt, 0 jpg, 2 png, 48 swf, 1 ico, 1638 ssl, 127 sta, 12 stt, 107 rdr
    So pixelserv detects a connection but never gets a message content, so doesn't reply and one end or other closes the connection. I will experiment some more.
     
  60. HunterZ

    HunterZ LI Guru Member

    Interesting.

    I caught and fixed that syslog a few days ago but didn't think it was important enough to roll a new release for until someone complained. I'll try to put one out tonight.
     
  61. mstombs

    mstombs Network Guru Member

    I ran wireshark on my VM and caught a few examples, which I think confirm all is OK. I don't use IPV6 but I know it is creeping in and enabled by default in various places.

    I can see what looks like a zero length broadcast IPV6 SYN request, on my test port No (8080 = http-alt), which gets immediately rejected

    Code:
    925    670.603908000    ::1    ::1    TCP    96    46572 > http-alt [SYN] Seq=0 Win=43690 Len=0 MSS=65476 SACK_PERM=1 TSval=981661 TSecr=0 WS=128
    926    670.603937000    ::1    ::1    TCP    76    http-alt > 46572 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
    But pixelserv still gets informed of a connection attempt, but no data. I think this is an example of a known Linux bug referred to in here:-

    http://linux.die.net/man/2/select

    So timing out, or getting no data is nothing to be worried about!
     
  62. HunterZ

    HunterZ LI Guru Member

    I guess that's okay since you've got it forking for each connection.
     
  63. HunterZ

    HunterZ LI Guru Member

    Internet is down. Will try to get an update out tomorrow.

    Any thoughts on using a pipe to report data from the child processes back to the main one, instead of using signals? I'm thinking this would allow me to show average request size as part of the stats reporting to help with tuning buffer sizes and such.
     
  64. mstombs

    mstombs Network Guru Member

    Original example code had the fork, I did try removing it which works fine until it doesn't, now understand more why. It seems to me that google Chrome is the source of these requests, I wonder if it is trying to identify/characterize the webserver it is communicating with, real webservers do have extra information in html headers, and like the favicon.ico request keeps requesting till its get one it can cache!

    Sorry don't know about pipes, but you need to handle SIGCHLD or the finished forked process gets stuck as a zombie, that was in the original example code. Re sizes - we currently don't necessarily read the whole request, possibly only get what was in the first tcp request, not subsequent or reconstructed fragments - because the GET file should be in the first few chars! Before closing the channel you need to read anything left in the channel, as closing a non empty channel results in a RST being sent. The original pixelserv.pl http://proxytunnel.sourceforge.net/files/pixelserv.pl.txt does all the reading first before replying, but I found you still needed to do it again just before closing. If the data is bigger than the CHAR_BUF_SIZE, need to do multiple reads to find out how big.
     
  65. HunterZ

    HunterZ LI Guru Member

    Well forking lets you move on to handling the next connection request *while* the current one is still being processed, because child processes can run concurrently. Not forking means that requests are handled sequentially, which has a much higher risk of clients timing out while waiting for someone else's request to be serviced.

    Pipes let you write data from a child process to a data stream that can then be read by the parent (or vice versa). I'm thinking the forked process could sum up the amount of data read and then write it to the pipe, and the parent could read it and factor it into a running average. If needed, response type could also be reported this way instead of via exit codes, but I don't think there's any special advantage over the signal handler method other than simplicity.
     
    koitsu likes this.
  66. koitsu

    koitsu Network Guru Member

    Speaking generally regarding parent and child and them "sharing information": I have to assume by "pipe" you're referring to a named pipe, as in a fifo (ex. mkfifo / mknod). If not (thus referring to an anonymous pipe, i.e. |), I don't see how what would work.

    A better solution is to simply use mmap() on a file in /tmp used for statistics tracking. The reason I mention that rather than POSIX shm is because the latter would require people mount /dev/shm as tmpfs (RAM) and it's just more administrative overhead, plus mmap() is easier to use. pixelserv is already in C, so it's an issue of writing the code.
     
  67. HunterZ

    HunterZ LI Guru Member

    I'm talking about using a C pipe(), which creates a file descriptor that one process can write() to and the other process can read() from: http://linux.die.net/man/2/pipe

    It should be a simple enough use case, since:
    • write() calls are atomic (thus making it safe for multiple children to write)
    • only the parent process will be reading
    • the record size can be a fixed number of bytes
     
  68. HunterZ

    HunterZ LI Guru Member

    Here's V35.HZ8 to fix the closed connection syslog spam, as promised.
     

    Attached Files:

  69. Toink

    Toink Network Guru Member

    Thanks, HunterZ!

    May I ask if your latest version works specifically for certain routers only?
    I just tried it in my E3000 and I'm getting this error: ADBLOCK ERROR: cannot start pixelserv
     
  70. HunterZ

    HunterZ LI Guru Member

    No idea, it's compiled the same as all the others and works on my RT-N66U. Maybe try running from an SSH or telnet session to see if you get an error?
     
  71. leandroong

    leandroong Addicted to LI Member

    @HunterZ, Thanks. Here is my initial run (static)
    /media/optware/adblock # pixelserv -help
    -sh: pixelserv: not found
    /media/optware/adblock # ./pixelserv -help
    Usage:./pixelserv [IP No/hostname (all)] [-n i/f (all)] [-o select_timeout (10 seconds)] [-p port (80) & (443)] [-r (enables redirect to encoded path in tracker links)] [-s /relative_stats_html_URL (/servstats) [-t /relative_stats_txt_URL (/servstats.txt) [-u user ("nobody")]
    /media/optware/adblock # wget -S -O - http://10.0.1.254:/servstats
    --2014-09-06 13:39:22-- http://10.0.1.254/servstats
    Connecting to 10.0.1.254:80... connected.
    HTTP request sent, awaiting response...
    HTTP/1.1 200 OK
    Content-type: text/html
    Content-length: 322
    Connection: close
    Length: 322 [text/html]
    Saving to: 'STDOUT'

    0% [ ] 0 --.-K/s <!DOCTYPE html><html><head><title>nullserv statistics</title></head><body>/media/optware/adblock/pixelserv version: V35.HZ7 compiled: Sep 2 2014 19:19:56 from pixelserv.c<br>2 req, 0 err, 0 tmo, 0 cls, 0 nou, 0 pth, 0 nfe, 0 ufe, 0 gif, 0 bad, 0 txt, 0 jpg, 0 png, 0 swf, 0 ico, 0 ssl, 2 sta, 0 stt, 0 rdr</body></html>
    100%[======================================>] 322 --.-K/s in 0s

    2014-09-06 13:39:22 (4.88 MB/s) - written to stdout [322/322]

    /media/optware/adblock # /media/optware/adblock/pixelserv 10.0.1.254 -r
    pixelserv[5383]: /media/optware/adblock/pixelserv version: V35.HZ8 compiled: Sep 5 2014 20:05:40 from pixelserv.c
    /media/optware/adblock #

    /media/optware/adblock # sh adblock.sh
    ADBLOCK: Download starting
    ADBLOCK: Unchanged: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext (Last-Modified: Fri, 29 Aug 2014 08:52:11 GMT)
    ADBLOCK: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Tue, 02 Sep 2014 09:11:50 GMT)
    read(net): Connection reset by peer
    ADBLOCK: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Wed, 20 Aug 2014 18:15:38 GMT)
    ADBLOCK: Filters unchanged
    ADBLOCK: Exiting
    /media/optware/adblock #
     
    Last edited: Sep 6, 2014
  72. HunterZ

    HunterZ LI Guru Member

    This just means that the current directory is not in your shell path.

    Cool, it runs!

    Looks like a good stats response, but I'm confused how you got one at this point, since you didn't run pixelserv with valid parameters yet?

    Okay, now it's running for real.

    Assuming you have adblock.sh configured to redirect to 10.0.1.254, and dnsmasq configured to read adblock.sh's blocklist, you should be in business.
     
  73. mstombs

    mstombs Network Guru Member

    Thanks, with timeout 20 I had quite a few cls, restarted with default 10

    Code:
    Sep 6 11:12:33 rtn66u daemon.err pixelserv[31381]: client closed connection without sending any data
    Sep 6 11:12:43 rtn66u daemon.err pixelserv[31384]: client closed connection without sending any data
    Sep 6 11:12:43 rtn66u daemon.err pixelserv[31385]: client closed connection without sending any data
    Sep 6 11:12:43 rtn66u daemon.err pixelserv[31382]: client closed connection without sending any data
    Sep 6 11:12:43 rtn66u daemon.err pixelserv[31383]: client closed connection without sending any data
    Sep 6 11:20:22 rtn66u daemon.err pixelserv[31437]: client closed connection without sending any data
    Sep 6 11:20:22 rtn66u daemon.err pixelserv[31438]: client closed connection without sending any data
    Sep 6 11:20:22 rtn66u daemon.err pixelserv[31439]: client closed connection without sending any data
    Sep 6 11:21:52 rtn66u daemon.err pixelserv[31458]: client closed connection without sending any data
    Sep 6 11:21:52 rtn66u daemon.err pixelserv[31459]: client closed connection without sending any data
    Sep 6 11:26:13 rtn66u daemon.info pixelserv[4775]: 10690 req, 4 err, 8 tmo, 1419 cls, 0 nou, 0 pth, 550 nfe, 4388 ufe, 113 gif, 102 bad, 0 txt, 0 jpg, 4 png, 133 swf, 1 ico, 3479 ssl, 148 sta, 12 stt, 329 rdr
    Sep 6 11:26:13 rtn66u daemon.notice pixelserv[4775]: exit on SIGTERM
    Sep 6 11:26:45 rtn66u daemon.info pixelserv[31742]: /mnt/usb4gb/pixelserv version: V35.HZ8 compiled: Sep 5 2014 20:05:39 from pixelserv.c
     
  74. HunterZ

    HunterZ LI Guru Member

    Got pipe()/read()/write() working to send the browser request sizes from the request handler child processes to the main/parent process. This is used to calculate/track running/moving average and max request sizes and report them as stats.

    I saw a max request size of 2.5KB, so I increased the default buffer size to 4KB (the next greater power of 2).

    Looks like Github lets you post binaries after all, so here are V35.HZ9 binaries with the pipe additions: https://github.com/HunterZ/pixelserv/releases/tag/V35.HZ9
     
    pharma and Goggy like this.
  75. Almaz

    Almaz Serious Server Member


    Just tested your binary with Shibby fw E3000. It worked fine for about half an hour and then I openned pixelserv:443 and took about 5 seconds to open. Tested later again and it openned fine, could be the page was already precached. Later again tried it and it took about 5 seconds to open the page. No errors, it works fine but it takes time to open. Put back v34 and it works fine.
     
  76. leandroong

    leandroong Addicted to LI Member

    Very slow loading of youtube main page.
     
  77. HunterZ

    HunterZ LI Guru Member

    Can you post a stats dump from a slow session?

    It's possible that the blocking pipe read is taking a long time to complete for some reason. I could use a non-blocking mode, but it will likely miss the child writes on the loop pass in which they are written and then catch up on the next pass. This is probably preferable to lower performance though.

    It would be cleaner to rewrite the whole thing in C++ with Boost.Asio, but the binary would probably end up a lot bigger.

    Edit: please also check syslog for error messages.
     
  78. leandroong

    leandroong Addicted to LI Member

    How to perform stats dump?
    you mean this
    /opt/home/admin # wget -S -O - http://10.0.1.254:/servstats
    --2014-09-07 23:05:19-- http://10.0.1.254/servstats
    Connecting to 10.0.1.254:80... connected.
    HTTP request sent, awaiting response...
    HTTP/1.1 200 OK
    Content-type: text/html
    Content-length: 177
    Connection: close
    Length: 177 [text/html]
    Saving to: 'STDOUT'

    0% [ ] 0 --.-K/s <!DOCTYPE html><html><head><title>nullserv statistics</title></head><body>194 req, 1 err, 0 gif, 0 bad, 174 txt, 0 jpg, 0 png, 0 swf, 0 ico, 0 ssl, 1 sta, 18 rdr</body></html>
    100%[======================================>] 177 --.-K/s in 0s

    2014-09-07 23:05:19 (2.49 MB/s) - written to stdout [177/177]


    edit2: no error from syslog
     

    Attached Files:

    Last edited: Sep 7, 2014
  79. HunterZ

    HunterZ LI Guru Member

    Open browser to /servstats on pixelserv IP (e.g. http://192.168.1.254/servstats).

    I'm thinking that I could change the main select() call to also monitor the pipe's file descriptor, but then it would only work in MULTIPORT mode (not an issue for Tomato non-tiny builds). This would allow decoupling of pipe processing from socket processing, allowing them to occur simultaneously and in a non-blocking manner.
     
  80. HunterZ

    HunterZ LI Guru Member

    Oops, I took too long to reply and you figured it out.

    I'm concerned that you have a nonzero err count that wasn't reported in the syslog. I thought I caught them all.

    Update: I'm breaking off the child process connection processing into a separate function in a separate file in order to clean things up a bit. After that I will look into having select() monitor both sockets and pipes and then use non-blocking pipe read()s to process incoming data on the latter.
     
    Last edited: Sep 8, 2014
  81. AndreDVJ

    AndreDVJ Addicted to LI Member


    Try my binaries. I use the router itself to compile the source.
    Code:
    pixelserv[15856]: /mnt/storage/adblock/pixelserv version: V35.HZ8 compiled: Sep  7 2014 23:19:28 from pixelserv.c
     

    Attached Files:

    Toink likes this.
  82. Almaz

    Almaz Serious Server Member


    /var/wwwext/pixelserv version: V35.HZ9 compiled: Sep 6 2014 14:38:54 from pixelserv.c
    158 req, 638 avg, 2020 rmx, 0 err, 14 tmo, 44 cls, 0 nou, 0 pth, 15 nfe, 59 ufe, 4 gif, 0 bad, 0 txt, 1 jpg, 0 png, 0 swf, 3 ico, 12 ssl, 5 sta, 0 stt, 0 rdr
     
  83. HunterZ

    HunterZ LI Guru Member

    It's possible that something is going wrong for the timeout or closed connection cases, but I'm too far along with HZ10 to back up and troubleshoot, so I would recommend sticking with HZ8 for now if you are having problems.
     
  84. Almaz

    Almaz Serious Server Member

    I'm using v34 & v32 and they are working great. Is there any advantages using hz8?
     
  85. HunterZ

    HunterZ LI Guru Member

    More features, mostly:
    • Can redirect the browser around sponsor/tracking links (e.g. sponsored links in google search results) instead of dead-ending it, if you specify the -r option when launching.
    • Provides blank .ico files on request (for favicon requests on blocked sites).
    • Can report stats at will via the browser, in text or html formats.
    • Can specify timeout time via a command line parameter.
    • I've also cleaned the code up here and there.

    HZ10 should be a significant code cleanup with the ability to report average and maximum request size as part of the stats. I'm not sure yet whether it will significantly change the size of the binary one way or the other, though, since I've split it apart into 3 .c files (pixelserv.c, util.c, socket_handler.c) and 3 .h files (util.h, socket_handler.h, responses.h - although I may merge the last one into socket_handler.c) so far.
     
  86. Almaz

    Almaz Serious Server Member

    HunterZ, can you explain the option of -r? What's the difference just using a standard way vs redirection to something else?
     
  87. HunterZ

    HunterZ LI Guru Member

    Well, here's a silly test case:

    Search for Home Depot on google.com. Note that the first result says "Ad" and has a URL of http://www.google.com/aclk?sa=l&ai=CqvjY4fUNVMyvH4mE-gO3wYGgAsynzowFpNWAwCO148kMCAAQAVCyoeawBmDJ_sSN9KTQGaAB7qjj_gPIAQGqBCBP0KLooo71iF41RSWsd24EDctrYKpBNcNAautV2IGpfboFEwiIot2ooNLAAhXKmogKHRYWACnKBQCAB_rWnAGQBwGoB6a-Gw&ei=4fUNVMitG8q1ogSWrIDIAg&sig=AOD64_0vFNy7hn5YxawdBA0knVgtK5xmOw&rct=j&q=&sqi=2&ved=0CB4Q0Qw&adurl=http://clickserve.dartsearch.net/link/click?lid=43700003817116349&ds_s_kwgid=58700000047538642&ds_e_adid=9436124484&ds_e_matchtype=search&ds_url_v=2&ds_dest_url=http://www.homedepot.com/?cm_mmc=SEM|G|BT1&cad=rja

    If you click on it without the -r feature active, you will likely dead end on a blank clickserve.dartsearch.net page. Using the -r option, however, it will pick up and decode the homedepot.com URL at the end and return a response that directs the browser to go there.

    Edit: I should mention that I did not implement the redirect feature. Instead, I merged it from the h0tw1r3 fork on Github back into the mstombs fork here, to make a new fork of my own (the HZ series).
     
  88. Almaz

    Almaz Serious Server Member

    That is an awesome feature. Now, what's the most stable pixelserv for mips with -r option?
     
  89. HunterZ

    HunterZ LI Guru Member

    HZ8 is the one you want to try, as people are reporting issues with HZ9 (the newest release).
     
  90. leandroong

    leandroong Addicted to LI Member

    @HunterZ, i'm on HZ9. No complain except for opening youtube main page, https://www.youtube.com/.
    Too slow loading but will eventually load, about 1 min waiting.
     
  91. HunterZ

    HunterZ LI Guru Member

    Yeah, I would recommend rolling back to HZ8 until HZ10 is released, but of course it's up to you.
     
  92. leandroong

    leandroong Addicted to LI Member

    Same issue. Will stick at HZ9
     
  93. Beast

    Beast Network Guru Member

    Im using HZ9 no delay to youtube, but I am wondering how can you tell if the -r switch is active or not?
    I added the switch in the adblock script:

    # additional options for pixelserv
    PIXEL_OPTS="-r"

    Or should it go in the adblock config file. I ask because I ended up at some blank page using your example address above (Home Depot).
     
  94. leandroong

    leandroong Addicted to LI Member

    Not blank page for HZ8. Youtube has no problem on android and IOS but with win7.
    ps w, will show you
    Code:
    20016 nobody    1176 S    /media/optware/adblock/pixelserv 10.0.1.254 -r
    
     
  95. Toink

    Toink Network Guru Member

    Thank you! There are two files in there one for dynamic and another for static. Which should I use for which?
     
  96. HunterZ

    HunterZ LI Guru Member

    If dynamic runs for you, use it (it will be smaller). If it gives you problems, use static.

    The difference is that dynamic requires libraries to be installed on the router (which usually are), while static has them baked into the binary.
     
    Toink likes this.
  97. jerrm

    jerrm Network Guru Member

    Either will do. If on Tomato, dynamic should be fine. If on another firmware use static.
     
    Toink likes this.
  98. Toink

    Toink Network Guru Member

  99. Almaz

    Almaz Serious Server Member

    Would it be possible to redirect port 443 to something else? I'm using OpenVPN on 443 port. I don't VPN as much as I used to before but still would love to find out a way to be able to run new Pixelserv with OpenVPN on port 443. The only way I was able to use OpenVPN with pixelserv v32 because of DNAT from 443 to 80.
     
  100. HunterZ

    HunterZ LI Guru Member

    I'm not sure I understand the question.

    pixelserv listens on its own IP (e.g. 192.168.1.254), and should not interfere with anything else that is going on on other destination IPs.

    At any rate, the port(s) on which pixelserv listens are fully configurable via the command line. If you don't specify any -p parameters, it will listen on 80 and 443 by default, otherwise it will only listen on port(s) specified via -p parameters.
     

Share This Page