pixelserv compiled to run on router WRT54G

Discussion in 'Tomato Firmware' started by Jedis, Sep 5, 2009.

  1. HunterZ

    HunterZ Network Guru Member

    I don't expect the 'bad' counter to get many hits now, as I broke out POST requests into their own 'pst' counter.
     
  2. The Master

    The Master Network Guru Member

    All OK @ R7000...

     
    WaLLy3K likes this.
  3. mstombs

    mstombs Network Guru Member

    Re hangs (that I haven't yet seen!). Previous versions of this code could appear to hang if the interface mode was used "-n br0" AND a gui change was made which rebuilds the lan bridge. I deduced pixelserv was left listening on an obsolete socket, workaround is to always restart pixelserv in firewall script, but watch-out tomato does have habit of calling the firewall script many times in quick succession. The interface mode is now only invoked if specifically called in command line (no default). I wonder if there are other ways for the main listening socket to become invalid?
     
  4. remlei

    remlei Networkin' Nut Member

    let's see what will happen

     
  5. HunterZ

    HunterZ Network Guru Member

    I instrumented the request handler code with timing checks, as well as adding stat counters to capture average and max time taken to process requests.

    This allowed me to determine that the code that drains the read queue at the end was frequently taking a long time waiting for select() to return. I replaced that implementation with non-blocking recv() calls so that it will instead just try reading until there's nothing left and then immediately stop. This seems to be much more efficient.

    I modified the timing checks to now only occur if you specify a -w parameter with a value greater than 0 milliseconds. If you experience hangs, I would highly recommend enabling this option with a value of somewhere in the range of 1000 to 5000 milliseconds and monitoring the syslog for warnings. This will help me figure out where things may be getting stuck.

    HZ11WIP8
    - add average (tav) and maximum (tmx) request processing time stats
    - add -w option to enable warning when request processing time exceeds specified value in milliseconds
    - decrease request processing time by using a non-blocking recv() instead of select() + blocking recv() to empty the socket read queue
    - code formatting tweaks

    EDIT: Just realized it is reporting time exceeded warnings when initial select() times out (tmo condition), with a misleading condition (response selection). This will be fixed in the next version.
     

    Attached Files:

    Last edited: Nov 9, 2014
  6. remlei

    remlei Networkin' Nut Member

    reviewed the syslogs and I see this

    and it floods the syslog for about, maybe 12 times?
     
  7. HunterZ

    HunterZ Network Guru Member

    Cool! If you don't get any crashes then I may make WIP8 an official HZ11 release.
     
  8. HunterZ

    HunterZ Network Guru Member

    Looks like someone on your LAN is sending garbage at pixelserv...
     
  9. AndreDVJ

    AndreDVJ LI Guru Member

    HZ11 WIP8 for some reason spawns many pixelserv processes after opening few websites, CPU usage goes to 100%, router becomes a slowpoke.
    Code:
    pixelserv[25565]: ./pixelserv version: V35.HZ11WIP8 compiled: Nov  9 2014 15:25:10
    root@WNR3500L:/tmp/mnt/storage/adblock# ./adblock.sh restart
    ADBLOCK[25567]: Running as /mnt/storage/adblock/adblock.sh restart
    ADBLOCK[25567]: Using config file /mnt/storage/adblock/adblock.ini
    ADBLOCK[25567]: Requested list mode is OPTIMIZE
    ADBLOCK[25567]: Enabling dnsmasq logging
    ADBLOCK[25567]: Logging previously enabled
    ADBLOCK[25567]: Logging to /mnt/storage/logs/dnsmasq.log
    ADBLOCK[25567]: Creating web link /www/user/adblock.sh
    ADBLOCK[25567]: Stopping
    ADBLOCK[25567]: Restarting dnsmasq
    Done.
    ADBLOCK[25567]: Download starting
    ADBLOCK[25567]: Unchanged: http://www.malwaredomainlist.com/hostslist/hosts.txt (Last-Modified: Sun, 09 Nov 2014 17:50:37 GMT)
    ADBLOCK[25567]: Unchanged: http://winhelp2002.mvps.org/hosts.txt (Last-Modified: Wed, 01 Oct 2014 15:44:48 GMT)
    ADBLOCK[25567]: Filters unchanged
    ADBLOCK[25567]: Setting up 192.168.1.254 on br0:adblk
    ADBLOCK[25567]: Setting up pixelserv on 192.168.1.254
    pixelserv[25850]: /mnt/storage/adblock/pixelserv version: V35.HZ11WIP8 compiled: Nov  9 2014 15:25:10
    iptables: Chain already exists
    ADBLOCK[25567]: Writing File /etc/dnsmasq.custom
    ADBLOCK[25567]: Restarting dnsmasq
    ..
    Done.
    ADBLOCK[25567]: Exiting /mnt/storage/adblock/adblock.sh 0
    root@WNR3500L:/tmp/mnt/storage/adblock# ps | grep pixelserv
    25851 nobody  432 S  /mnt/storage/adblock/pixelserv 192.168.1.254
    25931 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25932 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25934 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25936 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25937 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25938 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25940 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25941 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25943 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25944 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25945 nobody  432 R  /mnt/storage/adblock/pixelserv 192.168.1.254
    25947 root  2508 S  grep pixelserv
    WIP7 I have to check its behavior but it was working fine.
     
  10. HunterZ

    HunterZ Network Guru Member

    Hmm, yes I'm seeing that too. Thanks for pointing it out, I'll investigate.

    ...and of course I can no longer reproduce it. I'm also having trouble debugging forks in GDB; I can't attach to them after they're created, and telling GDB to auto-attach as they spawn causes them to spawn paused so that I have to switch to and unpause each one individually.

    I'm wondering if it's time to switch from a fork() based concurrency model to a POSIX threads based one. This would make it easier to debug and probably also more efficient. I think I've got things segregated cleanly enough now that it shouldn't be *too* painful.

    The other thing I've been thinking of is coming up with some kind of watchdog mechanism to kill off a socket handler if it's been running for too long. This may act as a blunt instrument to recover from hangs.
     
    Last edited: Nov 10, 2014
  11. mstombs

    mstombs Network Guru Member

    I wouldn't dismiss that so easily - isn't the euro symbol 0x80 a TCP continuation marker? Could mean continuation packet of large post request is being interpreted as new request, because previous connection closed too quickly? Need to capture traffic in wireshark to see full context of traffic.

    Better to ensure nothing in socket handler can BLOCK forever? ISTR thinking that replies are always very short so the send would never block - but the redirect code can create arbitrarily long messages, Beej's guide warns that it up to the sender program to ensure all parts of a long message are sent

    http://beej.us/guide/bgnet/output/html/multipage/syscalls.html#sendrecv
     
  12. HunterZ

    HunterZ Network Guru Member

    If so then we should probably look for and discard that case (or maybe send a 501 reply?).

    Send shouldn't block forever. Worst case it should return that it wasn't able to send everything.

    This case is already caught with the following code, and nobody has reported seeing the error message:
    Code:
    } else if (rv != rsize) {
    syslog(LOG_WARNING, "send() reported only %d of %d bytes sent; status=%d", rv, rsize, pipedata.status);
    }
    
     
  13. mstombs

    mstombs Network Guru Member

    You already send a 501 for anything other than GET - I see you have just added the message, so its probably just the log message that is new!

    The code has this comment on the send call

    Code:
    // send response
    // this is currently a blocking call, so zero should not be returned
    rv = send(new_fd, response, rsize, MSG_NOSIGNAL);
    
    But reading this, it does seem it will always return, so its not really a blocking call?
    http://beej.us/guide/bgnet/output/html/multipage/advanced.html#sendall
     
  14. HunterZ

    HunterZ Network Guru Member

    It looks like the log message was added in HZ11WIP5.

    Blocking doesn't mean it doesn't return. In this case, it means that it will keep trying until it sends everything, or until the connection goes away, or whatever.

    That shouldn't be a problem though, because the parent doesn't care what the fork() children are doing (unless something comes in on the pipe, or if an exited child process is ready to be reaped).
     
  15. HunterZ

    HunterZ Network Guru Member

    After letting it run over night, there were a half dozen pixelserv child processes running on my router.

    As previously mentioned, GDB seems to be no help with debugging the issue.

    As an alternative, I've instrumented socket_handler.c with some code to periodically record line numbers, plus a child signal handler that reports the last recorded line number to the syslog on a term or usr2 signal. This should allow me to poke the children to get a general idea of where they might be stuck.

    Now I just need to let it run for another ~24 hours to accumulate some more stuck child processes...

    Update: They're all getting stuck on this block of code:
    Code:
      do {
      // use non-blocking reads to avoid the need to call select()
      rv = recv(new_fd, buf, CHAR_BUF_SIZE, MSG_DONTWAIT);
      } while (rv >= 0);
    Looks like I misread the man page for recv(). I should change the loop condition to while(rv > 0) because a value of zero means that the peer has performed an orderly shutdown.

    I'll try to push out a new WIP build tomorrow night.
     
    Last edited: Nov 11, 2014
  16. kartun

    kartun Reformed Router Member

    Thanks. It starts on my RT-AC68U. But I can't manage to get it working ...[​IMG]
    I moved router webinterface to 8080, but still I can't get to pixelserv not on 192.168.1.63, not on 192.168.1.1. And I barely understand the setting in the script for last octet. If I try to start it on 254, it doesn't spawn at all.
    Any ideas what I've done wrong ?
     

    Attached Files:

  17. WaLLy3K

    WaLLy3K Networkin' Nut Member

    My knowledge of subnetting isn't up to scratch, but I believe with your LAN subnet mask set to 255.255.255.192, you're limiting your subnet to 62 hosts (the 63rd being your broadcast address, similar to 192.168.1.255).

    Either change the netmask to 255.255.255.0 and leave the pixelserv last octet of adblock.sh to 254 (default), or keep 255.255.255.192 and change the pixelserv last octet of adblock.sh to 62.
     
  18. kartun

    kartun Reformed Router Member

    Thanks. It really helped. I would play with it a little bit and post info if required

    What is the format for whitelist? For some reason it somehow blocked "mail.yandex.ru", so I can't get to my emails =/
     
    Last edited: Nov 12, 2014
  19. HunterZ

    HunterZ Network Guru Member

    Sorry for the delay, nature clearly didn't want me to work on pixelserv because I got sick and lost power at home yesterday evening due to a windstorm.

    Note that the debug version now contains the elapsed time warning and line number reporting on SIGUSR1 features, while the normal version does not. The normal version still collects statistics on connection handling time, however.

    The line number reporting feature is quite helpful. If you see a hung pixelserv child process, you can do a kill -usr1 on its pid to get a general idea of the line number where it might be stuck.

    HZ11WIP9
    - Add new DEBUG #define, used in debug build
    - Remove commented out stuff from build.sh
    - Only include time check code (including -w option) in debug build
    - Debug only: Record current line number in various places in socket handler code, and report current value to syslog on SIGUSR1
    - Convert time check code into a macro to avoid code duplication
    - Exit final recv() loop on rv=0 (fixes hung child process bug)
    - Minor code cleanup
     

    Attached Files:

  20. AndreDVJ

    AndreDVJ LI Guru Member

    Complied / testing already. I'll check tomorrow how things are going.
     
  21. HunterZ

    HunterZ Network Guru Member

  22. HunterZ

    HunterZ Network Guru Member

    Update: Haven't really tested it yet, but this should work better. I set a receive timeout in socket_handler() and removed select().

    Also forgot to mention in my changelog list below that if the initial recv() call returns a terminal error, the final recv() loop will be skipped. This should mean that no connection should take significantly longer than the timeout time to process.

    HZ11WIP10
    - set a global recv timeout on the socket connection
    - remove all traces of select()
    - made error handling consistent for initial and final recv calls
    - made logging messages consistent
     

    Attached Files:

  23. superdos

    superdos Networkin' Nut Member

    Here's a compiled version for Asus RT-AC68U
     

    Attached Files:

    HunterZ, The Master and kartun like this.
  24. mstombs

    mstombs Network Guru Member

    Last edited: Nov 13, 2014
  25. HunterZ

    HunterZ Network Guru Member

    @mstombs I think it will block until it gets acknowledged by the client (this is TCP after all) or until the connection blows up. One of those two things is probably guaranteed to happen, but I'm not sure if there's an upper bound on the time that they may take to occur.

    Returning immediately after the message is passed to the kernel buffer only occurs for non-blocking writes. We use blocking writes because they're easier to deal with (otherwise you may get interrupted in the middle and have to pick up where you left off), and the only advantage I can think of of using non-blocking would be to allow us to save a millisecond or two by doing the final recv() calls at the same time that the response is being sent.


    If anyone still suspects that they are seeing hangs due to blocking calls in WIP10, we should now be able to narrow it down using the debug version's line number and/or elapsed time warning features.


    Also, to clarify my previous post: select() was only removed from socket_handler. It is still in main() in pixelserv.c, as it is very useful there since we are monitoring for I/O on multiple file descriptors.
     
  26. AndreDVJ

    AndreDVJ LI Guru Member

    in WIP9 I saw one child process hung...

    Well, my test when pixelserv hangs or something is wrong with it is a e-commerce site which gets badly broken when pixelserv hangs up:
    Code:
    www.kabum.com.br
    I am testing the non-debug version as of now. In matter of hours this website gets broken. If does, I'll run the debug version and if anything is printed on the syslog I'll post here. That will take one more day.

    EDIT:

    The website mentioned did hang... Though instead of hanging up forever and all websites become unresponsive until I terminate/restart pixelserv, looks like when the request hangs, until it times out you can't retrieve pixelserv stats.
    The browser hangs trying to get either the html or txt version. I forgot to take a screenshot of that event. I noticed something wrong when I tried to access Guru3d website.

    Here's the pixelserv stats:
    Code:
    /mnt/storage/adblock/pixelserv version: V35.HZ11WIP10 compiled: Nov 13 2014 17:40:46
    17694 uts, 487 req, 649 avg, 3704 rmx, 290 tav, 10019 tmx, 0 err, 5 tmo, 42 cls, 0 nou, 0 pth, 34 nfe, 37 ufe, 5 gif, 0 bad, 184 txt, 0 jpg, 0 png, 0 swf, 3 ico, 80 ssl, 2 sta, 17 stt, 0 204, 72 rdr, 6 pst
    When it finally times out, websites start to work properly, finishing loading their pages. I mostly use Internet Explorer, though it hangs up on Chrome as well.

    I'll run the debug version, and will let you know what happens when pixelserv hangs.
     
    Last edited: Nov 14, 2014
  27. HunterZ

    HunterZ Network Guru Member

    When it is hung do you see multiple pixelserv processes running at once?

    Anything in the syslog?
     
  28. leandroong

    leandroong LI Guru Member

    I don't know if it is hung or what, working well on other sites but intermittent on person.com. I see adveertisement and stop it using firefox adblock plus

    Edit: to see the advertisement, you need to member first, free. and click refresh button several times and advertisement will eventually show up.

    edit2: advertisement come and go. Appear and disappear on refresh. Latly, this is adult dating site, adults only.

    edit3: Strange, no advertisement re-appear. Actually, I just updated the pixelserv to HZ11WIP10. Will observe this site and report again if it re-appears.

    edit4: advertisement shows up again on this site.
    edit5: That site keeps generating new ads site. Blacklist addition prevented this.
     
    Last edited: Nov 14, 2014
  29. jerrm

    jerrm Network Guru Member

    pixelserv has absolutely nothing to do with what ads you see.
     
  30. leandroong

    leandroong LI Guru Member

    It does and this is what "BLACKLIST" in config is use.
     
  31. jerrm

    jerrm Network Guru Member

    No. This is the pixelserv thread, not the adblock thread. Whether an ad is blocked or not is not relevant here.

    Adblock blocking is not dependent on pixelserv, but pixelserv greatly improves the experience.

    If a query is redirected to pixelserv and causes some unexpected response, hang, or other trouble, then it is relevant to this thread.
     
  32. HunterZ

    HunterZ Network Guru Member

    jerrm is correct. It's dnsmasq's blocklist (as generated by adblock.sh) that determines which ads get blocked by redirecting them to another IP address (usually pixelserv's), so the ad will not show up regardless of whether pixelserv responds. The purpose of pixelserv is to give a quick response so that the browser will not have to wait a long time for the redirected connection attempt to time out.
     
  33. AndreDVJ

    AndreDVJ LI Guru Member

    I had no luck so far, tried to run the debug both daemonized or not (-f), with gdb and without, can't make pixelserv throw me a bone, unless I'm doing it wrong.

    The best I got and not sure if related or not, was this:
    Code:
    Nov 14 08:28:53 WNR3500L daemon.warn pixelserv[1923]: write() to pipe reported error: Broken pipe
    Nov 14 08:28:53 WNR3500L daemon.warn pixelserv[1925]: write() to pipe reported error: Broken pipe
    Nov 14 08:28:57 WNR3500L daemon.warn pixelserv[1927]: write() to pipe reported error: Broken pipe
    Nov 14 08:28:57 WNR3500L daemon.warn pixelserv[1928]: write() to pipe reported error: Broken pipe
    Nov 14 08:28:57 WNR3500L daemon.warn pixelserv[1929]: write() to pipe reported error: Broken pipe
    And this:
    Code:
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3018]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3019]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3020]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3021]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3022]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3023]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3024]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3025]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3026]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3027]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3028]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3029]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3030]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3031]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3032]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3033]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3034]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3035]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3036]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3037]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3038]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3039]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3040]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3041]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3042]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3043]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3044]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3045]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3046]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3047]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3048]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3049]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3050]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3051]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3052]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3053]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3054]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3055]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:48 WNR3500L daemon.warn pixelserv[3056]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3057]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3058]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3059]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3060]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3061]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3062]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3063]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3064]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3065]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3066]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3067]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3068]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3069]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3070]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3071]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3072]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3073]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3074]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3075]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3076]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3077]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3078]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3079]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3080]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3081]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3082]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3083]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3084]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3085]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3086]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3087]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3088]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3089]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
    Nov 14 10:20:49 WNR3500L daemon.warn pixelserv[3090]: Sending HTTP 501 response for unknown method or non-SSL, non-HTTP request: HEAD
     
  34. HunterZ

    HunterZ Network Guru Member

    @AndreDVJ

    The pipe breaking between parent and child process means that something is terribly wrong. That should never, ever happen. Since implementing the pipe was a learning experience for me, I'm not sure what would reasonably cause that.


    It looks like HEAD is something pixelserv should reply to, but doing it correctly is going to be nontrivial because it would mean having to be able to report just the header part of each response type: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

    Do you know what is causing the HEAD requests?
     
  35. AndreDVJ

    AndreDVJ LI Guru Member

    The broken pipe was printed when I killed pixelserv (was SIGTERM, not SIGUSR1). Yesterday I wasn't that well. Killed pixelserv and forgot to report the message in a proper way.

    For the HEAD messages, I was sleeping, and the only thing connected was my cellphone, maybe some AWS server sent weird stuff to pixelserv. I don't keep browsing history on my router, but dnsmasq logs may help on what happened at the same second.
    Code:
    
    Nov 14 10:20:48 dnsmasq[1567]: query[A] eventlogger.soundcloud.com from 192.168.1.200
    Nov 14 10:20:48 dnsmasq[1567]: config eventlogger.soundcloud.com is 192.168.1.254
    Nov 14 10:20:48 dnsmasq[1567]: query[A] analytics.localytics.com from 192.168.1.200
    Nov 14 10:20:48 dnsmasq[1567]: forwarded analytics.localytics.com to 201.82.0.69
    Nov 14 10:20:48 dnsmasq[1567]: query[A] e.crashlytics.com from 192.168.1.200
    Nov 14 10:20:48 dnsmasq[1567]: forwarded e.crashlytics.com to 201.82.0.69
    Nov 14 10:20:48 dnsmasq[1567]: reply analytics.localytics.com is <CNAME>
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 54.225.145.167
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 54.243.32.234
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 174.129.237.210
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 54.83.33.165
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 54.225.153.141
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 54.243.104.107
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 54.225.233.184
    Nov 14 10:20:48 dnsmasq[1567]: reply queuerlb-599065538.us-east-1.elb.amazonaws.com is 184.72.221.196
    Nov 14 10:20:48 dnsmasq[1567]: reply e.crashlytics.com is <CNAME>
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.235.65.239
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.225.183.227
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.235.105.180
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.225.170.242
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.225.199.223
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.235.141.30
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.243.154.81
    Nov 14 10:20:48 dnsmasq[1567]: reply events-endpoint-455714294.us-east-1.elb.amazonaws.com is 54.225.235.202
    EDIT:

    The best I could get was pixelserv taking almost 1 minute to serve a 291-byte txt file
    Code:
    root@WNR3500L:/tmp/home/root# wget 192.168.1.254/servstats.txt
    --2014-11-15 18:47:08--  http://192.168.1.254/servstats.txt
    Connecting to 192.168.1.254:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 291 [text/plain]
    Saving to: 'servstats.txt'
    100%[======================================>] 291  --.-K/s  in 0s
    2014-11-15 18:47:56 (2.98 MB/s) - 'servstats.txt' saved [291/291]
    Whenever pixelserv hangs, that e-commerce website hangs loading its banner, and after the hang is gone, the site finishes loading okay.

    I am also attaching the screenshots so you can see how it looks, and how it should look.

    hang.jpg proper.jpg
     
    Last edited: Nov 15, 2014
  36. HunterZ

    HunterZ Network Guru Member

    HZ11WIP11:
    - Add HEAD to the list of recognized HTTP methods for which an HTTP 501 response is generated
    - Add 'hed' stat counter to track number of HEAD requests

    @AndreDVJ I probably can't help with the hangs until you're able to reproduce with the debug version and send a kill -s usr1 at the subprocesses.

    Interesting that your wget output shows the transfer rate as 2.98MB/s. It sounds like the delay must have been before the transfer started?
     

    Attached Files:

  37. Toink

    Toink Network Guru Member

    Thank you HunterZ! Do you have a version for E3000/E4200 dynamic?
     
  38. HunterZ

    HunterZ Network Guru Member

    @Toink No, I only ship static versions now because they include the newer libraries provided by the Tomatoware toolchain. The non-debug version is UPX-compressed, though, which cuts its size in half.
     
  39. Toink

    Toink Network Guru Member

    I understand. Time to test this on a Tenda W1800R soon then. Thanks!
     
  40. arthur king

    arthur king Networkin' Nut Member

    Hi You Have A Virus In Pixelserv And Not In Pixelserv-Debug
    Checked By Avast Antivirus And Remove.
    I Now Have It Running On My Rooter.
    The Virus In Pixelserv. Threat: ELF:Tsunami-O [Trj]
     
  41. HunterZ

    HunterZ Network Guru Member

    As best I can tell, this is a false detection by Avast that results from the UPX compression.

    Here is a scan of the UPX-compressed pixelserv binary, which shows only Avast detecting the trojan: https://www.virustotal.com/en/file/...ee6c421ec1086af206a222bf/analysis/1416158724/
    Here is a scan of the same binary after I unpack it: https://www.virustotal.com/en/file/...6aea5c2fed2eb67a246fc305/analysis/1416159011/

    I've reported this to Avast as a likely false positive so that they can fix their scanner.
     
  42. arthur king

    arthur king Networkin' Nut Member

    I unpacked it using UPX compressor and I ran the antivirus and had it fixed the file then I recompressed it with upx compressor it was not the compress the file but file itself that had the problem virus were talking about is a UNIX virus that is in Rooters
     
    Last edited: Nov 16, 2014
  43. HunterZ

    HunterZ Network Guru Member

    There is no virus. The virus scanner is incorrect. If you are not comfortable with this, then run the uncompressed or debug version.
     
  44. arthur king

    arthur king Networkin' Nut Member

    I found the problem. I used an old decompress, then a new compressed, after i had remove the problem vire avast antivirus, that was why it showed up as still having the problem. Because i had to update the compressor the compressor was updated so that it now when it decompress from the original the problem disappeared then i did several variations on the way the compressor is set up so your problem with it is not so much the file is bad it's the compression method which compression method method did you use what type of compression i tried was several and i got different results a few came through as being virused sum did not.

    pixelserv > [avast] = bad > [old-upx -d] > [avast] = bad > [avast -fix] = good > [new-upx --brute] = good 124k pixelserv

    pixelserv > [avast] = bad > [new-upx -d] > [avast] = good > [new-upx --brute] = good 124k pixelserv

    sse thank you.

    version upx391w
    upx --brute --preserve-build-id pixelserv
    good 124k
     
    Last edited: Nov 17, 2014
  45. WaLLy3K

    WaLLy3K Networkin' Nut Member

    The problem is that it's a false positive. This thing happens quite often with signature based detection, due to the signature being detected in this case, is UPX.

    Don't worry about it. HunterZ has done the right thing by reporting it to Avast, so hopefully soon your anti virus won't complain about it.
     
    HunterZ likes this.
  46. hammer

    hammer Connected Client Member

    A bit off topic but it compiles and runs fine on CyanogenMod 11 (Android 4.4.4) :) and probably other rooted ARM Android phones...

    Root needs to have "x" permission and if put in /data/local/userinit.d/ on CM11 it will start automatically after a reboot. I had to create the "userinit.d" directory as it did not exist on my phone.

    I did experience that it "hang" on a webpage and /servstats also stopped responding, but it recovered after a while, maybe after one minute or so.

    Android toolchain created like this
    Code:
    PLATFORM_PREFIX=~/android-ext/
    NDK_PATH=~/Downloads/android-ndk-r10c
    NDK_PLATFORM=android-19
    
    $NDK_PATH/build/tools/make-standalone-toolchain.sh --platform=$NDK_PLATFORM --install-dir=$PLATFORM_PREFIX
    
    And build.sh changed to
    Code:
    PREFIX="arm-linux-androideabi-"
    PFIL=pixelserv-$PVER.armeabi.zip
    2014-11-19 19.46.06.png
     

    Attached Files:

    Last edited: Nov 20, 2014
    HunterZ likes this.
  47. HunterZ

    HunterZ Network Guru Member

    Nice, an Android port!

    I suppose I should put time checking code into the main() process too.
     
  48. AndreDVJ

    AndreDVJ LI Guru Member

    Same thing I experience. There's no child process to kill (actually issue USR1 signal), only parent.
     
  49. HunterZ

    HunterZ Network Guru Member

    Okay, I'll get the parent process code instrumented with line number checkpoint reporting so that we can narrow down where it's happening. I may move it to USR2 for both parent and child processes since stats are already reported on USR1.

    I should also add the filename to the line number reporting.
     
  50. HunterZ

    HunterZ Network Guru Member

    Here's an update that will let you see approximately where the main process is hung at by using kill -s USR2 on its pid. The child processes respond on SIGUSR2 instead of SIGUSR1 too.

    Edit: I forgot to mention that this only applies to the debug version. The non-debug version is effectively unchanged.

    HZ11WIP12
    - use Tomatoware upx
    - add line number checkpointing and reporting to main process
    - move line number reporting to SIGUSR2
     

    Attached Files:

  51. AndreDVJ

    AndreDVJ LI Guru Member

    And there you go! From the main process:
    Code:
    Nov 21 21:27:26 WNR3500L daemon.info pixelserv[30174]: Main process caught signal 17 near line number 499 of file pixelserv.c
     
    Last edited: Nov 21, 2014
  52. HunterZ

    HunterZ Network Guru Member

    This suggests that the accept() call is hanging. Any socket gurus know why that might happen?

    I'll look over the code.

    Edit: It could be that the client is dropping the connection sometime between the select and accept calls: http://stackoverflow.com/questions/3444729/using-accept-and-select-at-the-same-time

    I may need to try going non-blocking on the listen sockets so that the accept call doesn't hang waiting for a connection request when the one that was pending has already disappeared.
     
    Last edited: Nov 22, 2014
  53. HunterZ

    HunterZ Network Guru Member

    I've changed the listening sockets to be non-blocking, so that the accept() call will abort immediately if there is no longer a pending connection to accept. Initial tests showed this case occurring, so this should be an improvement.

    HZ11WIP13
    - set non-blocking mode on listening sockets
    - fix potential memory leak by calling freeaddrinfo() once for each getaddrinfo()
    - move logging of listening sockets into socket setup loop (removes the need to loop twice)
    - if client drops connection before it can be accepted, increment the cls stat counter
     

    Attached Files:

    Goggy, hammer and koitsu like this.
  54. mstombs

    mstombs Network Guru Member

    Interesting - that "accept" blocking would definitely cause some of the reported problems - guess its been like that ever since the multiport and select code added (V32?). That users report slowdowns plus recovery, perhaps there is an a finite timeout on some strange connection requests. Then if new connections arrive faster than the timeout, get a queue building up. Still don't know what is making the strange requests, but whatever it is it is effectively DOSing pixelserv, on some folks LANs. I do remember an issue with syn floods with whole of router firmware before - believed to be caused by a restart of a p2p program with 100s of simultaneous connections.
     
  55. WaLLy3K

    WaLLy3K Networkin' Nut Member

    Have all the relevant changes been applied to the non debug version this time? :)

    I'm assuming so!
     

    Attached Files:

    Last edited: Nov 22, 2014
    HunterZ, The Master and superdos like this.
  56. HunterZ

    HunterZ Network Guru Member

    So it would seem, and the reason it eventually recovers is that another connection request eventually comes in. It's probably when it happens on the SSL port that is most noticeable, because that will block even stats requests due to the fact that accept() is port-specific.

    Yes, in fact there were no debug-only changes in WIP13.
     
    Last edited: Nov 23, 2014
    pharma and WaLLy3K like this.
  57. AndreDVJ

    AndreDVJ LI Guru Member

    So far so good. More than 24 hours, no hangs:
    Code:
    /mnt/storage/adblock/pixelserv version: V35.HZ11WIP13 compiled: Nov 21 2014 22:10:25
    94704 uts, 4769 req, 1290 avg, 8594 rmx, 950 tav, 10547 tmx, 0 err, 107 tmo, 223 cls, 0 nou, 0 pth, 306 nfe, 624 ufe, 106 gif, 0 bad, 1878 txt, 0 jpg, 2 png, 0 swf, 2 ico, 807 ssl, 0 sta, 30 stt, 0 204, 667 rdr, 17 pst, 0 hed
     
  58. HunterZ

    HunterZ Network Guru Member

    @AndreDVJ Cool, I'll make it a release version in a couple days if nobody reports any additional issues. I think fixing the accept() hang right here at the end is probably the single most important change that I've made since I started working on pixelserv, as it was probably the source of most reported hangs :)

    Our stats look a lot different for a similar amount of run time and request counts:
    Code:
    dist/pixelserv.debug version: V35.HZ11WIP13 compiled: Nov 21 2014 22:10:25
    137539 uts, 6859 req, 3272 avg, 82033 rmx, 1234 tav, 12990 tmx, 0 err, 27 tmo, 89 cls, 0 nou, 0 pth, 202 nfe, 98 ufe, 57 gif, 0 bad, 529 txt, 35 jpg, 294 png, 0 swf, 4 ico, 3640 ssl, 3 sta, 0 stt, 210 204, 674 rdr, 997 pst, 0 hed
    Some specific notable differences:
    • My average and maximum request sizes are much larger, possibly because of much higher pst and/or ssl counts.
    • My cls and especially tmo counts are much lower.

    I should probably add some logging of POST requests. It's a bit concerning that something on my network is blindly doing a POST to a blocked domain name on an average interval of 138 seconds.
     
  59. The Master

    The Master Network Guru Member

    Maybe it helps somebody:

    Code:
    V35.HZ11WIP13 compiled: Nov 22 2014 03:05:36
    75103 uts, 2357 req, 663 avg, 11755 rmx, 1101 tav, 10008 tmx, 0 err, 219 tmo, 57 cls, 0 nou, 0 pth, 300 nfe, 106 ufe, 21 gif, 0 bad, 512 txt, 4 jpg, 0 png, 1 swf, 0 ico, 862 ssl, 7 sta, 0 stt, 0 204, 254 rdr, 14 pst, 0 hed
    
    Running on a R7000:
    Code:
    time info:
    22:06:32 up 2 days, 23:31, load average: 0.02, 0.02, 0.04 
     
  60. pharma

    pharma Network Guru Member

    So far so good! No hangs on any sites I normally have issues with ... will update after a few days.
     
  61. hammer

    hammer Connected Client Member

    V35.HZ11WIP13 runs great :) no issues so far (TP-Link TL-WDR4300 with OpenWrt Barrier Breaker and ARM Android phone with CyanogenMod 11)
    Just a suggestion. Show input parameters so /servstats returns "dist/pixelserv.debug -n br-lan -o 5 version: ..."
     
    Last edited: Nov 24, 2014
  62. HunterZ

    HunterZ Network Guru Member

    That should be doable. I'll just have to pass argc, argv into the socket handler and the stats generation functions and then use a loop to print them.
     
    hammer likes this.
  63. mstombs

    mstombs Network Guru Member

    If you look at command line arguments, could you check "-o anything", I think it proceeds with timeout = 0 ... BTDTGTTS!
     
  64. HunterZ

    HunterZ Network Guru Member

    PEBKAC!

    The command line argument checking isn't very robust. It currently breaks things into two categories: one for arguments that don't require a followup argument and another for ones that do. For the latter case, the following argument is always interpreted as the followup with minimal error checking.

    I suppose timeout=0 isn't very useful, though, as it would probably end up basically cutting off every connection. I'll look into adding greater-than-zero enforcement.
     
  65. HunterZ

    HunterZ Network Guru Member

    V35.HZ11 final released! Thanks to everyone for the testing and feedback.

    Binaries posted at https://github.com/HunterZ/pixelserv/releases/tag/V35.HZ11

    V35.HZ11 final (NOTE: these changes are just ones made in the last checkin, not cumulative since HZ10; please refer to github for changelogs)
    - error out if -o or -w are not followed by a positive value
    - modify get_version() to take argc & argv and report command-line
    - modify socket_handler() to take argc & argv for use with get_version()
     
    vincom, Goggy, hammer and 1 other person like this.
  66. hammer

    hammer Connected Client Member

    Thanks for adding the "options:" :D

    One cosmetic issue is that you forgot to add spaces when outputting the options (both in /servstats and in the text shown when called from command prompt):
    Code:
    /home/pixelserv version: V35.HZ11 compiled: Nov 25 2014 07:41:20 options: -nbr-lan
    
    I've attached my ARM Android build.
     

    Attached Files:

    HunterZ likes this.
  67. HunterZ

    HunterZ Network Guru Member

    Doh, you're right. I concatenated the strings together without adding any whitespace between them. I'll get that fixed tonight.
     
    Goggy and hammer like this.
  68. leandroong

    leandroong LI Guru Member

    Samsung Tab 2 and not using CM11 FW. My "/data" is empty. After creation of /data/local/userinit.d then just copy "pixelserv" ?

    edit2: after copying pixelserv to /data/local/userinit.d. I test it using pocket wifi, ads not blocking and error on stats, "localhost/servstats". Something is missing ....

    edit3: compiled pixelserv is working. Maybe not applicable for android with no sim card.
    Code:
    130|root@android:/data/local/userinit.d # ./pixelserv -help
    Usage:./pixelserv [IP No/hostname (all)] [-2 (disables HTTP 204 reply to generate_204 URLs)] [-f (stay in foreground - don't daemonize)] [-n i/f (all)] [-o select_timeout (10 seconds)] [-p port (80) & (443)] [-r (deprecated - ignored)] [-R (disables redirect to encoded path in tracker links)] [-s /relative_stats_html_URL (/servstats) [-t /relative_stats_txt_URL (/servstats.txt) [-u user ("nobody")]
    1|root@android:/data/local/userinit.d #
    
     
    Last edited: Nov 25, 2014
  69. hammer

    hammer Connected Client Member

    Have you replaced "/system/etc/hosts" with anything redirecting to pixelserv? pixelserv is just a webserver.
     
  70. leandroong

    leandroong LI Guru Member

    Nope. Tell me the procedures and I will test it.
     
  71. hammer

    hammer Connected Client Member

    1. pixelserv should be running (you probably need to run as root else I don't think it will start). Become root with "su" and start with "./pixelserv" and verify it's running "ps | grep pixelserv" and that it's reachable at http://localhost/servstats
    2. Replace "/system/etc/hosts" with "http://someonewhocares.org/hosts/hosts" try to browse a couple of sites and look at http://localhost/servstats to see if some of the counters are increasing when hosts are redirected to pixelserv.
    3. For further questions not related to making pixelserv run (on Android) PM me as we are going off topic... this is not a thread about ad-blocking in general :)
     
    Last edited: Nov 25, 2014
  72. HunterZ

    HunterZ Network Guru Member

    Quick fix for the new options logging.

    HZ11FIX1
    - insert space between command line options in logging output
     

    Attached Files:

    AndreDVJ and hammer like this.
  73. leandroong

    leandroong LI Guru Member

    My testing of pixelserv works on my samsung tab2, arm build. Only issue, I need to invoke pixelserv manually and since it is always online, not an issue for me. Here is an interesting video on how to replace "hosts" file.
     
  74. leandroong

    leandroong LI Guru Member

    hammer likes this.
  75. WaLLy3K

    WaLLy3K Networkin' Nut Member

    My standard broadcom ARM compile
     

    Attached Files:

  76. The Master

    The Master Network Guru Member

    thx
     
  77. hammer

    hammer Connected Client Member

    And the ARM Android build :D
     

    Attached Files:

    Goggy and HunterZ like this.
  78. AndreDVJ

    AndreDVJ LI Guru Member

    @HunterZ

    Not sure if you noticed this, but with HZ11FIX1 when you run the pixelserv binary without any options (to check the version/compilation time), something with the malloc() function returns an error
    Code:
    pixelserv[14578]: ./pixelserv version: V35.HZ11FIX1 compiled: Nov 25 2014 21:32:12 options: <malloc() error>
    
    That message doesn't ocurr with the HZ11 release.
    Code:
    pixelserv[14607]: ./pixelserv version: V35.HZ11 compiled: Nov 25 2014 04:28:51 options:
    
    However, starting pixelserv normally passing as argument the IP address it should listen to, it works fine and as intended:
    Code:
    pixelserv[14944]: /mnt/storage/adblock/pixelserv version: V35.HZ11FIX1 compiled: Nov 28 2014 13:15:59 options: 192.168.1.254
     
  79. HunterZ

    HunterZ Network Guru Member

    I didn't think to test that case. Here's an HZ11FIX2.

    Note that if you try to run it without parameters, it will end up trying to listen on the router's main IP. This will likely fail due to the router's web UI already listening on ports 80 (HTTP) and/or 443 (SSL).

    I usually put a fake parameter like --help on the command line to check options.
     

    Attached Files:

    Goggy and hammer like this.
  80. hammer

    hammer Connected Client Member

    ARM Android build
     

    Attached Files:

    Goggy likes this.
  81. superdos

    superdos Networkin' Nut Member

    ARM build, Asus AC68U
    Note compile time :)
    V35.HZ11FIX2 compiled: Nov 29 2014 13:37:00
     

    Attached Files:

    The Master likes this.
  82. HorseCalledHorse

    HorseCalledHorse LI Guru Member

    Would love to try this on my E3000 but the file size is too big for the JFS partition. Is HZ8 the last of the dynamic builds? And is there no way to get a smaller build of the current version?
     
  83. HunterZ

    HunterZ Network Guru Member

    Where does your adblock blocklist live? I can't imagine it would be smaller than pixelserv.
     
  84. HorseCalledHorse

    HorseCalledHorse LI Guru Member

    ^ Lives on the JFS partition, too. HZ8 is 19.9KB (Static build is 107.1KB). HZ11 is 159.3KB, so it's a big jump.
     
  85. HunterZ

    HunterZ Network Guru Member

    Here is a one-off dynamic, upx-compressed build of HZ11FIX2. I don't want to produce these every time, because it's extra work (have to build on a separate machine) may result in buggy or suboptimal behavior since it's built with an older compiler and will end up using the older libraries included with the firmware.

    Personally I run everything (entware, tomatoware, adblock, etc.) off of a CIFS mount because I have a reliable Linux workstation/server on my LAN with lots of hard drives in it. It would probably be smarter for me to buy a USB stick to plug into my router, but this seems to work well enough.
     

    Attached Files:

    Michael Malone and mstombs like this.
  86. mstombs

    mstombs Network Guru Member

    The end of a long run:-

    Code:
    pixelserv.fast version: V35.HZ10 compiled: Sep 14 2014 22:08:48
    Nov 29 21:32:59 rtn66u daemon.info pixelserv[2457]: 6484283 uts, 149357 req, 1340 avg, 56822 rmx, 13 err, 16328 tmo, 342 cls, 1 nou, 1 pth, 4999 nfe, 56890 ufe, 1044 gif, 966 bad, 0 txt, 13 jpg, 16 png, 41 swf, 14 ico, 61545 ssl, 106 sta, 14 stt, 7020 rdr
    Nov 29 21:32:59 rtn66u daemon.notice pixelserv[2457]: exit on SIGTERM
    start of new
    Code:
    Nov 29 22:02:49 rtn66u daemon.info pixelserv[23247]: /mnt/usb4gb/pixelserv version: V35.HZ11FIX2 compiled: Nov 29 2014 13:32:49 options: 192.168.66.254 -p 80 -p 81 -p 8080 -p 8081 -p 443
    Nov 29 22:02:49 rtn66u daemon.notice pixelserv[23249]: Listening on :192.168.66.254:80
    Nov 29 22:02:49 rtn66u daemon.notice pixelserv[23249]: Listening on :192.168.66.254:81
    Nov 29 22:02:49 rtn66u daemon.notice pixelserv[23249]: Listening on :192.168.66.254:8080
    Nov 29 22:02:49 rtn66u daemon.notice pixelserv[23249]: Listening on :192.168.66.254:8081
    Nov 29 22:02:49 rtn66u daemon.notice pixelserv[23249]: Listening on :192.168.66.254:443
    
    HunterZ 12kB upx one above

    Suspect tmx not going to be very useful, I have deliberately challenged all the other counters, not got an "err" yet!

    Code:
    /mnt/usb4gb/pixelserv version: V35.HZ11FIX2 compiled: Nov 29 2014 13:32:49 options: 192.168.66.254 -p 80 -p 81 -p 8080 -p 8081 -p 443
    2999 uts, 1201 req, 240 avg, 692 rmx, 280 tav, 10000 tmx, 0 err, 14 tmo, 2 cls, 2 nou, 2 pth, 6 nfe, 11 ufe, 1 gif, 1 bad, 24 txt, 1001 jpg, 1 png, 1 swf, 2 ico, 94 ssl, 32 sta, 1 stt, 1 204, 3 rdr, 1 pst, 1 hed
    Guess you can tell from my path I have a 4GB USB stick...

    [edit - still running]

    Code:
    /mnt/usb4gb/pixelserv version: V35.HZ11FIX2 compiled: Nov 29 2014 13:32:49 options: 192.168.66.254 -p 80 -p 81 -p 8080 -p 8081 -p 443
    2942646 uts, 96130 req, 526 avg, 103143 rmx, 2537 tav, 19060 tmx, 0 err, 12668 tmo, 1206 cls, 2 nou, 2 pth, 4118 nfe, 7687 ufe, 628 gif, 6 bad, 20726 txt, 1004 jpg, 28 png, 121 swf, 10 ico, 43095 ssl, 43 sta, 1 stt, 1 204, 4323 rdr, 460 pst, 1 hed
     
    Last edited: Jan 2, 2015
  87. HorseCalledHorse

    HorseCalledHorse LI Guru Member

    ^^ Thanks, HunterZ. I should probably buy a USB stick myself…
     
  88. HunterZ

    HunterZ Network Guru Member

    @mstombs crazy that 10% of connection requests timed out on you.

    Regarding tmx, I've seen some numbers around 15000 which means that some connections got stuck for 50% longer than the 10 sec recv timeout. I should probably change my time check code to reset the elapsed time after logging a time warning so that it can catch when there are multiple time overruns during the same connection, then test with a ~5000ms warning threshold.
     
  89. HunterZ

    HunterZ Network Guru Member

    Got a report on the adblock thread that someone on a wrt54g got an error from the clock_gettime() call in get_version(). It looks like maybe the 2.4 kernel and/or uclibc on the wrt54g firmwares does not support CLOCK_MONOTONIC.

    Edit: Research indicates that the best approach is probably to adjust the code to fallback to CLOCK_REALTIME if attempting to use CLOCK_MONOTONIC fails.

    This is going to be a bit of a pain because I added time checking code in a few places too. I may need to add a wrapper function in util.c to encapsulate the fallback mechanism.

    Also, note that this means that the time checking/reporting on wrt54g and any other affected platforms may be less accurate, because CLOCK_REALTIME may be affected by time jumps caused by NTP synchronization and such. Unfortunately this can't be helped as far as I can tell.
     
    Last edited: Dec 6, 2014
  90. HunterZ

    HunterZ Network Guru Member

    Okay here is an HZ12WIP1 that falls back to CLOCK_REALTIME if CLOCK_MONOTONIC fails to work. I'm interested to know if wrt54g or other K24 users have any other issues.
     

    Attached Files:

  91. The Master

    The Master Network Guru Member

    @R7000 no problem so far...
     
  92. phuklok1

    phuklok1 Network Guru Member

    HunterZ, thank you for for the work that you have been doing. I have tried this new fix on the latest v1.28.7636 Toastman-IPT-ND ND Std release on a 2.4.37.11 kernel wrt-54g-TM and, unfortunately, it immediately returns with "Illegal instruction" upon executing the command. There are also no messages pertaining to the failure in /var/log/messages.
     
  93. HunterZ

    HunterZ Network Guru Member

    @phuklok1 sorry to hear that. It may be necessary for someone with a K24 toolchain to build a binary.

    As a shot in the dark, here is a dynamically linked build to try.
     

    Attached Files:

  94. phuklok1

    phuklok1 Network Guru Member

    Thanks, but now it returns with a "bus error".
     
  95. phuklok1

    phuklok1 Network Guru Member

    Hey HunterZ. Just FYI, since it is my only router and I wanted to get it working again, I changed builds over to the latest shibby 2.6 MIPSR1 build from 2.4. (it was more up to date than the last 2.6 Toastman for MIPSR1). While there is some performance degradation using 2.6 kernel on this older router, with your pixelserv-V35.HZ12WIP1.mips.zip build, pixelserv came up on the wrt54g-TM. Thank you again for your help.
     
    Last edited: Dec 8, 2014
  96. HunterZ

    HunterZ Network Guru Member

    Wow, I didn't know that K26 was even an option on the wrt54g. That tempts me to switch my wrt54g back over from OpenWRT (yes, I have one too, but I don't have a cross-compiler toolchain for it), which I installed in order to get a working K24 version of ntpd.

    Glad you found a solution for getting pixelserv to work.
     
  97. echo_off

    echo_off Reformed Router Member

    Been getting really slow page loads/refreshes with imdb.com. Anyone else with this problem?
     
  98. leandroong

    leandroong LI Guru Member

    Very quick display in my firefox browser.
     
  99. HunterZ

    HunterZ Network Guru Member

    No issues here.

    I've been having intermittent slow load issues with linksysinfo though, ironically.

    Note that slow loads can't be pixelserv's fault unless it's due to pixelserv not responding quickly to a request that was directed to it. I suggest using Firefox's Tools->Web Developer->Network feature to see what is taking so long to load.
     
  100. lancethepants

    lancethepants Network Guru Member

    I think it's just linksysinfo. Its been feeling really slow for a couple weeks
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice