1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

pixelserv compiled to run on router WRT54G

Discussion in 'Tomato Firmware' started by Jedis, Sep 5, 2009.

  1. mstombs

    mstombs Network Guru Member

    Thanks will do, going to have write a test program to deliberately break connections though to research this FIN_WAIT2 thing that's plagued the code for so long! Definitely something I don't understand in the all versions router Linux kernel code re "linger2". You can give it a value, but it is not used, there are only checks if negative or positive...

    Code:
    tp->linger2 = val * HZ
    Edit: Have now found how a proper webserver apache closes connections - need a 2 second select timer on the read before close.
     
  2. ~nephelim~

    ~nephelim~ LI Guru Member

    a linger2 value greater than zero might actually come into play

    Code:
    //net/tcp.h
    static inline int [B]tcp_fin_time[/B](struct tcp_opt *tp)
    {
            // [URL="http://en.wikipedia.org/wiki/%3F:#C"]GNU extension to C[/URL]
            // int fin_timeout = tp->linger2 ? tp->linger2 : sysctl_tcp_fin_timeout; 
    
             int fin_timeout = tp->linger2 ? : sysctl_tcp_fin_timeout; 
     
             if (fin_timeout < (tp->rto<<2) - (tp->rto>>1))
                     fin_timeout = (tp->rto<<2) - (tp->rto>>1);
     
             return fin_timeout;
     }

    Code:
    //net/tcp.c
    	/*	This is a (useful) BSD violating of the RFC. There is a
    	 *	problem with TCP as specified in that the other end could
    	 *	keep a socket open forever with no application left this end.
    	 *	We use a 3 minute timeout (about the same as BSD) then kill
    	 *	our end. If they send after that then tough - BUT: long enough
    	 *	that we won't make the old 4*rto = almost no time - whoops
    	 *	reset mistake.
    	 *
    	 *	Nope, it was not mistake. It is really desired behaviour
    	 *	f.e. on http servers, when such sockets are useless, but
    	 *	consume significant resources. Let's do it with special
    	 *	linger2	option.					--ANK
    	 */
    
    	if (sk->state == TCP_FIN_WAIT2) {
    		struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp);
    		if (tp->linger2 < 0) {
    			tcp_set_state(sk, TCP_CLOSE);
    			tcp_send_active_reset(sk, GFP_ATOMIC);
    			NET_INC_STATS_BH(TCPAbortOnLinger);
    		} else {
    			int tmo = [B]tcp_fin_time[/B](tp);
    
    			if (tmo > [I]TCP_TIMEWAIT_LEN[/I]) {
    				tcp_reset_keepalive_timer(sk,[B] tcp_fin_time[/B](tp));
    			} else {
    				atomic_inc(&tcp_orphan_count);
    				tcp_time_wait(sk, TCP_FIN_WAIT2, tmo);
    				goto out;
    			}
    		}
    	}
    Code:
     
    //net/tcp.c
     int sysctl_tcp_fin_timeout = [COLOR="Red"]TCP_FIN_TIMEOUT[/COLOR];
    
    
    //setsockopt
           case TCP_LINGER2:
                     if (val < 0)
                             tp->linger2 = -1;
                     else if (val > sysctl_tcp_fin_timeout/[COLOR="Blue"]HZ[/COLOR])
                             tp->linger2 = 0;
                     else
                             tp->linger2 = val*[COLOR="Blue"]HZ[/COLOR];
                     break;
     
    
    [strike]but just what HZ is?[/strike]


    Code:
     //asm-mips/param.h
    
    /* This is the external value of [COLOR="#0000ff"]HZ[/COLOR] as seen by user programs.  Don't change
      unless you know what you're doing - changing breaks binary compatibility.  */
    #define [COLOR="#0000ff"]HZ[/COLOR] 100
    
    
    Code:
    (1) [URL="http://en.wikipedia.org/wiki/Jiffy_(time)#Use_in_computing"]Jiffie[/URL]: internal timeunit for the kernel. On the i386 1/100s, on the
      Alpha 1/1024s. See the [COLOR="#0000ff"]HZ[/COLOR] define in /usr/include/asm/param.h for the exact
      value on your system. 
    
    Code:
    //net/tcp.h
    
    #define [I]TCP_TIMEWAIT_LEN[/I] (60*[COLOR="Blue"]HZ[/COLOR]) /* how long to wait to destroy TIME-WAIT
                                       * state, about 60 seconds     */
    #define [COLOR="#ff0000"]TCP_FIN_TIMEOUT[/COLOR] [I]TCP_TIMEWAIT_LEN[/I]
                                      /* BSD style FIN_WAIT2 deadlock breaker.
                                       * It used to be 3min, new value is 60sec,
                                       * to combine FIN-WAIT-2 timeout with
                                       * TIME-WAIT timer.
                                       */
    BTW I finally confirmed that 128 is the fork limit imposed on pixelserv using getrlimit RLIMIT_NPROC.
     
  3. mstombs

    mstombs Network Guru Member

    Thanks, who would put code in a header file ...

    Looks like its a "live-lock", kernel doesn't close the connection because something actively waiting on it. I will be able to fix it now... shutdown now looks like

    Code:
                /* clean way to flush read buffers and close connection */
                if (shutdown (new_fd, SHUT_WR) == 0) {
                    /* Initialize the file descriptor set. */
                    FD_ZERO (&set);
                    do {
                        FD_SET(new_fd, &set);
                        /* Initialize the timeout data structure. */
                        timeout.tv_sec = 2;
                        timeout.tv_usec = 0;
                        /* select returns 0 if timeout, 1 if input available, -1 if error. */
                        select_rv = select(FD_SETSIZE, &set, NULL, NULL, &timeout);
                    } while ( (select_rv > 0) && ( read (new_fd, buf, CHAR_BUF_SIZE) > 0 ) );
                }
    need to test and tune timeout with wireshark before declaring success
     
  4. ~nephelim~

    ~nephelim~ LI Guru Member

    I found some slides with sourcecode examples of server implementations using poll and select.

    It look like these functions are used to time the connection handling starting from accept()

    In a webserver example from IBM fork is used along signal(SIGCLD, SIG_IGN); this might provide an alternative to waitpid

    A FAQ about forks advises to use _exit() in child processes (or call exit() only once for each entry into main)

    /uClibc/libc/unistd/daemon.c misses open fd closure step described in Unix Daemon Server Programming or How do I get my program to act like a daemon?

    How do I change the name of my program (as seen by `ps')? looked interesting (eg for diagnostic/info purposes) but I was not able to get it working.
     
  5. mstombs

    mstombs Network Guru Member

    I recommend Beej's guide

    http://beej.us/guide/bgnet/output/html/multipage/clientserver.html

    which is where the original server.c code came from, and I've just used client.c to test broken connections - which should now be fixed, linger2 option even has a noticeable effect after connection forcibly closed.

    Attached V22 (for tomatousb and older stock Tomato - tested on Tomato 1.22).

    Have changed the signal used away from the system SIGHUP to SIGUSR1 as per Rodney's advice above, so use

    Code:
    /tmp/home/root# kill -SIGUSR1 $(pidof pixelserv)
    to prompt stats
    Code:
    Dec  2 12:19:23 unknown daemon.info pixelserv[673]: 2 pixels served
    
    options

    Code:
    Usage:pixelserv [-i] [IP] [-p 80] [-n IF] [-u user]
    i = inetd mode, IP or hostname to listen on (all), p = port No/name, IF = interface name (all), user ("nobody")
    example

    Code:
    ./pixelserv -n br0 -p 88
    results in

    Code:
    Dec  2 12:12:40 unknown daemon.info pixelserv[671]: ./pixelserv V22 compiled: Dec  2 2010 20:11:09 from pixelserv22.c
    Dec  2 12:12:40 unknown daemon.notice pixelserv[673]: Listening on br0 0.0.0.0:88
    Bug Report: "-u user" option is broken, can't change from user "nobody" on 32 bit machines
     

    Attached Files:

  6. ~nephelim~

    ~nephelim~ LI Guru Member

    Thanks for the fix. :)

    Those slides contained multiple implementations featuring also one based on a different example from Beej's guide.
    Code:
    /* 
    TCP echo server using select (more compact)
     based on example from Beej's tutorial to network programming
    */ 
    It came as no surprise the author recommended Beej's guide as well.
     
  7. rhester72

    rhester72 Network Guru Member

    So far, so good, mstombs - looking very solid over here!

    Rodney
     
  8. Jedis

    Jedis LI Guru Member

    Do I want to upgrade from v7 now on my WRT54G?
     
  9. mstombs

    mstombs Network Guru Member

    Ha-Ha. You started this thread - I underestimated how tricky this was going to be, but have learnt a lot in process! Try it and see if it works for you! I test basic operation of all versions before posting - but now know previous versions do have potential to bring down the router by consuming memory/connections (V8-V14 single threaded so just stop working). Would take a long time at 1 stuck process every couple of days - but it is a vulnerability so I have taken down most intermediate versions above.

    Still issue with error messages when browser expecting javascript and it gets a gif in reply...
     
  10. ~nephelim~

    ~nephelim~ LI Guru Member

    I carried multiple tests and it look like hanged instances are gone.

    I still get a small number of truncated TTP replies.

    V17 format recognition was a good idea. It could be adapted to recognize few non binary formats (.js,.css) that would be normally unable to return an image (php could, for example) while replying with a gif in all other cases.

    This way the number of mismatched contents will decrease without impairing pixelserv main purpose.
     
  11. srouquette

    srouquette Network Guru Member

    updated my router with it aswell, I'll report here if I see something strange.
    thanks for the update :)
     
  12. QSxx

    QSxx LI Guru Member

    With pixelserv v22 (normal version) run on config from my signature i keep getting script errors in IE. For ex. when pulling this one http://www.google-analytics.com/ga.js

    I kinda jumped from v17 (where it worked fine) to v22 - I didn't notice anything in your previous post so I'm wondering is this behaviour normal/expected.

    Please disregard if yes :biggrin:
     
  13. mstombs

    mstombs Network Guru Member

    Sorry yes, V17 only sends a gif if it is sure an image is requested, but the null script - just an HTTP 200 header broke things for others (and myself). I should produce a version with V17 behaviour selectable - V17 as is will have the hanging processes bug - which is fixed in V22 - in the meantime i recommend you schedule a regular "killall pixelserv" (note this didn't work for all) and restart to clean up any dormant processes.
     
  14. ~nephelim~

    ~nephelim~ LI Guru Member

    I recall srouquette mentioned some amazon ad that did not have an image extension in the URL and thus no null pixel was sent.

    It would be possible to address these cases by having pixelserv default to null pixel for anything but .js

    for js the reply could be something like
    Code:
    	static const unsigned char httpnulltext[] = 
    	"HTTP/1.1 200 OK\r\n"
    	"Content-type: text/javascript\r\n"
    	"Accept-ranges: none\r\n"
    	"Content-length: 0\r\n"
    	"Connection: close\r\n"
    	"\r\n";
    or perhaps with content length = 1 and retuning a \n
     
  15. mstombs

    mstombs Network Guru Member

    I've tried that, but I 'think' this is better

    Code:
        static const unsigned char httpnulltext[] =
        "HTTP/1.1 204 No Content\r\n"
        "Content-type: text/javascript\r\n"
        "Content-length: 0\r\n"
        "Connection: close\r\n"
        "\r\n";
    Attached test version should run on tomatousb.

    Added back the test for images from V17, added a test for js, anything undecided defaults to pixel, but can be changed using "-d t" to default to null text.

    Still get js errors due to variables not set, but hopefully not the syntax ones where browser tries to execute the gif.
     
  16. ~nephelim~

    ~nephelim~ LI Guru Member

    Thanks. The new version works well.

    204 No Content is a bit tricky as http://192.168.0.1/test.js will work fine but won't return a blank page and instead remain on the current page.

    Code:
    <!DOCTYPE HTML>
    <html lang="en">
    <head>
    	<title>Hello World!</title>
    	<meta charset="UTF-8">
    <script type="text/javascript" src="http://192.168.0.1/test.js"></SCRIPT>
    <body>
    
    	<h1>This is a test</h1>
    	<p>Test <a href="http://192.168.0.1/test.js">blacklisted js content</a> here</p>
    
    </body>
    </html>
    If a webpage contain a link to a js from some blackisted server clicking on it (or testing http://192.168.0.1/test.js in the addressbar) will have no visible effect.

    As far I know this would be the only difference in using 204 instead of 200.

    There will be still some errors if a page attempt to reference variable and functions that were meant to be provided in the blocked js files but it will work like a charm for selfcontained ads added through an external js.
     
  17. mstombs

    mstombs Network Guru Member

    The alternate null text is in the source-code, so if you can re-compile, uncomment one, uncomment the other to test. Testing not very scientific, is also dependent on cache contents - but I think I got broken image symbols in IE from the 200 reply, and the "do nothing" action of 204 was better, but quite possibly counter examples!
     
  18. ~nephelim~

    ~nephelim~ LI Guru Member

    maybe something related to this:

    I was not able to understand the reason but http://rcm.amazon.com/example does not return a null image.

    http://rcm.amazon.com/example.gif and http://rcm.amazon.com/example.js work as intended :confused:

    I even removed the test for jpg,gif etc without luck
    Code:
    			tex = 0; //default
    syslog (LOG_ERR, "before test - tex: %i",tex);
    			if ( (ext != NULL) && (
    				(strcmp (ext, "js") == 0)
    				|| (strncmp (ext, "js", 2) == 0)	/* jsx ?*/
    				) ){
    				tex = 1;
    			}
    syslog (LOG_ERR, "after test - tex: %i",tex);
    			if (tex) {
    				rv = send (new_fd, httpnulltext, (sizeof httpnulltext) - 1, 0);
    			} else 	{
    #endif /* TEXT_REPLY */
    				rv = send (new_fd, httpnullpixel, (sizeof httpnullpixel) - 1, 0);
    #ifdef TEXT_REPLY
    			}
    #endif /* TEXT_REPLY */
    Edit: With http://rcm.amazon.com/example it does not reach the second syslog line
    Code:
    Dec  8 17:06:04 router daemon.err pixelserv[4830]: before test - tex: 0
    Dec  8 17:06:05 router daemon.err pixelserv[4831]: before test - tex: 0
    Dec  8 17:06:05 router daemon.err pixelserv[4832]: before test - tex: 0
    Dec  8 17:06:05 router daemon.err pixelserv[4833]: before test - tex: 0
    edit2: this will have ext =1 if strrchr returns null so (ext != NULL) later on will not apply

    Code:
    ext = strrchr (ext, '.') + 1;


    For adblocking tests www.majorgeeks.com would provide a good testcase

    Code:
    # dnsmasq.custom
    address=/intellitxt.com/doubleclick.net/rubiconproject.com/invitemedia.com/192.168.0.1
    address=/googleadservices.com/tribalfusion.com/google-analytics.com/quantserve.com/192.168.0.1
    address=/exponential.com/scorecardresearch.com/yieldmanager.com/tradepub.com/192.168.0.1
    #address=/rcm.amazon.com/pagead2.googlesyndication.com/192.168.0.1
     
  19. mstombs

    mstombs Network Guru Member

    You "might" be seeing same problem I saw - the browser "remembers" the results of a previous test despite clearing cache ctrl-F5 etc. We don't set any dates/times or cache control flags in the http header. In general this should work, reducing load on router and speeding up browser - but it makes testing tricky! Asking for slightly different filenames each time is one way round during testing!
     
  20. ~nephelim~

    ~nephelim~ LI Guru Member

    I'm now using
    Code:
    #ifdef TEST
    			printf ("trimpath: '%s'\n", ext);
    #endif
    			ext = strrchr (ext, '.');
    #ifdef TEST
    			printf ("ext: '%s'\n", ext);
    #endif
    			tex = 0;
    			if ( (ext != NULL) && (
    				(strncasecmp (ext, ".js", 3) == 0)	/* jSx ?*/
    				) ){
    				tex = 1;
    			}
    			if (tex) {
    				rv = send (new_fd, httpnulltext, (sizeof httpnulltext) - 1, 0);
    			} else 
    #endif /* TEXT_REPLY */
    			{
    				rv = send (new_fd, httpnullpixel, (sizeof httpnullpixel) - 1, 0);
    			}
    
    EDIT: revised snippet for case insensitive test on dotted extensions.
    Code:
    # dnsmasq.custom
    address=/intellitxt.com/doubleclick.net/rubiconproject.com/invitemedia.com/192.168.0.1
    address=/googleadservices.com/tribalfusion.com/google-analytics.com/quantserve.com/192.168.0.1
    address=/exponential.com/scorecardresearch.com/yieldmanager.com/tradepub.com/192.168.0.1
    address=/rcm.amazon.com/pagead2.googlesyndication.com/192.168.0.1
    http://rcm.amazon.com/nodottest works fine

    Using majorgeeks I tested

    Code:
    	static const unsigned char httpnulltext[] =
    	"HTTP/1.1 200 OK\r\n"
    	"Content-type: text/javascript\r\n"
    	"Content-length: 0\r\n"
    	"Connection: close\r\n"
    	"\r\n";
    and the content length 1 alternative on IE and chrome and reset the cache each time.

    It works fine here.

    I still get few image errors on linksysinfo.org , however these [​IMG] are not a result of pixelserv
     
  21. mstombs

    mstombs Network Guru Member

    He He - quite believe my parsing broken - but that could fail with a segfault - "ext" is a calculated pointer, so should be checked for NULL before use - but you then use "ext + 1". I need to check with "-DTEST" what you get when no detected file extension - what reply do you want for "nodottest"?

    BTW the 2 tests for "js" are not needed - just left the construct expecting to add other extensions to get the text reply!

    Will use your httpnulltext, sounds like your testing more thorough than mine!

    [edit] Doh it was V23 (and also V17) that must actually crash due to bug!
     
  22. ~nephelim~

    ~nephelim~ LI Guru Member

    I added syslog to output the value of ext just before was tested for NULL.

    When the url had no dot ext pointer was == 1 (NULL + 1)

    I'm not really comfortable with pointers. What I intended to do there using ext+1 was something mimicking ext[1]. I tough ext+1 would work the same way.

    I now changed the test to use one strncasecmp on plain ext pointer checking extensions with the dot included.
    A null gif would be fine. the pupose is to trap the ad testcase reported by srouquette

    Overall it would trap even webpages (or archives, executables etc) from blacklisted domains. But for these a null gif would be fine.

    There is a risk of trapping css and replying with a gif but I guess most whitelisted sites would not use the style sheet from a blacklisted site.
    Else it would be possible to return a text/css reply just in case.
     
  23. mstombs

    mstombs Network Guru Member

    You are correct, I mistakenly reduced code steps by moving the "+1" up - the V23 code is broken, ext as returned from strrchr must be checked for NULL before use. If not NULL it points to an offset in memory reserved by buf, even the TEST printf will currently fail.

    Found and fixed couple of other potential issues, and an pathname with extra "/" which didn't get correctly parsed.

    So quick repair here is V24, a version compiled for tested on stock Tomato 1.28, so should run on all versions of Tomato. The recent additions increased size to 13kB, so made some minor changes to get it back to less than 10KB (a tomatousb one will be 2-3kB smaller).

    The behaviour is basically that of V22, except if .js* requested when ~nephelim~ 's preferred null text is returned. Does not have option of only testing images as per V17, hope still ok?

    Unfortunately you do still get javascript errors because variables and functions that are supposed to be defined by the block js do not exist.
     
  24. ~nephelim~

    ~nephelim~ LI Guru Member

    Thanks for the update.

    pixelserv works fine.

    js detection does fairly good though is not able to suppress js errors in all cases.

    eg on majorgeeks wipe out things like this (which in turn add the ad images) without errors
    Code:
    <script type="text/javascript" src="http://a.tribalfusion.com/j.ad?site=majorgeekscom&amp;adSpace=ros&amp;tagKey=4214928978&amp;th=27709887882&amp;tKey=aumn6y0bFZbT6mu56MgQABnv0Gmu2&amp;size=160x600|120x600&amp;p=1210261&amp;a=2&amp;flashVer=10&amp;ver=1.20&amp;center=1&amp;noAd=1&amp;url=http%3A%2F%2Fwww.majorgeeks.com%2F&amp;f=0&amp;rnd=1216613"></script>
    but trigger an error for google adsense due to embedded snippets calling missing functions.

    Code:
    <script type="text/javascript">
    
      GS_googleAddAdSenseService("ca-pub-6960825562757852");
    
      GS_googleEnableAllServices();
    
    </script>
    


    While testing found out a script tag that requests
    Code:
    http://majorgeeks.us.intellitxt.com/intellitxt/front.asp?ipid=11563
    but it gets a null pixel.



    There is really no simple solution for cases like these as asp (or php, etc) pages can return also images.




    I'm looking if it would be possible to add some more tests for malformed requests (eg count the number of spaces or /).

    It should be possible to have a 1024 bytes null-termined buf
    Code:
    char buf[CHAR_BUF_SIZE + 1]; /* buf + one more byte for \0 */
    ...
    
    
    rv = recv (new_fd, buf, CHAR_BUF_SIZE, 0); /*  -1 < rv < CHAR_BUF_SIZE */
    if (rv > 0 ) 
    buf[rv] = '\0'; /*   buf [CHAR_BUF_SIZE] ==  (CHAR_BUF_SIZE +1)th byte */
    
     
  25. mstombs

    mstombs Network Guru Member

    The >1 & [rv-1] was a bug (generally harmless, truncates last character), forgot moved +1 to the declaration, but had noticed.... I have wondered if there's a library function that can be called to get the path - because should really check "GET" is there (what to do with "HEAD" or "POST"?) and check each strtok call return for NULL. Maybe able to crash it at present (but only the replying subprocess) with carefully constructed nc (or definitely c program) call? Could get away with a much smaller buf, but its easy (but lazy!) to assume that the 1k will be enough for the whole request, except in TEST we are only interested in first line. Internet Explorer has a limit of 2kB for a path!

    There will always be some websites that don't work, and then you have a choice of whitelisting or putting up with error.
     
  26. ~nephelim~

    ~nephelim~ LI Guru Member

    uclibc implementation of strtok behave slightly differently from what I understood Open Group strok specification described.

    In fact it doesn't appear to return NULL if the tested-against token is missing, instead it will return the tested string in such case.

    I was able to have strtok return NULL by using

    Code:
    buf[0] = '\0';
    ext =  strtok (buf, "\n"); /* mark \n */
    EDIT: This should be similar to what happen for a rv==1 request containing only a token ( eg \n)


    So I added null tests in most cases in the following snippet


    Code:
    			rv = recv (new_fd, buf, CHAR_BUF_SIZE, 0);
    			if (rv > 0) {
    				buf[rv] = '\0';
    #ifdef TEST
    				TESTPRINT ("\nreceived %d bytes\r\n '%s'\n", rv, buf);
    				char *ext = strtok (buf, " ");
    				TESTPRINT ("method: '%s'\n", ext);
    				ext = strtok (NULL, " ");
    
    				TESTPRINT ("path: '%s'\n", ext);
    				char *prot = strtok (NULL, "\r\n");
    				TESTPRINT ("protocol: '%s'\n", prot);
    				char *host = strtok (NULL, " "); // HTTP/1.1 must also include Host:
    				host = strtok (NULL, "\r\n");
    				TESTPRINT ("host: '%s'\n", host);
    #endif
    #ifndef TEST
    				char *ext =  strtok (buf, "\n"); /* mark \n */
    #endif
    				if (ext != NULL) {
    					ext = strtok (ext, "?");
    					TESTPRINT ("trimpath: '%s'\n", ext);
    				}
    				if (ext != NULL) {
    					ext = strrchr (ext, '/'); /* to skip catch /fred.js/blank */
    				}
    				if (ext != NULL) {
    					TESTPRINT ("trimpath: '%s'\n", ext);
    					ext = strrchr (ext, '.');
    				}
    				if (ext != NULL) {
    					TESTPRINT ("ext: '%s'\n", ext);
    					 if ( strncasecmp (ext, ".js", 3) == 0 ) {	/* jsx ?*/
    						tex = 1;
    					}
    				}
    			}
    Though this way will return a null gif even if the request contained at least one character.

    It could be possible to add an additional test after rv > 0 to ensure it does at least match the minimal length for a barebone request (GET / HTTP 1.1 is at least 16 bytes long) and log all those attempts

    I searched for html parsers some time ago. I found them to be overly complex even for the tiniest server ( there are lots of micro, femto and whatever servers/libraries)

    http://sourceforge.net/projects/httpico/
    http://sourceforge.net/projects/yops2009/
    http://www.dolda2000.com/~fredrik/ashd/
    http://inglorion.net/software/muhttpd/
    http://www.gnu.org/software/libmicrohttpd/ <-- not really micro


    I guess it would be easier to test only for GET (eg strncmp(buf,"GET ",4) ) and return something like

    Code:
    "HTTP/1.1 405 Method not allowed\r\n"
    "Allow: GET\r\n"
    "Connection: close\r\n"
    "\r\n"
    for whatever else. I don't really know how much the browser will like that nor what is really to be expected though.


    EDIT:
    Alternatives replies to mismatched methods could be:
    HTTP/1.1 501 Not Implemented
    HTTP/1.1 500 Internal Server Error

    It would be probably better to send HTTP/1.1 501 for anything but get and/or perhaps use 405 for head
     
  27. mstombs

    mstombs Network Guru Member

    There's also http://code.google.com/p/mongoose/ but some very similar simple GET string parsing in www.jbox.dk/sanos/webserver.htm which has some more checks that could be copied - this has an inverse GET check and a 501 response.

    The first call to strtok should be OK, it's the subsequent calls that might return NULL (but it appears further calls to strtok also return NULL, other str functions may not be ok). I don't think the internet discussions about strtok not being thread-safe are relevant here?

    [edit] for future ref the request parsing in http://www.acme.com/software/mini_httpd/ easy to read, uses large temporary buffer and not strtok
     
  28. ~nephelim~

    ~nephelim~ LI Guru Member

    Great! the only example I found with 200 line was a lacking implementation from an ibm tutorial.
    As far I understood thread-safety is not a concern for fork-based implementation.
    The only recommendations I found were to use _exit in forked processes.


    On the other hand, stricter requirements might be imposed over signal handlers where only async-safe functions would do.

    For uclibc there no list of async-safe function. The most I found searching the mailing list was about reentrant functions..

    Signals appear very unforgiving. In some cases there might be a catch even to set a simple variable.

    I only have a partial comprehension about the latter example and additional resources like atomic usage patterns made it look even worse (it had me wondering what would happen when another unrelated signal come into play).
     
  29. mstombs

    mstombs Network Guru Member

    @~nephelim ~You have mentioned _exit and daemon both before, I notice that uclibc daemon, calls _exit twice. I would have expected the "_" to be to deliberately hide the function call from a user - I don't see a problem with the child doing as full an exit as possible - we don't want it to leave any residue waiting for the parent to die! As above discussion with Rodney, the older SIGHUP hit both the parent and child (and it just happened to be fortutitous), and the child contained a copy of the signal handler. Maybe the child should mask that signal when parsing urls?
     
  30. 1_fast_rsx

    1_fast_rsx Networkin' Nut Member

    mstombs, I can't seem to get v24 to run on all the interfaces like v17 did by default. what exactly is the command?

    /jffs/dns/pixelserv 192.168.1.254 -n IF=all -p 80

    gives me connection reset from VPN and SSH Tunnel
     
  31. mstombs

    mstombs Network Guru Member

    @1_fast_rsx - default should be all interfaces with null if "" - but are you using dd-wrt ? - because that doesn't like the "user" setting, needs to be compiled without support for this "-u root" didn't work, I posted a V22 over there, which is still the 'best version' this parsing of file extensions is proving tricky.

    But if you want to help with development, here is a latest test version (without user), compiled using old Linksys toolchain, tested on Tomato 1.28, when unable to parse the request it doesn't reply but puts a message in the log, for example

    Code:
    Dec 10 00:02:56 wrt54gs daemon.err pixelserv[8015]: recv: No data
    A final version will have fewer messages and more appropriate replies as discussion above.

    In test mode I've seen strange requests for example "GET /unpixel?id=1022562 HTTP/1.1" and others such as "GET /adi/N3814.yahoouk.mcaf/B5060429.4;sz=300x250;click=http://u..." which broke the parsing.

    Also seen long requests for example
    Code:
    received 1023 bytes
    'GET /__utm.gif?utmw ...
    ...
    method: 'GET'
    path[915]= '/__utm.gif?utmw...
    ext: '.gif'
    Sending Gif response
    
     
  32. ~nephelim~

    ~nephelim~ LI Guru Member

    I read that double fork is usually required in demon ( Once to disassociate a program from the terminial and another to disassociate it from the process group) and this bring forth the two _exit().

    As far I understood this followed the rule one exit for each entry into main so that exit could be used later on.

    c: to exit or to _exit? article clarify the difference in greater detail.

    From what I understood there are some internal/private cleanup functions that are supposed to be called only once (atexit and on_exit ) thus _exit provide an alternative when fork() is involved.

    Guess it would be possible to have handlers set only a variable and carry the actual action in the parent section of the fork if/switch.

    Adding masking too might be a safer approach but the examples mentioned in the articles had me confused. :frown:
     
  33. mstombs

    mstombs Network Guru Member

    sorry I don't really follow that "c: to exit or to _exit?" and we are not using c++, my understanding is that the child gets a copy of the parent's variable space on creation, so its not the same function being accessed, its the local copy - there are already actions in the code after forking - the child closes its copy of the listening file descriptor, the parent its copy of the new connection, so nothing shared from then on - so I suspect doesn't really matter in this example. I think it just needs a "signal (SIGUSR1, SIG_IGN)" in the child to make sure only the parent responds, but the child should have a 2-4 second life if its all working (but the recv could hang again I guess), and this would therefore be difficult to test!
     
  34. ~nephelim~

    ~nephelim~ LI Guru Member

    uclibc exit function finally calls _exit as well.

    some cleanup performed by exit is meant to be invoked for functions the developer registered though atexit. So there is a degree of discretionary purpose in exit.

    I don't really understand the difference caused by the additional cleanup what affects. exit calls _stdio_term would this be fine even with concurrently running childs+ parent?


    Adding to this syslog might not be async-safe and it is also called in other parts of the while loop upon errors. Not sure how much it does matter though. the whole signal literature is really difficult for me.


    [libvirt] PATCH: Block & reset signals when fork/exec'ing children address a related issue by masking before forking and unmasking afterwards (mention sigprocmask was replaced in case the library is called from multi thread apps).

    I was unable to find something better about disabling handler vs fork. But it looks like that SIG_DFL could be used to reset signal handlers in child instances.
     
  35. mstombs

    mstombs Network Guru Member

    V25 does block on the first recv, you can use "telnet 192.168.0.1 80" to talk to pixelserv - it will wait for ever for you to type something! So if connection broken before any data sent, easy to fix in ... V26, but this does give you a way to test all the error messages i.e. 'echo "hello world"|telnet 192.168.0.1 80' or signals while in this state! BTW already know 1023 too small for some paths in 'real use'.

    anything relevant re signals/exit in http://freshmeat.net/projects/libdaemon/ ?
     
  36. ~nephelim~

    ~nephelim~ LI Guru Member

    Couldn't signal reset be carried between accept and recv?

    Request will probably exceed 1023 bytes but the path request without ? fragment has 1019 bytes though it would not carry the \r\n.

    Though Something like that has been bugging me for some time. what would happen with fragmented packets <1023 (I guess it won't normally happen on the LAN though)?


    Perhaps this snippet from http://www.cs.odu.edu/~cs476/fall09/lectures/unixproc.pdf

    Code:
    if (pid == 0) { 
    signal(SIGINT, SIG_DFL);  
    signal(SIGQUIT, SIG_DFL);  
    signal(SIGCHLD, SIG_DFL);  
     
  37. mstombs

    mstombs Network Guru Member

    I challenge you to simulate that in V25 - will be possible in V26 with a timeout inbetween. We don't particularly want parent and child sending messages, but its not a big deal!

    I'm sure you are correct, what I mean is that the 1023 bytes didn't include all the first line of the request with the HTTP protocol bit. Guess we can ignore the HTTP bit and just monitor length of "/path/to/filename".

    Normally MTU on LAN Ethernet is 1500, I use 1400 as safe default for my ADSL modem. You could manually lower your lan interface MTU to test, but there's still a chance the kernel will wait and re-assemble before before it get passed to program. I wouldn't worry about it!

    Code:
    if (pid == 0) { 
    signal(SIGINT, SIG_DFL);  
    signal(SIGQUIT, SIG_DFL);  
    signal(SIGCHLD, SIG_DFL);  
    Agreed, to make sure only the single parent responds to the program defined actions for CHLD, TERM and USR1 signals, KILL should always get through.
     
  38. ~nephelim~

    ~nephelim~ LI Guru Member

    I probably misunderstood. Though in case you meant to use telnet 192.168.0.1 80 and have pixelserve wait for interactive input it works fine (v25 waits).

    I reset the handlers for the signal you mentioned just after forking.
    Code:
    if ( fork () == 0 ) {
    			/* this is the child process */
    			signal(SIGCHLD, SIG_DFL);
    			signal(SIGUSR1, SIG_DFL);
    			signal(SIGTERM, SIG_DFL);
    Though it looks like running killall -SIGUSR1 pixelserv will have the connection drop. :frown:

    Wouldn't this meant that resetting SIGUSR1 didn't actually take place? :confused:



    One thing that I noticed is that the counter is incremented even for failed attempts.

    Moving it in the test for send would be better.
    That was exactly my though. The fragment part ([?][a-z#]*) is not really needed so V25 trimming helps to cut the length strrchr will parse.

    The protocol error syslog would still help indirectly to monitor these cases.

    [/QUOTE]Normally MTU on LAN Ethernet is 1500, I use 1400 as safe default for my ADSL modem. You could manually lower your lan interface MTU to test, but there's still a chance the kernel will wait and re-assemble before before it get passed to program. I wouldn't worry about it![/QUOTE]


    I didn't really worry about that (no way I'm going to reduce the MTU) but I was considering how complex the receive end of an http daemon is and how parsing and error checking would be supposedly carried under the burden of this additional restriction.

    Currently I would be happy If I can manage to understand some more about signals.
     
  39. mstombs

    mstombs Network Guru Member

    I'll leave you to move the count to the child - problem is the parent is the one that keeps count, incrementing the child copy achieves nothing! So with selective replies can only really report "no of connection attempts" accepted - I did change the text from "Pixels served" to "Requests served" which isn't quite right!
     
  40. ~nephelim~

    ~nephelim~ LI Guru Member

    Oops. I should have realized sooner the full extent of fork effects.
    Guess something like this will be a royal pain as well.

    Only way in theory would be to carry this through sigchld handler but then again signals... :frown:
     
  41. mstombs

    mstombs Network Guru Member

    Actually this should be straightforward - the status in exit(status) should be available to the sigchld handler - and this make sense why zombies exist - to give their dying words to the parent before they are "reaped"...
     
  42. ~nephelim~

    ~nephelim~ LI Guru Member

    I did a preliminary test (just count EXIT_SUCCESS) but I'm sure it has some bug lurking somewhere.

    Code:
    #ifdef DO_COUNT
    volatile sig_atomic_t count = 0;
    volatile sig_atomic_t status = EXIT_FAILURE;
    #endif
    
    ...
    
    	case SIGCHLD :
    		while ( waitpid (-1, &status, WNOHANG) > 0 ) 
    		( status ) ? : count++ ; 
    		return;
    
    ...
    
    #ifdef DO_COUNT
    //		count++;
    #endif
    
    
    EDIT2: probably status is something more than a return code. as far I read WIFEXITED and WEXITSTATUS would be needed.

    A while ago I came across mention that volatile sig_atomic_t might yield pretty small values (eg sizeof char maybe signed as well)

    Plain variables leave the signal race-condition mess open.

    EDIT: uclibc sig_atomic_t appear to be large enough

    Code:
    # define SIG_ATOMIC_MIN         (-2147483647-1)
    # define SIG_ATOMIC_MAX         (2147483647) // now that's a whopping lot of ads!
    
    I cannot find some article I read to confirm if a sigchd could be interrupted even from another sigchd (I don't really remember exactly but it was another recommendation of sort.)
     
  43. mstombs

    mstombs Network Guru Member

    You need those macros to extract status - I've seen an example. Multiple SIGCHLDs not a problem that's why there's a while loop in the handler, there may be more than one to reap.

    Don't know about "volatile sig_atomic_t", simple ints declared outside above use should work!
     
  44. ~nephelim~

    ~nephelim~ LI Guru Member


    sig_atomic should be fine (checked uclibc header against the define constaints mentioned in the article).

    I started testing WEXITSTATUS(status) and tought of triggering EXIT_FAILURE using echo -e "GET "|nc 192.168.0.1 80.

    I got "null prot, path length = 1".

    uclibc's strtok doesn't quite behave. As far I understood char *path = strtok (NULL, " "); should have returned null :frown:
    EDIT: just to make sure it is not my toolchain (hndtools-mipsel-uclibc-4.3.5-2.tar.bz2) do you get the same result?

    using echo -e "GET"|nc 192.168.0.1 80 works as expected.
     
  45. mstombs

    mstombs Network Guru Member

    Can't keep up ... I'm actually using old Linksys toolchain for Tomato 1.28 at the moment, or Tomatousb 4.2.4, potential issue if you don't use same version as libraries...

    My testing confirms you need

    Code:
                signal(SIGUSR1, SIG_IGN);
    for child to ignore SIGUSR1.

    Have you changed the exit from child? My code always sets it to SUCCESS at moment?
     
  46. ~nephelim~

    ~nephelim~ LI Guru Member

    Was telnet dropping the connection using SIG_DFL (eg because recv was restarted) an indication that signal were passed to child as well?

    There [strike]are probably[/strike] could also be issues with uclibc's strtok ([strike]eg ext = strtok (ext, "?"); in V23 should not have returned the full string if no token was found[/strike] edit: I read strtok manual again)
    Please test echo -e "GET "|nc 192.168.0.1 80 case as well.

    One is there in V25 I can trigger this only if I send "GET" without a space.

    Code:
    			char *path = strtok (NULL, " ");
    			if (path == NULL) {
    				syslog (LOG_ERR, "null path");
    				exit (EXIT_FAILURE);
    			}
    char *method = strtok (buf, " "); should have transformed "GET \0" into "GET\0\0"
    char *path = strtok (NULL, " "); should have then returned null.

    in case strtok considered \0 a valid character strlen would have not ( instead len=1 )
     
  47. mstombs

    mstombs Network Guru Member

    Ahah that exit was an oversight I already changed...

    I now have

    Code:
    Dec 10 22:43:49 wrt54gs daemon.info pixelserv[10859]: 515 requests, 2 errors, 507 gif, 6 txt replies
    yes, presumably no default action for USR1, reading the manual has also allowed me to fix the breaking of the main accept loop using sig flag SA_RESTART.

    nc is sending an extra carriage return or newline

    Code:
    received 5 bytes
    'GET 
    '
    method: 'GET'
    path[1]= '
    '
    pixelserv[9354]: null prot, path length = 1

    attached as far as I've got - still need to generate error reply messages
     

    Attached Files:

  48. srouquette

    srouquette Network Guru Member

    i'm silently following this thread, a little message to say thanks for the updates ^^
     
  49. ~nephelim~

    ~nephelim~ LI Guru Member

    I thought that as user signal they would do nothing by default but found out USR1 and USR2 are supposed to terminate the application (as if there weren't enough signals for that :eek: ).

    Indeed it is possible to terminate pixelserv using killall -SIGUSR2 pixelserv


    Found out the 2 sec timeout. As an added bonus the current implementation provides indirectly also informations about pending requests (== spawned pixelservs). :)

    Code:
    Dec 11 13:00:01 router daemon.info pixelserv[15211]: 2 requests, 1 errors, 0 gif, 0 txt replies
    eg: requests - (gif+txt+errors) = 1 pending.


    My bad. I completely forgot echo adds a new line. :blush:



    Thanks. It works great.

    Code:
    Dec 11 11:41:41 router daemon.info pixelserv[13866]: 1000 requests, 0 errors, 1000 gif, 0 txt replies
    Found out that sig_atomic_t is nothing more than plain int.

    Nor that it appear necessary but wouldn't be possible to make it volatile to comply with signal recommendations? :confused:

    I thought error handling is quite complete.

    It would be possible to add HTTP 501 easily with a new value for tex and a new goto label in unknown method handler.

    "HTTP/1.1 501 Method Not Implemented\r\n"
    "Connection: close\r\n"
    "\r\n"

    would probably be enough.
     
  50. mstombs

    mstombs Network Guru Member

    added to avoid the pre V22 problem of stuck processes if nothing to read and connection broken. I assume if "select" reports something to read recv will not block, I did try "non-blocking" sockets before, don't think needed!

    Did they exist, and if so only for a couple of seconds?, could also mean children killed by other means than exit(status)

    Need to learn what "volatile" means in this context, placing declaration outside code effectively makes them "external static". They can also be "unsigned" to give another 2 billion counts! But I'm sure you are correct:-

    http://www.linuxfordevices.com/c/a/Linux-For-Devices-Articles/Do-you-volatile-Should-you/

    Currently 9 errors detected, I suggest to not reply at all if nothing received (as at present), if "GET not found could the 501 and then only send txt if positively identify .js default gif if unsure. Still wondering if alternative txt response could work better...

    If ignore the HTTP protocol but can simplify code a bit to reduce size

    Code:
                char *path = strtok (NULL, " ?;");
     
  51. ~nephelim~

    ~nephelim~ LI Guru Member

    I come across the 2 seconds timeout while testing USR1 (along with USR2) using telnet 192.168.0.1 80.

    For the 1 pending request I had to be fast enough to send USR1. Generalizing over this I thought that pending would be a way to get an idea of active request not yet completed.

    Under the condition I carried the test (telnet wait) pending state was expected and the counter has been updated as soon the request was completed.

    I didn't really consider how fast are requests provided AFAIK there is a limit (getrlimit RLIMIT_NPROC) of 128 forked processes (so the maximum concurrent load is capped).


    An unsigned int will be probably atomic as well.

    Mention of volatile for use in signal handlers was made in SIG31-C. Do not access or modify shared objects in signal handlers.

    That was where I initially got the idea sig_atomic_t could have sizeof char size though it was instead specified it is implementation defined.

    This would do. after all .js in most cases are scripts and gifs would be fine in most of the other cases.

    501 response appear the better option without increasing complexity (once stated a method is unuspported clients will not use it anymore)

    The step in question would be B2 in this flow-chart of server responses.


    That would be fine as well. Truncation might include also # for (http://server/page.thm#section urls).
     
  52. mstombs

    mstombs Network Guru Member

    You can easily change the 2s timeout to test, I copied the 2sec for connection close from apache (concerns over DOS), generally on a Lan if there isn't an instant response there won't be one (I wish Microsoft could learn that, they clearly have multiple 30sec timeouts in some places!).

    Good catch on "#", have you come across a definitive list of characters that will mark the end of filename, currently have " ?;#" definite but could add "():*<>[]='\"\\,|!~" for example. Not a big issue, if not simple probably not going to like Gif or txt response!

    Now I need to lookup "atomic" in this context...
     
  53. ~nephelim~

    ~nephelim~ LI Guru Member

    2 second is fine. Under similar considerations a week or so ago I searched if it was possible to similarly reduce TIMEWAIT but I finally desisted.
    It doesn't look possible to set TIMEWAIT duration for a singe application :frown:


    As path separators only ? " " and # would really count.

    Code:
    scheme://username:password@domain:port/path?query_string
    scheme://username:password@domain:port/path#fragment_id
    scheme://username:password@domain:port/path?query_string#fragment_id
    
    ; or & and = are commonly used in the query so they will always follow ?

    Reading and writing this data type is guaranteed to happen in a single instruction
     
  54. mstombs

    mstombs Network Guru Member

    OK, thanks ~nephelim~, I think I have included everything now, and had this version on domestic test for a few hours in 'real use'

    Have re-arranged the code to avoid "goto"'s and made sure the various selective compile options work again

    Current stats collected are

    Code:
    Dec 12 22:01:52 wrt54gs daemon.info pixelserv[13110]: 840 requests, 8 errors, 11 bad, 364 gif, 457 txt replies
    without too many complaints...

    An error is when a connection is made and closed/broken before any data received or sent.

    A bad request is something other than "GET", for example "POST", for these a 501 response is sent.

    Null text response are sent when a .js* file requested, otherwise the usual null gif.

    To keep size down, here is a binary compiled with tomatousb, and old tomato toolchain (.oldT) and only a few options: (recompile with -DVERBOSE to get more messages in log)

    Code:
    /tmp/home/root # ./pixelserv -h
    Usage:./pixelserv [IP No/hostname (all)] [-n i/f (all)] [-u user ("nobody")]
    ...
    ./pixelserv 192.168.1.1 -n br0
    Code:
    Dec 12 21:57:50 WRT54G-TM daemon.info pixelserv[2890]: ./pixelserv V27 compiled: Dec 12 2010 21:49:05 from pixelserv27.c
    Dec 12 21:57:50 WRT54G-TM daemon.notice pixelserv[2892]: Listening on br0 192.168.1.1:80

    Code:
    kill -SIGUSR1 $(pidof pixelserv)
    or

    Code:
    kill -USR1 $(pidof pixelserv)
    or

    Code:
    killall -USR1 pixelserv

    to get stats into log
     

    Attached Files:

  55. ~nephelim~

    ~nephelim~ LI Guru Member

    Code:
    Dec 13 11:18:30 router daemon.info pixelserv[2343]: 563 requests, 14 errors, 0 bad, 526 gif, 23 txt replies
    Thanks for the new release. Works great. :)
     
  56. Jedis

    Jedis LI Guru Member

    Just tried it and upon trying to start it:

    # ./pixelserv
    ./pixelserv: can't load library 'libgcc_s.so.1'
     
  57. mstombs

    mstombs Network Guru Member

    You are not using tomatousb then, please try pixelserv.oldT - rename without the ".oldT", this was compiled with old Linksys toolchain as used Stock Tomato, and should therefore point to the correct library. The new toolchain produces smaller binaries.
     
  58. Jedis

    Jedis LI Guru Member

    I was wondering what those files were for! Thanks, working great now :)
     
  59. mstombs

    mstombs Network Guru Member

    Long time no update, and no need for you to if above version does what you want.

    There is a bug in V27 in that it stops working when certain tomato gui config changes are made. It appears that this is because of the SO_BINDTODEVICE socket option used which must look up a binary code ID of the interface that matches the name when the program starts, and doesn't track any changes. So by default I no longer compile with the option IF_MODE or support the "-n br0" type parameter.

    New feature in this version is the option to load the image response string from a file, with "-f warning.bin" the file must include the appropriate http header, with "-g warning.gif" it adds the correct header for a gif file. The size of gif is not restricted by the program, but watch-out many copies may be held in ram at any time.

    Code:
    root@easy:/tmp/mnt/myusb/share# ./pixelserv -h
    Usage:./pixelserv [IP No/hostname (all)] [-u user ("nobody")] [-f response.bin] [-g name.gif]
     
    root@easy:/tmp/mnt/myusb/share# ./pixelserv 192.168.10.254 -g 48px-dialog-warning.gif
    pixelserv[5321]: ./pixelserv V30 compiled: Nov  6 2012 09:13:44 from pixelserv30.c
     
    ...log...
    Nov  6 09:41:08 unknown daemon.info pixelserv[5321]: ./pixelserv V30 compiled: Nov  6 2012 09:13:44 from pixelserv30.c
    Nov  6 09:41:08 unknown daemon.notice pixelserv[5323]: Listening on 192.168.10.254:80
    Nov  6 09:42:42 unknown daemon.info pixelserv[5323]: 2 requests, 0 errors, 0 bad, 2 gif, 0 txt replies
    
    When used with an adblock scheme, such as the one in http://www.linksysinfo.org/index.php?threads/all-u-need-ad-blocking.33191/ the option to display something other than a null gif is useful for testing to show where something is happening - but note the script in that thread currently uses "-n br0" option.

    V30 since interim versions have been posted on dd-wrt and also been using this as example code on a RaspberryPi
     

    Attached Files:

  60. Bird333

    Bird333 Network Guru Member

    I am trying to get this running with All U Need ad blocking script. It seems pixelserv is not running. I extracted the latest zip file from Windows 7 to my usb drive at /opt/pixelserv. I put this script in WAN up
    Code:
    ps | grep [p]ixelserv
    if [ $? == 1 ]; then
        /opt/pixelserv/pixelserv 192.168.22.2
    fi
    All I am getting are 'server not found' errors from the browser. I'm sure there is something simple I am not doing right. Can someone help?
     
  61. rhester72

    rhester72 Network Guru Member

    Is 192.168.22.2 the IP you put in the adblocking script?
     
  62. Bird333

    Bird333 Network Guru Member

    Yes.
     
  63. Bird333

    Bird333 Network Guru Member

    Looking in /var/log/messages and this is what I see
    Code:
    root@DD-WRT:/tmp/var/log# cat messages | grep -i pixelserv
    Nov 28 16:14:43 DD-WRT daemon.info pixelserv[1101]: /opt/downloads/pixelserv/pixelserv V30 compiled: Nov  6 2012 09:13:44 from pixelserv30.c
    Nov 28 16:14:43 DD-WRT daemon.err pixelserv[1103]: Abort: Cannot assign requested address
    Nov 28 16:14:54 DD-WRT user.notice root: ADBLOCK ERROR: cannot start pixelserv
    root@DD-WRT:/tmp/var/log#
     
  64. rhester72

    rhester72 Network Guru Member

    Do you in fact have that IP assigned to an interface? Output from "ip addr" would be handy here.

    Rodney
     
  65. Bird333

    Bird333 Network Guru Member

    No I don't. I thought the script(s) would take care of what was needed.
     
  66. Bird333

    Bird333 Network Guru Member

  67. mstombs

    mstombs Network Guru Member

    Unless the adblock script has been changed you can't use V30 which no longer supports the "-i" parameter. But are you using dd-wrt? A different script is required there since the web gui by default hogs Port 80 on the router - any IP address, it may even have been Rodney above who recommended the tomato patch to bind the web gui to the IP address not the interface which makes this possible on Tomato!

    The dd-wrt ad-block script on http://www.howtogeek.com/51477/how-to-remove-advertisements-with-pixelserv-on-dd-wrt/ moves the dd-wrt web gui off Port 80, but puts an iptables divert in so you can still access it where expected.
     
  68. Bird333

    Bird333 Network Guru Member

    No, I am running Tomato. I'll try the previous script.
     
  69. Bird333

    Bird333 Network Guru Member

    V27 works fine.
     
  70. tismon

    tismon Serious Server Member

    If I may ask out of ignorance, what is the benefit of pixelserv over saving a 1x1 gif (called say pix.gif) to /var/wwwext and calling it with http://*router_IP*/ext/pix.gif assuming that the web interface is enabled on port 80 for LAN? Is this more efficient or more capable?

    Thank you
     
  71. jerrm

    jerrm Network Guru Member

    Pixelserv is designed to respond regardless of the url it receives.

    The redirect for the ad block script is based on dns intercepts - I don't think the packet data is being touched. For the built in server to respond gracefully, the url would have to be rewritten, or some sort of 404 response script installed.

    Then there is the issue of the authentication requirement, not sure if that can be disabled for a folder or not in the stock httpd- I've never looked into the built in server that much.
     
  72. Frequenzy

    Frequenzy Networkin' Nut Member

    whats the latest version that i could use and where can i download it? using pixelserv v18 on shibby build 102. sometimes v18 doesnt work when changes are made on the configs. running pixelserv on usb
     
  73. mstombs

    mstombs Network Guru Member

    V27 is most commonly used, by the adblock script for example http://www.linksysinfo.org/index.ph...run-on-router-wrt54g.30509/page-2#post-149413 It also has the bug on certain GUI changes the lan bridge interface ID gets changed without it noticing, but a workaround is to always restart the app in the firewall script.

    V30 removes that interface option, so won't work with the published adblock script until it gets updated. It also introduces a new feature where the nullgif can be replaced with an arbitrary gif
     
  74. Frequenzy

    Frequenzy Networkin' Nut Member

    does v27 also works when pixelserv is run via USB
     
  75. mstombs

    mstombs Network Guru Member

    The location will not make a difference, if it runs and fails to set up the ip address and port it should put an error message in log. If you have path wrong (not available when accessed) or binary not executable then the there will be a script level message only - try running from console (telnet/ssh) to see what it says.
     
  76. Frequenzy

    Frequenzy Networkin' Nut Member

    thanks for the input. i just tried and it works :)
     
  77. srouquette

    srouquette Network Guru Member

    well, the pb with v30 is it doesn't have the -n option (to specify br0).
     
  78. fubdap

    fubdap Addicted to LI Member

    @srouquette - what do you thing about the new adblock by Haarp? Are you going to give it a try. For a very long time you have been the main support behind ALL-U-NEED Ad Blocking script.
     
  79. srouquette

    srouquette Network Guru Member

    yep, I saw Haarp's script.
    it's always nice to see people trying new things, that's how things improve :)

    ALL-U-NEED Ad Blocking fits my needs, and if Haarp think it's bloated with useless stuff, I guess it can't be helped.

    I agree ALL-U-NEED is slow, because it removes duplicates after merging the list, 'sort -u' isn't enough for that.

    I thought he wanted to keep it simple, but with the latest version you have to install it on a partition and what not.

    So yeah... ALL-U-NEED wasn't the first, and it won't be the last.
    people will use what they prefer and it's fine that way, I hope it will breed a better adblock script :)
    ALL-U-NEED works fine for me. But I'm using a different setup, I'm building the list with it on my ubuntu (cron), upload it on a ftp server, and get the list on my router withtout optimizing it, it's faster this way.
     
    philess likes this.
  80. mstombs

    mstombs Network Guru Member

    The "-n" option was a reason for occasional hangs, the code grabs a handle for the interface, but doesn't know how to track if the interface is redefined by gui change for example. v30 also has the option to load an image in place of the null gif from file
     
  81. haarp

    haarp LI Guru Member

    Heh. Sorry, man. I meant no offense! Your script just didn't sit right with me, from the first time I looked at it.

    Yeah, I wanted to keep it simple, but people want features and always find unique ways of breaking it ;) Well, that, and I had to admit that WAN-up was limiting the size and flexibility. I'm sorry for those who really just wanted a WAN-Up script, but I figured if they already had to install pixelserv, they might aswell go all the way. If it wasn't for pixelserv, this wouldn't have been an option for me.

    I actually think that sort -u is enough to kill duplicates. What other scripts do (yours too?) is remove the subdomains and block the entire domain. Sure, you can do that, if you don't care about false positives. I'd rather have some duplicates than blocking something that shouldn't be, however.
     
  82. srouquette

    srouquette Network Guru Member

    only for duplicated subdomains like:

    http://winhelp2002.mvps.org/hosts.txt

    Code:
    127.0.0.1  s02.flagcounter.com
    127.0.0.1  s03.flagcounter.com
    127.0.0.1  s06.flagcounter.com
    127.0.0.1  s07.flagcounter.com
    127.0.0.1  s08.flagcounter.com
    127.0.0.1  s09.flagcounter.com
    127.0.0.1  2.s09.flagcounter.com
    127.0.0.1  s10.flagcounter.com
    
    -> flagcounter.com

    never had a false positive with 30k domains in my list.
     
  83. roadkill

    roadkill Super Moderator Staff Member Member

    added sticky seems to be a must ;)
     
  84. eahm

    eahm LI Guru Member

    Apple uses linksynergy.com to link to iTunes. To me that's a false positive. But even Google shopping links for Newegg etc., false positive with ads are personal I guess :)
     
  85. srouquette

    srouquette Network Guru Member

    false positive like that is something else and is related to the blocklist you use, that's why you can whitelist some domains if needed.
    But it isn't related to the script.
     
  86. haarp

    haarp LI Guru Member

    Well, I don't want to go too off-topic in here, but if a blocklist blocks ads.legitdomain.com, and the script then blocks legitdomain.com, I would indeed call that a script issue :/
     
  87. srouquette

    srouquette Network Guru Member

    but it doesn't work like that, it only removes the subdomain when there are multiple entries with the same domain, like flagcounter.com.
    if it didn't work, I would fix it...
     
  88. Kelet

    Kelet Reformed Router Member

    Trying to grab PixelServer v30, and I apparently don't have the permissions. Edit: I guess I'm in need of moderator approval.

    LinksysInfo.org - Error
    You do not have permission to view this page or perform this action.
     
  89. mstombs

    mstombs Network Guru Member

    downloads are available to anyone logged in, could be hosted elsewhere if you still have a problem.
     
  90. pharma

    pharma Network Guru Member

    Just updated from PixelServer v27 to v30 with no issues. Everything is working perfectly!:)
     
  91. jbcdidgosir

    jbcdidgosir Serious Server Member

    Do you want to create a Web server on Tomato? I have already realized it with python socket.
     
  92. jgeezy24

    jgeezy24 Reformed Router Member

    Moderator...may I have approval to download the latest build? Thank you.
     
  93. HunterZ

    HunterZ LI Guru Member

    Just to get this in the correct thread:

    There's another open-source (GPLv2) program out there called nullserv, which was inspired by pixelserv: https://github.com/flexiondotorg/nullserv

    It has one main advantage over pixelserv: It can serve up null JPG, PNG, SWF (flash) and text files in addition to null GIF files, each chosen by matching on ~16 filename extensions in the URL that is being intercepted.

    Unfortunately it currently only supports running as an inetd plugin and not as a standalone daemon (my understanding is that pixelserv supports both modes), which limits is ability to run on a router (lancethepants has already ported it for us and posted about it in another thread, but the inetd requirement means that you will probably need entware/optware/whatever: Script: Clean, Lean and Mean Adblocking).

    The author of nullserv is open to the idea of cross-pollinating ideas from pixelserv and nullserv, regardless of which one serves as a base for the other's improvements, but he doesn't currently have time to look into it himself.

    I may poke at it since I am a software engineer and know C, but I can't make any promises either - especially since I also don't currently have a toolchain for compiling Tomato-compatible binaries (lancethepants' native build environment looks more promising than some of the cross-compiler HOWTOs I've seen, but only if I can make it coexist with entware in some fashion).
     
  94. techlifeweb

    techlifeweb Reformed Router Member

    Same problem. Tried Chrome and Firefox. I am a new member so maybe that's it.
     
  95. Toxic

    Toxic Administrator Staff Member

    Before anyone can do something on this site, all New members are moderated. this is the last defence on Spambots and advertisers. Sorry but this is policy. we have approx 100-200 spambots a day here trying to register so I have enforced policies and processes that try to block this and keep the forums clean of spam.
     
  96. mstombs

    mstombs Network Guru Member

    I have a hacked pixelserv.c into a V31 with a version of the nullserv responses, but haven't yet tested on a router, need to test since many version of Ubuntu, Mint & Debian have been and gone from my disks since last version!

    Finding it hard to find combinations of browser and website that are improved. I have seen an error message in past where browser (probably IE) tried to process a gif as a script, but maybe that browser bug (ignoring the http header content) is now fixed?
     
  97. HunterZ

    HunterZ LI Guru Member

    Not quite sure what preventing downloads will do to stop spam, but maybe you can't manage permissions at that level of granularity?

    In the meantime, I've installed entware xinetd on my RT-N16 and got lancethepants' nullserv binary to work with it via the following /opt/etc/xinetd.conf:
    Code:
    defaults
    {
            instances               = UNLIMITED
            log_type                = SYSLOG authpriv
    #        log_on_success          = HOST PID
            log_on_failure          = HOST
            cps                     = 50 10
    }
    
    service www
    {
            flags = IPv4
            socket_type = stream
            protocol = tcp
            wait = no
            user = root
            server = /cifs1/adblock/nullserv
            disable = no
            only_from = 127.0.0.1 192.168.0.0/16
    }
    
    includedir /opt/etc/xinetd.d
    
    The only thing I don't like about it so far is that redirected HTTPS URLs bounce off of my router's web GUI since I'm still running that on port 443. It looks like pixelserv has a solution for this, though.
     
  98. mstombs

    mstombs Network Guru Member

    Don't know how to respond to an https request - the adblock scripts solve this problem by running pixelserv off a secondary interface/IP br0:0 - not the router lan ip. You can easily change the web gui https port though, then there is just nothing on router ip port 443.
     
  99. HunterZ

    HunterZ LI Guru Member

    Yes, that was the solution I was referring to.
     
  100. jerrm

    jerrm Network Guru Member

    That will work with xinetd too. Use the same ifconfig commands to setup a second IP and then use xinetd's bind parameter pointing to the new address.
     

Share This Page