1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

NGINX setup problem

Discussion in 'Tomato Firmware' started by 4char, Aug 5, 2014.

  1. 4char

    4char Network Guru Member

    I'm try to use nginx as reverse proxy server.
    I have two web server running on two machine in LAN side and try to setup nginx on the router to direct the traffic to those servers depends on where it coming from.
    I'm using Tomato RAF Firmware v1.28.9014 MIPSR2-RAF-v1.3g K26 USB on RT-N66U.

    I entered following into NGINX Custom configuration:
    Code:
    http {
        server_names_hash_bucket_size 64;
        server {
            listen 85;
            server_name www.domain1.com domain1.com;
            access_log /tmp/mnt/sda1/logs/nginx.access.log;
            error_log /tmp/mnt/sda1/logs/nginx.error.log;
            location / {
                proxy_pass http://192.168.1.100:80;
                proxy_redirect off;
                proxy_buffering off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            }
        }
       
        server {
            listen 85;
            server_name www.domain2.com domain2.com;
            access_log /tmp/mnt/sda1/logs/nginx.access.log;
            error_log /tmp/mnt/sda1/logs/nginx.error.log;
            location / {
                proxy_pass http://192.168.1.120:80;
                proxy_redirect off;
                proxy_buffering off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            }
        }
    }
    
    When I try to start the nginx from GUI (click the 'Start Now' button), it fails. The log in the /tmp/var/log/nginx/error.log shows that:
    Code:
    2014/08/05 13:12:34 [emerg] 24563#0: bind() to 0.0.0.0:85 failed (125: Address already in use) 
    2014/08/05 13:12:34 [emerg] 24563#0: bind() to 0.0.0.0:85 failed (125: Address already in use) 
    2014/08/05 13:12:34 [emerg] 24563#0: bind() to 0.0.0.0:85 failed (125: Address already in use) 
    2014/08/05 13:12:34 [emerg] 24563#0: bind() to 0.0.0.0:85 failed (125: Address already in use) 
    2014/08/05 13:12:34 [emerg] 24563#0: bind() to 0.0.0.0:85 failed (125: Address already in use) 
    2014/08/05 13:12:34 [emerg] 24563#0: still could not bind()
    
    I know there's nothing listening on the port 85 in router (also from 'netstat -nl').

    Does anyone knows how to fix this? Am I doing something wrong?

    Thx!
     
  2. koitsu

    koitsu Network Guru Member

    The problem might be that you have two server directives, both saying "listen on TCP port 85". Once the first is bound, the 2nd cannot bind. Otherwise the issue is that you didn't kill off nginx fully before trying to launch it again with new settings (don't "restart" it, actually fully stop/start it, make sure there are no lingering nginx processes once you kill it off too).

    My gut feeling is that you basically need one server clause, and then different conditionals for what you're trying to match against.

    What you want to match against, by the way, is essentially the HTTP Host: field that is submit by an HTTP client. I.e.

    http://someplace.com/ (HTTP Host: header of someplace.com) gets transparently redirected (under the hood) to http://192.168.1.100/ while http://ilikesnakes.com/ (HTTP Host: header of ilikesnakes.com) gets transparently redirected (under the hood) to http://192.168.1.120/

    And don't forget about the situation where HTTP host headers aren't submit by the client (legacy web browsers), if it applies to your environment. In that case there is nothing you can do -- there is no way to delineate the traffic.

    All of this is a question for the nginx community support folks, and has absolutely nothing to do with Tomato/TomatoUSB. I cannot figure out (based on their documentation) how to accomplish the above task because I lack familiarity with nginx, but it is extremely easy with Apache 2.x.
     
  3. lancethepants

    lancethepants Network Guru Member

    I also use nginx (though not on tomato) as a reverse ssl proxy (also handles ssl connection to plain http service running behind nginx).
    I have half-a-dozen "server" callouts done in similar fashion each with port 80 or port 443 callouts, several domains all running on the same set of ports (using sni for ssl usage).
    I'm not sure what's causing the issue you have, but it should be achievable.
     
  4. lancethepants

    lancethepants Network Guru Member

    Don't know if the way its compiled has anything to do with it. I'll take a look at it a little later if you haven' been able to resolve the issue.
     
  5. 4char

    4char Network Guru Member

    Thanks for all the replies. I got kind of workaround.
    @koitsu gave me some ideas to try. What I found is that if I set the "Web Server Port" (in GUI) to a different port number (say port 86) than what I needed (port 85), it works. So that means the default rules/directories are somehow interfere with the custom settings.
    Here's the nginx.conf file it generated (located in /tmp/etc/nginx/nginx.conf):
    Code:
    # NGinX generated config file
    user   root;
    worker_processes   1;
    worker_cpu_affinity   0101;
    master_process   off;
    worker_priority   10;
    error_log   /tmp/var/log/nginx/error.log;
    pid   /tmp/var/run/nginx.pid;
    worker_rlimit_nofile   8192;
    events {
       worker_connections   512;
       }
    http {
       include   /tmp/etc/nginx/mime.types;
       include   /tmp/etc/nginx/fastcgi.conf;
       default_type application/octet-stream;
       log_format main '$remote_addr - $remote_user [$time_local]  $status ''"$request" $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';
       sendfile on;
       server {
         listen   86;
         server_name   testserver.com;
         access_log   /tmp/var/log/nginx/access.log   main;
         location   /   {
           root   /www;
           index   index.html   index.htm   index.php;
           error_page 404   /404.html;
           error_page 500   502   503   504   /50x.html;
           location   /50x.html   {
             root   /www;
           }
           location ~ \.php$ {
             root   /www;
             fastcgi_pass   127.0.0.1:9000;
             fastcgi_index   index.php;
           }
           location ~ ^/(images|javascript|js|css|flash|media|static)/  {
             root   /www;
             expires 10d;
           }
         }
       }
    }
    
    # NGINX Custom Parameters.
    http {
       server_names_hash_bucket_size 64;
       server {
         listen 85;
         server_name www.domain1.com domain1.com;
         location / {
          proxy_pass http://192.168.1.100:80;
          proxy_redirect off;
          proxy_buffering off;
          proxy_set_header Host $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         }
      }
    
       server {
         listen 85;
         server_name www.domain2.com domain2.com;
         location / {
          proxy_pass http://192.168.1.120:80;
          proxy_redirect off;
          proxy_buffering off;
          proxy_set_header Host $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         }
       }
    }
    
    May be I need to create the nginx.conf file by hand and use "Keep Config Files".

    Not sure how is this persist through power cycle or I need to re-generate it after power cycle. I'll try it later to see how/if that works.

    Thx!
     
  6. koitsu

    koitsu Network Guru Member

    Oh, I think I see the issue now, now that you posted your full config file.

    You have two separate http{} blocks. If you were to use the same port numbers in both of those, there would be a conflict -- essentially you're telling nginx "fork two http servers, one on port 85, and the other on port 85". The 2nd binding would fail because the first already is connected.

    If you moved everything into a single http{} block, with multiple server{} blocks (which can reference the same port number), that should work. But it depends on what you actually want behaviour-wise.

    So the reason using port 85 in one http{} block and port 86 in another worked is because you are essentially creating two separate HTTP servers using two http{} blocks. That's my guess anyway.

    /tmp is RAM. /etc is a symlink to /tmp/etc. There are some magic TomatoUSB internals that create certain things (from NVRAM variables) in /etc on startup.

    There are ways to persist file contents through reboots, and the easiest I can think of is to do something like this in Scripts / Init:

    Code:
    /bin/mkdir -p /etc/nginx
    /bin/cat <<- EOF > /etc/nginx/nginx.conf
    put the raw contents
    of your config file here
    EOF
    
    This has one danger: if you use the $ character anywhere in your config file contents (for example, in a regex), it will be parsed/expanded by the underlying shell as a variable. To inhibit that, do this instead (note the single quotes around EOF (not double quotes!)):

    Code:
    /bin/mkdir -p /etc/nginx
    /bin/cat <<- 'EOF' > /etc/nginx/nginx.conf
    put the raw contents
    of your config file here
    EOF
    
    Otherwise if you do need variable expansion in some spots, but not in others, then you need to escape the $ manually in your content, ex. \$ instead of just $.
     
  7. 4char

    4char Network Guru Member

    Thanks @koitsu.
    The problem is that tomatoRAF auto generate the first part of the nginx.conf file. I can't change that (unless I can overwrite the generated nginx.conf file, or the generated nginx.conf does not have the http directory). The 2nd part is the one I put in the NGINX Custom Configuration field and tomatoRAF appended to the nginx.conf file.
    May be tomato can make it not generate http directory if the Custom Configuration is not empty (that would be great! :) ).

    I'll take a look at the persist part. I don't think I have the '$' in the contents.

    Thx again @koitsu, great help!

    Edit: Sorry, I guess I didn't even look at my own code. :( it does have '$'!
     
    Last edited: Aug 6, 2014
  8. koitsu

    koitsu Network Guru Member

    Ah I see the predicament. Yeah, you don't have enough control over what's in the first http{} block to add the server{} parts you want.

    So yes, the easiest solution would be to use a separate http{} block and put things on a different port (either in the first http{} block (RAF-managed) or in your own http{} block). I know that makes things a bit more confusing for you, but there's no real other way to do it. :/
     
  9. 4char

    4char Network Guru Member

    @koitsu, it works! I just used your script and it completely replaced the nginx.conf file. So I can put anything I need in it and don't have to worry about what's in the generated file.
    Thanks again for all the help.
     
  10. koitsu

    koitsu Network Guru Member

    I wouldn't be so sure -- I imagine clicking Save in some parts of the GUI will overwrite the file. The only time Scripts / Init is called, if I remember right, is upon reboot. This is something to keep in mind.
     

Share This Page