i discovered some REALLY weird issue (dd-wrt 11/1)

Discussion in 'DD-WRT Firmware' started by flexy, Nov 9, 2005.

  1. flexy

    flexy Network Guru Member

    i run dd-wrt v23 , built 11/1

    I also tweaked my rc_startup which reads (nvram get rc_startup)
    echo "300 1800 120 60 120 120 10 60 30 120" > /proc/sys/net/ipv4/ip_conntrack_tcp_timeouts

    this is the standard tweak for limiting the time the router holds stale connections. In seconds.

    I can also confirm those settings with cat /proc/sys... so those settings sem to be applied right.

    Someone said to check current number of connections with
    cat /proc/net/ip_conntrack | wc -l

    this gives me (right now) approx 190 or so connections.

    If i type cat /proc/net/ip_conntrack

    i see MANY connections like

    {read further, the funny part comes below !!!}

    -------SNIP ---------

    tcp 6 356097 ESTABLISHED src= dst= sport=4842 dpor
    t=50032 src= dst= sport=50032 dport=4842 [ASSURED] use
    =1 rate=1 mark=0
    udp 17 19 src= dst= sport=4672 dport=10878 [UNREP
    LIED] src= dst= sport=10878 dport=4672 use=1 rate=32
    tcp 6 357755 ESTABLISHED src= dst= sport=50586 dpo
    rt=50032 src= dst= sport=50032 dport=50586 [ASSURED] u
    se=1 rate=53 mark=0

    I see many references and ESTABLISHED (!) connections obviously on port 50000+ which is my bittorrent ports.

    The KICKER is that i did not have any torrents going on today (only yesterday)...and my computer was even off over night.

    There should NOT be ANY references to old bittorrent ports/coinnections - ESPECIALLY since i even used the tweaks in rc_startup and limit my connection timeouts.


    IF YOU WONDER...many problems, eg. slowdowns, memory loss of the router.....check what connections are active on the router....btw. as said before it says (right now) 190+ connections active.....in windows itels i can only see 4 FOUR !!!!

    This is some serious bugs ! Those connections have to die...respective the connection timeouts need to work. Also, in addition to the onme in the rc_startup etc. i also use the timeouts (300) in router gui ! Still - tthose old connections are there right now !
  2. robmack

    robmack Network Guru Member

    Observations about TTLs in /proc/net/ip_conntrack

    I did some playing around with cat /proc/net/ip_conntrack on my router. Running 08/11/05.

    I think the third field is the TTL of the entry. It sems to me (and I'm only guessing) that the persistent connections have extremely high TTLs. Generally, those connections have a large number in the third field. I was looking at my table and I have a generally quiet router. Connections I know I established early this afternoon are still in the process fs;they shouldn't be there but they have a large TTL value of 407901. Connections that have numbers like 17 or 153 generally disappear after a minute or two.

    # cat /proc/net/ip_conntrack
    tcp 6 408154 ESTABLISHED src= dst= sport=44161 dport=80 [UNREPLIED] src= dst= sport=80 dport=44161 use=1 rate=0 mark=0
    ... a few minutes later...
    tcp 6 407902 ESTABLISHED src= dst= sport=44161 dport=80 [UNREPLIED] src= dst= sport=80 dport=44161 use=1 rate=0 mark=0

    The third field is decremented by about 250, about the same time in seconds between issuing the commands.

    Maybe these huge TTLs are causing connections to eat up memory in the router, causing instability.
  3. Toxic

    Toxic Administrator Staff Member

    are these connection though from BT clients outside hitting your WRT54G as hey are looking for files even if your PC is off you cannot stop internet clients from trying to access your WRT54G.

    i would at a guess think most are from outside ie WAN. trying to connect.

    maybe timeouts are broken?

    try the IP Filter Settings in management.asp maybe they may help?
  4. robmack

    robmack Network Guru Member

    Possible timeouts broken

    In my example, is an external system and is the DHCP-supplied address for my external interface from my ISP. So, this is an inbound connection to port 80. There is no port forwarding going on in the router for port 80 except that is a DMZ box so it gets forwarded the connection attempt.
  5. itsmeohmy

    itsmeohmy Network Guru Member

    Re: Possible timeouts broken

    This seems to be ok and working with my copy of DD-WRT from today 11/8/05. It would be great if Brainslayer could add this tweaked file with the stock DD-WRT installation. It would greatly improve speed for everyone not knowing of it.
  6. flexy

    flexy Network Guru Member

    Re: Observations about TTLs in /proc/net/ip_conntrack

    nice observations...i am pretty sure you're right.

    So..where do those TTLs come from ???

    I did some more testing, just double checking my router settings, router reboot....ul/dl some torrents.....then quit bittorrent and waited 40mins.

    Still have a bunch of those connections in there with those insane numbers and see them counting down the seconds as you said.

    Any idea how to prevent that ???
  7. robmack

    robmack Network Guru Member

    Re: Observations about TTLs in /proc/net/ip_conntrack

    Reading on the web, ip_conntrack is related to netfilter and lists all tracked / masqueraded connections, similar to 'ipchains -L -M' in 2.2.x kernels.

    Connection tracking by default handles up to a certain number of simultaneous connections. This number is dependent on you system's maximum memory size (at 64MB: 4096, 128MB: 8192, ...).

    You can easily increase or lower the number of maximal tracked connections, but be aware that each tracked connection eats about 350 bytes of non-swappable kernel memory!

    To increase this limit to e.g. 8192, type:

    echo "8192" > /proc/sys/net/ipv4/ip_conntrack_max

    So, maybe this will help your router's stability to reduce the number of tracked connections. I don't know the downside to reducing the number of tracked connections however. More info about connection tracking in iptables can be found here:
  8. flexy

    flexy Network Guru Member

    Re: Observations about TTLs in /proc/net/ip_conntrack

    i dont know if the maximum number of those connections is even an issue...i never reach 4096/8192 or something anyway.

    The problem is the duration/TTL. Those connections have a TTL of 120hrs.

    I also think i have some value in ip_conntrack_max already
  9. robmack

    robmack Network Guru Member

    Reading that description of how connection tracking works, there were a couple of points in it that apply here. First, ASSURED connections are not dropped from the state table when the connection is under load and, second, the TTL is reset to the maximum when traffic is detected on the connection. Looking at my examples, the connection was ESTABLISHED but did not get a reply. I think the initial traffic reset the TTL value to the kernel's maximum (guessing here) which is somewhere around 120hrs. The router is now waiting for a FIN/ACK to tear down the connection, which it will never get. So, the connection will time out after a few days.

    I don't think there is anything that you can do at the user level to modify the max TTL value. This is a kernel issue. The only thing I can suggest is that someone file a bug report asking for a feature change where they implement tcp window tracking in the kernel.

    My suggestion of reducing the value in ip_conntrack_max will limit the amount of memory used by iptables conn tracking, thus making the router more stable (more non-pagable kernel memory available) at the risk of firewall statefullness. You have a value today in ip_conntrack_max. Try to reduce it and see what happens.
  10. robmack

    robmack Network Guru Member

    Possible work around for the TCP established timeout

    Here is a possible work around for reducing the number of TCP established connections in your box.

    First, there is a lot of info about conntrack at http://iptables-tutorial.frozentux.net/iptables-tutorial.html#STATEMACHINE

    In there, they said there were IPTABLEs variables that can be manipulated to adjust the timeouts. The variables are in /proc/sys/net/ipv4/netfilter. The DD-WRT does have this directory.

    You can list the various timeouts with the following script:

    cd /proc/sys/net/ipv4/netfilter
    for i in *; do
    echo -n "$i "
    cat $i

    There is a variable called ip_conntrack_tcp_timeout_established. That variable is default set to 432000. Set it to something lower like 4000 with the command:

    echo "4000" >ip_conntrack_tcp_timeout_established

    This will reduce the default TCP established timeout from 432000 seconds to 4000 seconds.

    Experiment with different settings first to see the effect of reducing the timeouts before committing the changes permanently. If an established connection times out before it finishes, you may not regain the session (iptables will block packets for non-established connections) and will have to restart it. This may or may not work in all applications.

    To make this change permanent through reboots, enter it into the Administration-> Diagnostics-> Command Shell Parameters section of the DD-WRT control panel.
  11. flexy

    flexy Network Guru Member

    oh nice ! thanks !



    the question here is...how can a two day old bittorrent connection be established and assured *right now* ?

    The numbers itself make sense, well an established and assured connection might very well have a 5 day TTL.....but the underlying problem is why this connection is still established and existing.

    Bittorrent is of course not running anymore for over a day now...

    I dont get it.

    btw. do i understand this right ? I am kinda hesitating setting the value for an established connection....if i were to set it to, say, one hour or so.....this would effectively mean each connection would be lost each hour and then (succesfully or not) tried to re-established (as i understood it)

    thx anyway...nice info !
  12. dd2k

    dd2k Network Guru Member

    What I guess is, the problem is a bug in your torrent client. The program is supposed to "tell" the router to close the current connections to the peers when it finishs the job. If the client fails to do so, it will leave the connection established/open until TTL. I use bittorrent official client which is almost buggy free and don't have too much problem of this. I also use a p2p tv program, which I believe is buggy, has the exactly same problem as yours.

    My suggestion is to change your torrent client and see how the thing goes.

  13. robmack

    robmack Network Guru Member

    I guess you're aksing why are inactive TCP connections still showing up in the conntrack table and the answer is that the states of those two systems are independant of each other; but loosly related.

    The connntrack state machine tracks TCP/IP connection states to the best of its ability so that iptables can statefully accept or deny IP traffic more intellegently than what is possible with a basic packet filtering mechanism. When a client initiates a connection it sends out a SYN packet which initiates a NEW connection entry in the contrack state machine table. When the remote answers with an ACK, the entry becomes ESTABLISHED. When the client finishes with a SYN/ACK, the ESTABLISHED connection becomes ASSURED. When the server is finished with the connection, it starts a close sequence, which at its conclusion changes the contrack entry to CLOSED. Then, for some predefined period after that, the contrack entry remains valid until it is torn down. Those default values are listed in Table 7.2 on the page refered to in the link above.

    Those connections that are hanging around long after the TCP connection has dissappeared are in the contrack table because they never received a FIN sequence or RST packet from the remote server to tear down the TCP connection. The application probably timed out but the conntrack table entry remained, waiting for that elusive FIN packet.

    BTW, the TCP and UDP timeout values in the DD-WRT page I think manage the variable /proc/sys/net/ipv4/ip_contrack_tcp_timeouts. I don't know how these relate to iptables or the ip_contrack mechanism itself.

    I think that is right. If the contrack entry's TTL expires, then the firewall will reject further packets for that TCP connection. So you have to select a TTL that, in most cases, will exceed the lifetime of the majority of TCP connections. Other protocols are connectionless so their lifetimes are much shorter.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice