Neighbor table overflow

Discussion in 'Tomato Firmware' started by Sean B., Apr 5, 2018.

  1. Sean B.

    Sean B. LI Guru Member

    Hey guys. I started getting neighbor table overflow lines in the syslog for unknown reasons:

    Code:
    Apr  2 23:20:10 Storage user.warn kernel: printk: 150 messages suppressed.
    Apr  2 23:20:10 Storage user.warn kernel: Neighbour table overflow.
    I bumped up the max connections a bit, a long with hash and buckets. Upped the GC values as well:

    Code:
    net.ipv4.neigh.default.gc_thresh1 = 512
    net.ipv4.neigh.default.gc_thresh2 = 1024
    net.ipv4.neigh.default.gc_thresh3 = 2048
    And this seems to have eliminated the overflow logs. However, I was left curious as to why the default limits were being exceeded. As my family and I don't torrent, or do much internet activity that is connection-count intensive. So I created a cronjob to run a script that poll's /proc/sys/net/netfilter/nf_conntrack_count every 5 minutes. If it get's a value over 500 it then saves a copy of /proc/net/ip_conntrack for later viewing. What this contained has me puzzled:

    Code:
    root@Storage:/tmp/mnt/Flash/Log# cat OVERFLOW.log | grep 192.168.1.47 | grep -m 2 192.168.1.1
    ipv4     2 tcp      6 88 TIME_WAIT src=192.168.1.47 dst=192.168.1.1 sport=61726 dport=19516 src=192.168.1.1 dst=192.168.1.47 sport=19516 dport=61726 [ASSURED] mark=0 use=2
    ipv4     2 tcp      6 25 TIME_WAIT src=192.168.1.47 dst=192.168.1.1 sport=61143 dport=19516 src=192.168.1.1 dst=192.168.1.47 sport=19516 dport=61143 [ASSURED] mark=0 use=2
    root@Storage:/tmp/mnt/Flash/Log# cat OVERFLOW.log | grep 192.168.1.47 | grep -c 192.168.1.1
    1106
    root@Storage:/tmp/mnt/Flash/Log#
    
    1106 entries from the family desktop at 192.168.1.47 to the router. All in TIME_WAIT, all with differing ports on the desktop side but same 19516 port on the router side. No idea what this is or what is causing it. Not sure of a good way to continue narrowing it down, as with its randomly occurring nature any steps taken will have to be automated in some fashion. Any insight or ideas are welcome.
     
  2. koitsu

    koitsu Network Guru Member

    This message does not have to do with conntrack (NAT state table entries) or TCP requests. It has to do with ARP cache entries (what Linux calls a "neighbour table"). It's too bad the kernel doesn't give some more details in the log entry (e.g. actual numbers), but a script running every minute in a cronjob (logging output from the below) could probably shed light on things.

    The Linux arp(7) man page covers details regarding what each /proc tunable (e.g. gc_threshX) controls. This is a setting that is individually controlled per IP stack -- IPv4 and IPv6 have separate /proc tunables. There's a German web page describing the gc_threshX settings a bit more clearly (Google Translate does an OK job here); just remember there is no sysctl on TomatoUSB, only /proc tunables.

    ip neigh show should show you the contents of the table. You may be more familiar with arp -a -n.

    My notes say that for people who are dual-homed (i.e. both IPv4 and IPv6 are enabled), the default thresholds tend to get hit on occasion. Many ISPs (Comcast in particular) are incredibly chatty ARP-wise with IPv6. My notes reference this post of mine:
    http://www.linksysinfo.org/index.php?threads/ipv6-and-comcast.38006/page-2#post-184563

    I've used this for quite some time reliably, which also handles if IPv6 is disabled in Tomato:

    Code:
    # Increase ARP cache sizes and GC thresholds; may alleviate "Neighbour table
    # overflow" warnings that some users are seeing.
    #
    # http://www.linksysinfo.org/index.php?threads/ipv6-and-comcast.38006/page-2#post-184563
    #
    echo  256 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
    echo 1024 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
    echo 2048 > /proc/sys/net/ipv4/neigh/default/gc_thresh3
    
    # IPv6 adjustments for the above.
    #
    if [ -n "`nvram get ipv6_service`" ]; then
      echo  256 > /proc/sys/net/ipv6/neigh/default/gc_thresh1
      echo 1024 > /proc/sys/net/ipv6/neigh/default/gc_thresh2
      echo 2048 > /proc/sys/net/ipv6/neigh/default/gc_thresh3
    fi
    
    I'd also suggest you make sure that rp_filter is enabled across all of your interfaces. Tomato by default enables this, but some people may turn it off manually, which could (depending on network setup/configuration) lead to excess entries. The link to rp_filter should explain its purpose. To review the /proc tunables, for i in /proc/sys/net/ipv[46]/conf/*/rp_filter; do printf "%-50s" "$i"; cat $i; done should do the trick. I can explain the difference between all and default if asked.
     
  3. Sean B.

    Sean B. LI Guru Member

    Woops, I left these out:

    Code:
    Apr  2 23:20:09 Storage user.warn kernel: nf_conntrack: table full, dropping packet.
    Apr  2 23:20:10 Storage user.warn kernel: nf_conntrack: table full, dropping packet.
    
    Where neighbor overflow was ARP, best I could tell these were conn limits, so I raised values for both. But perhaps both are related to ARP? There we far far less of these than neighbor overflow lines. Only a couple here and there, compared to pages of neighbor ones.

    I've been using sysctl via optware to set variables for quite some time. I find it much easier than navigating the /proc tree. Looks like we came to the same result for a script:

    Code:
    #!/bin/sh
    LIST="net.ipv4.neigh.default.gc_thresh1=512
    net.ipv6.neigh.default.gc_thresh1=512
    net.ipv4.neigh.default.gc_thresh2=1024
    net.ipv6.neigh.default.gc_thresh2=1024
    net.ipv4.neigh.default.gc_thresh3=2048
    net.ipv6.neigh.default.gc_thresh3=2048"
    
    if [ "$(sysctl -n net.ipv4.neigh.default.gc_thresh1)" != 512 ]
      then {
        for values in $LIST
        do
          sysctl -wq $values
        done
      }
    fi
    Perhaps the connections I spoke of are coincidental to the overflows, but seemingly out of place none the less. I'll read through the links you posted and see if I can determine more places that may containing better information I can set up a capture/script to obtain.

    Don't think I've got a good grasp on the functional difference. So whenever you have time, your info would certainly be appreciated.
     
  4. koitsu

    koitsu Network Guru Member

    I did some digging in the kernel. Kernel version (thus CPU architecture) matters when encountering this message. I compared Toastman-NRT (Linux 2.6.22) to kille72 (Linux 2.6.36):

    Linux 2.6.22 -- and this message is only emit for IPv4 per release/src/linux/linux/net/ipv4/route.c:
    Code:
    printk(KERN_WARNING "Neighbour table overflow.\n");
    
    Linux 2.6.36 -- IPv4 per release/src-rt-6.x.4708/linux/linux-2.6/net/ipv4/route.c:
    Code:
    printk(KERN_WARNING "ipv4: Neighbour table overflow.\n");
    
    Linux 2.6.36 -- IPv6 per release/src-rt-6.x.4708/linux/linux-2.6/net/ipv6/route.c:
    Code:
    printk(KERN_DEBUG "ipv6: Neighbour table overflow.\n");
    
    2.6.22's log message doesn't indicate IP protocol version -- but the message is only emit for IPv4.

    2.6.36's log messages indicate IP protocol version. And for IPv6, it only emits this message if klogd/syslogd is configured to actually log messages that are of kern facility, debug priority.

    The below paragraph I've written is hard to understand, I know, but...

    Tomato runs two daemons for logging: syslogd and klogd. These are both part of Busybox (directory sysklogd). syslogd handles syslog data from userland programs, specifically anything using the syslog() libc function (or maybe it has its own, I don't know). klogd handles reading data from /proc/kmsg (and siphoning it through a call to syslog()), as well as adjusting kernel log level (through the -c argument). The more I dug through this code (both syslogd.c and klogd.c) the more confused I got, so I've basically concluded -- going completely off of code, not by testing -- that by default you would not see the IPv6 neighbour overflow messages (in dmesg or /var/log/messages). I found the man page on klogctl(3) to be useful, and a great example of how parts of Linux are a complete and total mess. :D
     
  5. koitsu

    koitsu Network Guru Member

    There's probably some uncomfortable relationship between the ARP cache/neighbour table overflow and the conntrack table (NAT state table) overflowing, given that the log messages came in at the same time. I'd really have to sit down and try to understand the Linux 2.6.22 route/ipv4.c code to know what it's really iterating over. It could be as simple as "known routes with ARP entries" that also recursively iterates across, say, a TCP session table. I don't know.

    I can't tell you what's going on with your network. But I can tell you that there could be firewall rules being utilised that are creating excessive numbers of conntrack entries for LAN traffic that might not otherwise be necessary. I think I've mentioned this in a post somewhere in my past, but I have no desire to go dig it up. I've always questioned the "design" of Tomato's firewall rules as a whole (not saying there are rules "causing problems", it's more like omission of certain rules, and lack of understanding on the parts of people who keep doing things like adding services to the router itself). I talk more about how to avoid that below.

    I could speculate for days, but my gut feeling is that this is one of the "side problems" with running services on the router itself. Since these were TCP connections, you should be able to find out what daemon they're talking to by using the -p flag to netstat -- logging something like netstat -a -n -t -p at the same time as your conntrack check would be ideal. I would not be surprised if these are UPnP/NAT-PMP, CIFS/SMB, DLNA or "media services", etc.. Anything is possible here. The fact there's a common destination port on the router -- TCP port 19516 -- should be helpful. I bet, however, that port number is dynamic (i.e. if you were to reboot the router whatever daemon is listening on it picks a different port). Otherwise what you saw could be a classic TCP SYN attack from a client on your network to your router. It's literally impossible to tell from the data provided.

    The maximum entries in the conntrack table is in /proc/sys/net/netfilter/nf_conntrack_max but this setting shouldn't be played with necessarily (IIRC it's dynamic at kernel boot and based on several system resources, including memory -- pushing it past its max is not a wise idea necessarily). It's better to find out what's causing the grief and stop it.

    There are a substantially large number of expiries you can alternately apply for TCP sessions that are tracked in the conntrack table. Faster expiration means the table can contain less entries (ideally) -- but like the above, I don't generally recommend this. Advanced -> Conntrack/Netfilter -> TCP Timeout. If you had a large number of connections which were all in TIME_WAIT, you could consider shortening it, but I believe the default (120) is pretty aggressive. In general, Tomato's conntrack TCP (and UDP) values are incredibly aggressive compared to a normal TCP stack.

    Ideally for traffic of this sort (LAN traffic traffic destined at the router itself), you want to bypass the conntrack table. There's an iptables target called NOTRACK that is for this exact purpose, and it is included in TomatoUSB. It can only be used in the raw table. You need to be extremely careful when using this -- I strongly suggest matching against source and/or destination IPs (i.e. the router or NVRAM lan_ipaddr), as well as relevant interface (ex. br0 or NVRAM lan_ifname). You don't want to NOTRACK the wrong kind of traffic. Here are some random resources online talking about NOTRACK:

    * https://distracted-it.blogspot.com/2015/05/iptables-firewall-rules-for-busy.html
    * https://security.stackexchange.com/...fe-to-disable-connection-tracking-in-iptables

    Good luck.

    Edit: removed link to strongarm.io blog. I really do not like the advice being given there. There's good information, but bad execution. I don't get the impression he understands the security nightmare that could result as a result of some of said rules, so I removed the link.
     
  6. Sean B.

    Sean B. LI Guru Member

    Ding! You win. I had checked active/listening ports at the time I was inspecting the log but 19516 didn't show up, nor did a quick google for that port number turn up anything. Yet I happened to run it again and there it is:

    Code:
    tcp        0      0 *:19516                 *:*                     LISTEN      19208/miniupnpd
    Not sure if miniupnpd was dead at the time I looked before, or if I simply glazed right over it. Something must be amiss on the desktop, as no other devices ( laptops, cell phones, tablets, xbox one etc ) on the network go nuts with connection counts like that. Thanks for all the great info as usual Koitsu!
     
  7. koitsu

    koitsu Network Guru Member

    Possibly SSDP was responsible? I dunno what 192.168.1.47 on your network is, but if it's a Windows box, it sounds like it may have lost its mind for a while. Or some software on there was really banging away on UPnP traffic (it's a surprisingly verbose protocol). But that dynamic port number really sounds like it's a dynamic port forward of some sort. Does an entry for it show up under Port Forwarding -> UPnP/NAT-PMP ? If so, sometimes the description field can shed light on what did it (this data comes from the UPnP-aware client itself). There's no messages from miniupnpd in /var/log/messages around the same time frame?

    Related-yet-not-related: if you want to disable conntrack for traffic directed to/from the LAN side of the router, then this is pretty simple to do. In Scripts -> Firewall (followed by service firewall start if you want to avoid a reboot):

    Code:
    # Disable nf_conntrack entries for LAN traffic directed to (or originating
    # from) the router itself.  This can help alleviate conntrack table exhaustion
    # if running a service/daemon on the router.
    #
    lan_ifname=$(nvram get lan_ifname)
    lan_ipaddr=$(nvram get lan_ipaddr)
    
    iptables -t raw -A PREROUTING -i $lan_ifname -d $lan_ipaddr -j NOTRACK
    iptables -t raw -A OUTPUT     -o $lan_ifname -s $lan_ipaddr -j NOTRACK
    
    Before doing just blindly using that though, be sure to check what those NVRAM variables return + ensure it'll work with your network configuration.
     
    Last edited: Apr 5, 2018
  8. Sean B.

    Sean B. LI Guru Member

    A Win10 desktop

    There are no entries showing currently. However, I have "Inactive rule cleaning" enabled, so there may very well have been an entry at the time this occurred and has sense been flushed.

    None. I'm going to run a continuing capture of that port on the router with dumpcap. Perhaps the packets will give some detail as to what's behind such a connection count.

    Is there much reason for this not being default? I would think this would save a fair amount of resources for the router, especially on some of the cpu/ram lacking routers Tomato runs on. Great advice! I was going to see if this would cut it down:

    Code:
    iptables -t filter -I INPUT -s 192.168.1.47 -p tcp -m connlimit --connlimit-above 300 -j DROP
    
    Just to cap the amount it loads up until I got it figured out, but your suggestion covers the entire field of tracking LAN only traffic.
     
  9. koitsu

    koitsu Network Guru Member

    Follow-up on this, as it's something I came cross today, specifically with regards to the script in post #7:

    Not all iptables tables are flushed (nuked) when doing Save in the GUI: only mangle, nat, and filter are. raw is left untouched. As such, my script in post #7 will keep appending rules to the PREROUTING and OUTPUT chains every time this happens, which is (obviously) bad.

    Here's an updated version that checks to see if the rules are already in place, and only adds them if missing. It keys off the PREROUTING rule only -- because the odds of only one of the pair being present is pretty unlikely:

    Code:
    # Disable nf_conntrack entries for LAN traffic directed to (or originating
    # from) the router itself.  This can help alleviate conntrack table exhaustion
    # if running a service/daemon on the router.  Details:
    # http://www.linksysinfo.org/index.php?threads/neighbor-table-overflow.74104/
    #
    # The raw table is not managed (flushed/nuked and restored) by TomatoUSB,
    # which means we can't assume the below chains are empty.
    #
    lan_ifname=$(nvram get lan_ifname)
    lan_ipaddr=$(nvram get lan_ipaddr)
    
    if ! iptables -t raw -C PREROUTING -i $lan_ifname -d $lan_ipaddr -j NOTRACK 2>/dev/null
    then
      iptables -t raw -A PREROUTING -i $lan_ifname -d $lan_ipaddr -j NOTRACK
      iptables -t raw -A OUTPUT     -o $lan_ifname -s $lan_ipaddr -j NOTRACK
    fi
    
    Two footnotes and an alternate (crappier) way to solve this:

    1. This won't handle situations where, for example, lan_ifname or lan_ipaddr change via the GUI, but it should be sufficient for most use cases.
    2. Ideally rc/firewall.c should be updated to save the raw table to /etc/iptables (restoration being done by iptables-restore and ip6tables-restore), but that's a bunch of effort I don't feel like coding right now -- look at mangle_table(), nat_table(), and filter_table() for general details.

    There's a "simpler" script below, but I don't like it because it leaves a small window of time between -F and -A calls where flowing packets would end up creating some conntrack entries. Yes they'd expire eventually, but it's still unclean (and not something you'd want to do esp. on a network with high amounts of relevant traffic). Not recommended:

    Code:
    lan_ifname=$(nvram get lan_ifname)
    lan_ipaddr=$(nvram get lan_ipaddr)
    
    iptables -t raw -F PREROUTING
    iptables -t raw -F OUTPUT
    iptables -t raw -A PREROUTING -i $lan_ifname -d $lan_ipaddr -j NOTRACK
    iptables -t raw -A OUTPUT     -o $lan_ifname -s $lan_ipaddr -j NOTRACK
    
     
  10. Sean B.

    Sean B. LI Guru Member

    Is there a specific page in the GUI you're speaking of? I tried a save under Advanced->Firewall and the raw table rules didn't duplicate:

    Code:
    root@Storage:/tmp/home/root# iptables -t raw --list-rules
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -A PREROUTING -d 192.168.1.1/32 -i br0 -j NOTRACK
    -A PREROUTING -d 192.168.2.1/32 -i br1 -j NOTRACK
    -A OUTPUT -s 192.168.1.1/32 -o br0 -j NOTRACK
    -A OUTPUT -s 192.168.2.1/32 -o br1 -j NOTRACK
    
    **CLICKED SAVE IN GUI**
    
    
    root@Storage:/tmp/home/root# iptables -t raw --list-rules
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -A PREROUTING -d 192.168.1.1/32 -i br0 -j NOTRACK
    -A PREROUTING -d 192.168.2.1/32 -i br1 -j NOTRACK
    -A OUTPUT -s 192.168.1.1/32 -o br0 -j NOTRACK
    -A OUTPUT -s 192.168.2.1/32 -o br1 -j NOTRACK
    root@Storage:/tmp/home/root#
     
  11. koitsu

    koitsu Network Guru Member

    Making modifications to Administration -> Scripts will not trigger this. Only pages that trigger running of run_nvscript("script_fire", NULL); will do it. A common place is Administration -> Admin Access.
     
  12. Sean B.

    Sean B. LI Guru Member

    Tried save in Administration->Admin Access, didn't trigger duplicate rules for me either. What build are you running? I'm on a fairly customized v1.28.9008 Toastman-ARM-MOD K26ARM USB VPN-64K.
     
  13. koitsu

    koitsu Network Guru Member

    FreshTomato/kille72 2018.3 beta does it. Don't know if 2018.1 did or didn't, so it's very possible newer-ish Shibby builds or MultiWAN introduced this somehow. If this is a regression of some sort, then I should post about that in their thread. Either way, still an OK safety net to have in place anyway.
     
  14. Sean B.

    Sean B. LI Guru Member

    Absolutely, never hurts to add some checks. Nice catch! And thanks for the update.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice