1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Memory Leak

Discussion in 'Tomato Firmware' started by Riddlah, Aug 14, 2012.

  1. Riddlah

    Riddlah Networkin' Nut Member

    Hi Guys,

    Recently reflashed my Asus RT-N16 with shibby's build 100. Looking at the main status page I can see the free memory dropping constantly. Within a 6 hour window, I usually lose 40mb and I'm down to approximately 60mb free. I don't think it's an issue with the firmware as it does the same for toastman's. Usually with toastman's build it levels out around 95mb and stays constant.

    I did clear nvram settings and reconfigured from scratch. What I'm wondering is there a way to tell what could be causing it? Also, is it possible it's a hardware issue?

    Thanks,

    Riddlah


    ---
    Riddlah
     
  2. GhaladReam

    GhaladReam Network Guru Member

    I'm running build 100 on my RT-N66U. Mind you I have very few services running, but my Total / Free Memory is 249.74 / 237.41 (95.06%) after 21 hours uptime.

    When you clear NVRAM and reconfig from scratch, make your config settings one at a time and observe memory usage. I know bittorrent client and TOR both use a decent chunk of memory, especially bittorrent if you have a lot of torrents going.
     
  3. Riddlah

    Riddlah Networkin' Nut Member

    Thanks GhaladReam, I'll give that a try when I reconfigure it. As for the Bittorrent Client and TOR, I'm currently not using either of them. Just the regular DHCP Services, IPv6 Tunnel, DDNS, Virtual Wireless, Port Forwarding, File Sharing, VPN (OpenVPN & PPTP)

    *Checked earlier today, and I'm at 56mb free from 123mb
     
  4. Cyberian75

    Cyberian75 Network Guru Member

    I'm noticing the same thing. The problem is that once it reaches around 50%, the router reboots. I too set it up from scratch.

    I'm only utilizing DHCP, DNSCrypt, IPv6 Tunnel and DDNS.
     
  5. koitsu

    koitsu Network Guru Member

    Yes, but no tools that come with Busybox (thus Tomato/TomatoUSB) out-of-the-box will show you the information you need. Even commands like top -- specifically the Busybox version -- do not show you RES/RSS. Likewise, Busybox ps also does not provide a way to see RES/RSS. Both tools only show VSZ/VIRT, which is not helpful in this situation. As usual, Busybox = pile of junk.

    You could install Entware and then install htop (opkg install htop) + run htop --sort-key=RSS, which will provide the necessary information. How to install Entware + its pre-requisites is outside of the scope of this thread.

    There is also the possibility that the kernel contains a memory leak. cat /proc/meminfo would be helpful here. Please do not look at the output and start reaching your own conclusions; it requires someone familiar with the attributes to know what's important / what isn't. Output from lsmod would also be helpful.

    Absolutely not.
     
  6. Cyberian75

    Cyberian75 Network Guru Member

    Simple YouTube videos also cause leaks / reboots.
     
  7. Riddlah

    Riddlah Networkin' Nut Member

    Thanks koitsu,

    Here is the output from /proc/meminfo:

    root@MIRAGE-ASUS:/tmp/home/root# cat /proc/meminfo
    MemTotal: 126760 kB
    MemFree: 34108 kB
    Buffers: 19548 kB
    Cached: 20204 kB
    SwapCached: 0 kB
    Active: 33792 kB
    Inactive: 23932 kB
    HighTotal: 0 kB
    HighFree: 0 kB
    LowTotal: 126760 kB
    LowFree: 34108 kB
    SwapTotal: 0 kB
    SwapFree: 0 kB
    Dirty: 12 kB
    Writeback: 0 kB
    AnonPages: 17980 kB
    Mapped: 4240 kB
    Slab: 30208 kB
    SReclaimable: 1448 kB
    SUnreclaim: 28760 kB
    PageTables: 452 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    WritebackTmp: 0 kB
    CommitLimit: 63380 kB
    Committed_AS: 23292 kB
    VmallocTotal: 1015800 kB
    VmallocUsed: 3944 kB
    VmallocChunk: 1009688 kB

    Also, here is the output from htop (already had entware installed).

    root@MIRAGE-ASUS:/tmp/home/root# htop --sort-key=RSS

    CPU[|| 2.6%] Tasks: 34, 0 thr; 1 running
    Mem[||||||||||||||||||||||51/123MB] Load average: 0.00 0.06 0.09
    Swp[ 0/0MB] Uptime: 04:19:28

    PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
    1 root 15 0 1412 736 612 S 0.0 0.6 0:02.67 /sbin/init noinit
    290 root 15 0 752 240 196 S 0.0 0.2 0:00.02 hotplug2 --persis
    332 root 15 0 1396 216 164 S 0.0 0.2 0:00.00 buttons
    333 root 25 0 1364 244 192 S 0.0 0.2 0:00.01 console
    334 root 15 0 1728 432 360 S 0.0 0.3 0:00.04 /bin/sh
    336 root 15 0 1720 332 276 S 0.0 0.3 0:03.10 syslogd -L -s 102
    338 root 15 0 1720 300 240 S 0.0 0.2 0:01.85 klogd
    534 root 25 0 1720 244 184 S 0.0 0.2 0:00.00 telnetd -p 23
    536 root 15 0 1196 344 276 S 0.0 0.3 0:00.00 dropbear -p 22 -p
    552 root 15 0 1084 264 200 S 0.0 0.2 0:00.00 eapd
    556 root 15 0 1228 464 288 S 0.0 0.4 0:00.57 nas
    575 root 15 0 1776 1000 540 S 0.0 0.8 0:29.56 ntfs-3g -o iochar
    593 root 18 0 1740 404 320 S 0.0 0.3 0:00.00 crond -l 9
    595 root 18 0 1028 392 272 S 0.0 0.3 0:00.00 rstats
    603 root 18 0 1420 792 276 S 0.0 0.6 0:06.54 cstats
    605 root 25 0 756 292 236 S 0.0 0.2 0:00.01 pptpd -c /tmp/ppt
    607 root 15 0 764 308 248 S 0.0 0.2 0:39.32 /usr/sbin/bcrelay
     
  8. GhaladReam

    GhaladReam Network Guru Member

    I am not noticing this memory leak. I am using the services described by you guys above, but I am NOT using IPv6 Tunnel, DNSCrypt or OpenVPN. Does disabling any or all of these features fix the memory leak? Might be worthwhile to pinpoint what is causing it by disabling them one at a time.
     
  9. koitsu

    koitsu Network Guru Member

    Please be sure to use code blocks around your pastes in the future; loss of terminal formatting makes this very hard to read.

    That said, where's the problem? I see below the system is using roughly 50MBytes of 123MBytes total with a 4-hour uptime:

    Code:
    root@MIRAGE-ASUS:/tmp/home/root# htop --sort-key=RSS
     
    CPU[|| 2.6%] Tasks: 34, 0 thr; 1 running
    Mem[||||||||||||||||||||||51/123MB] Load average: 0.00 0.06 0.09
    Swp[ 0/0MB] Uptime: 04:19:28
    
    Output from free will probably show something similar, but with different numbers (htop does its calculations differently than what free looks at; free looks at /proc/meminfo, if I remember right).

    The htop output doesn't show any processes taking up excessive amounts of RAM in userland; the biggest is init taking up roughly 600KBytes.

    /proc/meminfo shows LowFree: 34108 kB which indicates the lowest amount of memory available/free in the past 4 hours was 34MBytes, which may be an indicator of some process that may have died + been restarted, or possibly kernel-level issue (I'm thinking this).

    Thus I am left to believe the RAM issue you may be experiencing (I need actual proof) over time is probably related to the Web Usage or IP Traffic features, since these store massive amounts of data in kernel memory space, which makes it very hard to examine. Please read this thread fully to understand that fact (don't skim it, read it -- I provide insights near the end):

    http://www.linksysinfo.org/index.php?threads/shibby-web-usage-history.38908/

    If you are using either of these features, disable them and reboot your router and see if the situation improves.

    Here's my RT-N16 for comparison, using Toastman build tomato-K26USB-1.28.0500.2MIPSR2Toastman-RT-N-VPN.trx. I do not use Web Usage or IP Traffic features.

    Code:
    root@gw:/tmp/home/root# htop --sort-key=RSS
      CPU[#**                                                3.2%]    Tasks: 20, 0 thr; 1 running
      Mem[|||||##*****                                    10/124MB]    Load average: 0.00 0.00 0.00
      Swp[                                                  0/0MB]    Uptime: 13 days, 07:45:36
     
      PID USER      PRI  NI  VIRT  RES  SHR S CPU% MEM%  TIME+  Command
        1 root      15  0  1376  700  576 S  0.0  0.6  0:39.70 /sbin/init noinitrd
      281 root      15  0  748  236  196 S  0.0  0.2  0:00.03 hotplug2 --persistent --no-coldplug
      320 root      15  0  1360  216  164 S  0.0  0.2  0:00.25 buttons
      321 root      25  0  1328  244  192 S  0.0  0.2  0:00.01 console
      322 root      18  0  1716  408  336 S  0.0  0.3  0:00.04 /bin/sh
      324 root      15  0  1708  332  276 S  0.0  0.3  0:00.04 syslogd -L -s 50 -b 1
      326 root      15  0  1708  304  244 S  0.0  0.2  0:00.02 klogd
      448 root      15  0  1712  300  232 S  0.0  0.2  0:00.48 telnetd -p 23
      451 root      15  0  1084  264  200 S  0.0  0.2  0:00.14 eapd
      454 root      15  0  1188  424  288 S  0.0  0.3  0:00.45 nas
      473 root      18  0  1724  404  324 S  0.0  0.3  0:00.01 crond -l 9
      475 root      20  0  1028  396  276 S  0.0  0.3  0:01.08 rstats
      650 root      15  0  872  332  252 S  0.0  0.3  0:00.00 dhcp6c -T LL vlan2
      733 nobody    18  0  1184  692  420 S  0.0  0.5 22:05.32 dnsmasq -c 1500 --log-async
      830 root      16  0  1724  400  320 S  0.0  0.3  0:00.56 udhcpc -i vlan2 -b -s dhcpc-event -H gw -m -S
      832 root      18  0  916  388  308 S  0.0  0.3  0:01.19 miniupnpd -f /etc/upnp/config
    4513 root      15  0  816  296  236 S  0.0  0.2  0:06.61 radvd
    4516 root      18  0  2680  476  328 S  0.0  0.4  0:00.04 httpd
    4979 root      15  0  1720  488  408 S  0.0  0.4  0:00.04 -sh
    4983 root      15  0  1632  796  612 R  2.0  0.6  0:00.14 htop --sort-key=RSS
     
    root@gw:/tmp/home/root# free
                total        used        free      shared      buffers
    Mem:        127000        24784      102216            0        2912
    -/+ buffers:              21872      105128
    Swap:            0            0            0
     
    root@gw:/tmp/home/root# cat /proc/meminfo
    MemTotal:      127000 kB
    MemFree:        102180 kB
    Buffers:          2912 kB
    Cached:          11112 kB
    SwapCached:          0 kB
    Active:          7644 kB
    Inactive:        8156 kB
    HighTotal:          0 kB
    HighFree:            0 kB
    LowTotal:      127000 kB
    LowFree:        102180 kB
    SwapTotal:          0 kB
    SwapFree:            0 kB
    Dirty:              0 kB
    Writeback:          0 kB
    AnonPages:        1784 kB
    Mapped:          1644 kB
    Slab:            5076 kB
    SReclaimable:      796 kB
    SUnreclaim:      4280 kB
    PageTables:        252 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    WritebackTmp:        0 kB
    CommitLimit:    63500 kB
    Committed_AS:    4608 kB
    VmallocTotal:  1015800 kB
    VmallocUsed:      3364 kB
    VmallocChunk:  1009788 kB
    
    P.S. -- It appears htop's --sort-key feature is completely broken. It obviously does not sort by RSS in your example or my example. More useless and broken software, time to uninstall it.

    Please do this instead:

    Code:
    opkg remove htop
    opkg install procps
    /opt/bin/ps auxw | sort -n -k6 -r
    
    This should sort all the processes by the 6th column, numerically, and reverse the sort order (so highest-RSS process is at the top).
     
  10. Riddlah

    Riddlah Networkin' Nut Member

    @koitsu

    The output from the previous post was after about 3 hours runtime from previously being rebooted.

    Here is the current output after running the command you provided. This was after a reboot around 5:00, I also disabled the IPv6 Tunnel previous to the reboot. Currently I am sitting at Total / Free Memory123.79 MB / 95.96 MB (77.52%) and it seems to be holding steady.

    Code:
    root@MIRAGE-ASUS:/tmp/home/root# root@MIRAGE-ASUS:/tmp/home/root# /opt/bin/ps au
    xw | sort -n -k6 -r
    -sh: root@MIRAGE-ASUS:/tmp/home/root#: not found
    root@MIRAGE-ASUS:/tmp/home/root# /opt/bin/ps auxw | sort -n -k6 -r
    nobody    1250  0.4 10.6  14000 13480 ?        S    17:02  0:16 dnsmasq --conf-file=/tmp/gen
    root      999  0.0  1.4  3072  1848 ?        S    17:00  0:00 /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
    root      696  0.0  1.1  3356  1496 ?        SNs  17:00  0:00 smbd -D
    root      683  0.0  0.9  2512  1252 ?        Ss  17:00  0:00 nmbd -D
    root      573  0.1  0.8  1848  1064 ?        S    17:00  0:06 ntfs-3g -o iocharset=utf8,noatime,nodev /dev/sda1 /tmp/mnt/Cruzer
    root      684  0.0  0.6  2460  800 ?        S    17:00  0:00 nmbd -D
    root      602  0.0  0.6  1388  792 ?        S    17:00  0:01 cstats
    root        1  0.0  0.5  1380  736 ?        Ss  16:59  0:01 /sbin/init noinitrd
    root      1931  0.7  0.4  1260  556 ?        Ss  18:02  0:02 dropbear -p 22 -p 2222 -a
    root      2155  0.0  0.4  1520  528 pts/0    R+  18:07  0:00 /opt/bin/ps auxw
    root      1938  0.0  0.4  1736  512 pts/0    Ss+  18:02  0:00 -sh
    root      978  0.0  0.3  2692  476 ?        S    17:00  0:00 httpd
    root      564  0.0  0.3  1196  456 ?        S    17:00  0:00 nas
    root      339  0.0  0.3  1728  432 ttyS0    Ss+  17:00  0:00 /bin/sh
    root      975  0.0  0.3    944  412 ?        S    17:00  0:01 miniupnpd -f /etc/upnp/config
    root      596  0.0  0.3  1740  404 ?        Ss  17:00  0:00 crond -l 9
    root      598  0.0  0.3    996  392 ?        S    17:00  0:00 rstats
    root      995  0.0  0.2  1736  376 ?        Ss  17:00  0:00 udhcpc -i vlan2 -b -s dhcpc-event -H MIRAGE-ASUS -m
    root      548  0.0  0.2  1196  336 ?        S    17:00  0:00 dropbear -p 22 -p 2222 -a
    root      335  0.0  0.2  1720  332 ?        Ss  17:00  0:01 syslogd -L -s 1024 -O /tmp/mnt/Cruzer/Logging/Messages -b 1
    root      2156  0.0  0.2  1720  308 pts/0    S+  18:07  0:00 sort -n -k6 -r
    root      609  0.1  0.2    764  308 ?        S    17:00  0:05 /usr/sbin/bcrelay -i ppp[4-9].* -o br0 -n
    root      337  0.0  0.2  1720  300 ?        Ss  17:00  0:00 klogd
    root      608  0.0  0.2    756  292 ?        S    17:00  0:00 pptpd -c /tmp/pptpd/pptpd.conf -o /tmp/pptpd/options.pptpd -C 6
    nobody    1092  0.0  0.2    732  288 ?        S    17:01  0:00 /tmp/mnt/Cruzer/pixelserv 192.168.1.100 -n br0
    root      798  0.0  0.2    808  276 ?        S    17:00  0:00 udpxy -S -p 4022 -c 10 -m vlan2
    root      560  0.0  0.2  1052  264 ?        S    17:00  0:00 eapd
    root      537  0.0  0.1  1720  244 ?        Ss  17:00  0:00 telnetd -p 23
    root      333  0.0  0.1  1364  244 ?        Ss  17:00  0:00 console
    root      290  0.0  0.1    752  240 ?        Ss  17:00  0:00 hotplug2 --persistent --no-coldplug
    root      788  0.0  0.1    776  228 ?        S    17:00  0:00 igmpproxy /etc/igmp.conf
    root      332  0.0  0.1  1364  216 ?        Ss  17:00  0:00 buttons
    root      407  0.0  0.0      0    0 ?        S<  17:00  0:00 [usb-storage]
    root      406  0.0  0.0      0    0 ?        S<  17:00  0:00 [scsi_eh_0]
    root        96  0.0  0.0      0    0 ?        S<  16:59  0:01 [mtdblockd]
    root        54  0.0  0.0      0    0 ?        S<  16:59  0:00 [aio/0]
    root        53  0.0  0.0      0    0 ?        S<  16:59  0:00 [kswapd0]
    root        52  0.0  0.0      0    0 ?        S    16:59  0:00 [pdflush]
    root        51  0.0  0.0      0    0 ?        S    16:59  0:00 [pdflush]
    root        24  0.0  0.0      0    0 ?        S<  16:59  0:00 [khubd]
    root        21  0.0  0.0      0    0 ?        S<  16:59  0:00 [kblockd/0]
    root        5  0.0  0.0      0    0 ?        S<  16:59  0:00 [khelper]
    root        4  0.0  0.0      0    0 ?        S<  16:59  0:00 [events/0]
    root        3  0.0  0.0      0    0 ?        S<  16:59  0:00 [ksoftirqd/0]
    root        2  0.0  0.0      0    0 ?        S<  16:59  0:00 [kthreadd]
    USER      PID %CPU %MEM    VSZ  RSS TTY      STAT START  TIME COMMAND
    root@MIRAGE-ASUS:/tmp/home/root#
     
  11. koitsu

    koitsu Network Guru Member

    Your dnsmasq process is taking up 13MBytes of memory (RES/RSS), while mine is taking up 420KBytes (even after 15 days of uptime).

    Possibly there is a memory leak in the version of dnsmasq used in the firmware you're using.

    I know that at least in the Toastman builds, a dnsmasq upgrade was done fairly recently (see the ChangeLog for details/when).

    The dnsmasq version in the firmware I use is:

    Code:
    root@gw:/tmp/home/root# dnsmasq --version
    Dnsmasq version 2.61  Copyright (c) 2000-2012 Simon Kelley
    Compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack
    
     
  12. Riddlah

    Riddlah Networkin' Nut Member

    looks to be the same version in both builds

    Code:
    root@MIRAGE-ASUS:/tmp/home/root# dnsmasq --version
    Dnsmasq version 2.61  Copyright (c) 2000-2012 Simon Kelley
    Compile time options: IPv6 GNU-getopt no-RTC no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack
     
  13. myersw

    myersw Network Guru Member

    I am running Shibby 100 now and was running 099 on a E4200. Do not have reboots but I am NOT using IPv6 Tunnel, DNSCrypt or OpenVPN. My wife and I watch youtube videos plus watch netflex movies and progams using a Roku box. Router has never rebooted. --bill
     
  14. mbreslin

    mbreslin Networkin' Nut Member

    At least I'm not the only one, hopefully someone figures it out, 97 seems rock solid though.

    I use this custom config in dnsmasq:

    Code:
    server=2001:4860:4860::8888
    server=2001:4860:4860::8844
    server=8.8.8.8
    server=8.8.4.4
    no-resolv
    strict-order
     
  15. Dark_Shadow

    Dark_Shadow Addicted to LI Member

    shibby found several problems with 99-100. Should recheck with build after 100. For now shibby's suggestion is to return to 97.
     
  16. Cyberian75

    Cyberian75 Network Guru Member

    It must be IPv6 Tunnel then.
     
  17. shibby20

    shibby20 Network Guru Member

    Did you use 100-AIO build?? On AIO ipv6 works correct and i dont see any of memory problem
     
  18. mbreslin

    mbreslin Networkin' Nut Member

    both 99-AIO and 100-AIO randomly rebooted for me. On 97-AIO now and up with 0 problems for 22 hours.
     
  19. Cyberian75

    Cyberian75 Network Guru Member

    Same here.

    I said it's IPv6 only because I hadn't tried turning it off before downgrading.
     
  20. Riddlah

    Riddlah Networkin' Nut Member

    Just figured I'd give an update on the memory leak I'm seeing

    Code:
    root@MIRAGE-ASUS:/tmp/home/root# /opt/bin/ps auxw | sort -n -k6 -r
    root    14602  0.0  1.4  3072  1848 ?        S    08:33  0:05 /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
    root    14239  0.0  1.1  3356  1484 ?        SNs  08:33  0:00 smbd -D
    root    14174  0.0  1.0  2508  1284 ?        Ss  08:33  0:00 nmbd -D
    root      573  0.1  0.8  1868  1088 ?        S    Aug14  3:05 ntfs-3g -o iocharset=utf8,noatime,nodev /dev/sda1 /tmp/mnt/Cruzer
    root    16965  0.0  0.6  1420  824 ?        S    11:41  0:12 cstats
    root    14175  0.0  0.6  2460  800 ?        S    08:33  0:00 nmbd -D
    root        1  0.0  0.5  1380  744 ?        Ss  Aug14  0:04 /sbin/init noinitrd
    nobody  16909  0.0  0.5  1212  696 ?        S    11:40  0:03 dnsmasq -c 1500 --log-async
    root    22176  6.5  0.4  1260  556 ?        Ss  19:50  0:01 dropbear -p 22 -p 2222 -a
    root    22192  0.0  0.4  1520  528 pts/0    R+  19:50  0:00 /opt/bin/ps auxw
    root    22181  0.0  0.3  1736  504 pts/0    Ss+  19:50  0:00 -sh
    root    16926  0.0  0.3  1376  504 ?        Ss  11:40  0:04 dnscrypt-proxy -d -P 40
    root    14585  0.0  0.3  2704  488 ?        S    08:33  0:00 httpd
    root    14098  0.0  0.3  1196  464 ?        S    08:33  0:00 nas
    root    14582  0.0  0.3    948  440 ?        S    08:33  0:11 miniupnpd -f /etc/upnp/config
    root      339  0.0  0.3  1728  432 ttyS0    Ss+  Aug14  0:00 /bin/sh
    root    14125  0.0  0.3  1736  400 ?        Ss  08:33  0:00 crond -l 9
    root    14127  0.0  0.3    996  388 ?        S    08:33  0:00 rstats
    root    14606  0.0  0.2  1736  376 ?        Ss  08:33  0:00 udhcpc -i vlan2 -b -s dhcpc-event -H MIRAGE-ASUS -m
    root      548  0.0  0.2  1196  344 ?        S    Aug14  0:00 dropbear -p 22 -p 2222 -a
    root    13950  0.0  0.2  1720  328 ?        Ss  08:33  0:09 syslogd -L -s 1024 -O /tmp/mnt/Cruzer/Logging/Messages -b 1
    root    22193  0.0  0.2  1720  308 pts/0    S+  19:50  0:00 sort -n -k6 -r
    root    14148  0.3  0.2    764  308 ?        S    08:33  2:10 /usr/sbin/bcrelay -i ppp[4-9].* -o br0 -n
    root    14157  0.0  0.2    816  304 ?        S    08:33  0:01 radvd
    root    13952  0.0  0.2  1720  296 ?        Ss  08:33  0:08 klogd
    root    14147  0.0  0.2    756  292 ?        S    08:33  0:00 pptpd -c /tmp/pptpd/pptpd.conf -o /tmp/pptpd/options.pptpd -C 6
    nobody    1092  0.0  0.2    732  288 ?        S    Aug14  0:00 /tmp/mnt/Cruzer/pixelserv 192.168.1.100 -n br0
    root    14422  0.0  0.2    808  276 ?        S    08:33  0:00 udpxy -S -p 4022 -c 10 -m vlan2
    root    14095  0.0  0.2  1052  264 ?        S    08:33  0:00 eapd
    root      537  0.0  0.1  1720  244 ?        Ss  Aug14  0:00 telnetd -p 23
    root      333  0.0  0.1  1364  244 ?        Ss  Aug14  0:00 console
    root      290  0.0  0.1    752  240 ?        Ss  Aug14  0:00 hotplug2 --persistent --no-coldplug
    root    14419  0.0  0.1    776  228 ?        S    08:33  0:00 igmpproxy /etc/igmp.conf
    root      332  0.0  0.1  1364  216 ?        Ss  Aug14  0:00 buttons
    root      407  0.0  0.0      0    0 ?        S<  Aug14  0:00 [usb-storage]
    root      406  0.0  0.0      0    0 ?        S<  Aug14  0:00 [scsi_eh_0]
    root        96  0.0  0.0      0    0 ?        S<  Aug14  0:01 [mtdblockd]
    root        54  0.0  0.0      0    0 ?        S<  Aug14  0:00 [aio/0]
    root        53  0.0  0.0      0    0 ?        S<  Aug14  0:00 [kswapd0]
    root        52  0.0  0.0      0    0 ?        S    Aug14  0:00 [pdflush]
    root        51  0.0  0.0      0    0 ?        S    Aug14  0:00 [pdflush]
    root        24  0.0  0.0      0    0 ?        S<  Aug14  0:00 [khubd]
    root        21  0.0  0.0      0    0 ?        S<  Aug14  0:00 [kblockd/0]
    root        5  0.0  0.0      0    0 ?        S<  Aug14  0:00 [khelper]
    root        4  0.0  0.0      0    0 ?        S<  Aug14  0:00 [events/0]
    root        3  0.0  0.0      0    0 ?        S<  Aug14  0:01 [ksoftirqd/0]
    root        2  0.0  0.0      0    0 ?        S<  Aug14  0:00 [kthreadd]
    USER      PID %CPU %MEM    VSZ  RSS TTY      STAT START  TIME COMMAND
    root@MIRAGE-ASUS:/tmp/home/root#
    
    Also, here is the display from the admin panel. As you can see, the free memory has dropped to 56.29MB over a 24-hour period.

    [​IMG]
     
  21. koitsu

    koitsu Network Guru Member

    Okay, so a process is not what's taking up all of your memory right now, which further points to something in the kernel. Need output from:

    free
    cat /proc/meminfo
    slabtop -o (requires that you have done opkg install procps)

    slabtop output is probably going to be the most helpful here... I hope. It may or may not tell me what in the kernel is actually allocating the memory; welcome to the world of UNIX and *IX kernels.

    Thanks.

    P.S. -- Your amount of free NVRAM is nearing low levels (4148 bytes; slightly more than 4KBytes), so that may be something you look into as well. It isn't related to the RAM utilisation issue however, so don't misunderstand. :)
     
  22. Riddlah

    Riddlah Networkin' Nut Member

    @koitsu

    Here is the output from the three commands
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# free
                total        used        free      shared      buffers
    Mem:        126760      115520        11240            0        17988
    -/+ buffers:              97532        29228
    Swap:            0            0            0
    
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# cat /proc/meminfo
    MemTotal:      126760 kB
    MemFree:        11184 kB
    Buffers:        17992 kB
    Cached:          9308 kB
    SwapCached:          0 kB
    Active:          19456 kB
    Inactive:        13000 kB
    HighTotal:          0 kB
    HighFree:            0 kB
    LowTotal:      126760 kB
    LowFree:        11184 kB
    SwapTotal:          0 kB
    SwapFree:            0 kB
    Dirty:              0 kB
    Writeback:          0 kB
    AnonPages:        5164 kB
    Mapped:          4532 kB
    Slab:            78452 kB
    SReclaimable:    1432 kB
    SUnreclaim:      77020 kB
    PageTables:        424 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    WritebackTmp:        0 kB
    CommitLimit:    63380 kB
    Committed_AS:    10300 kB
    VmallocTotal:  1015800 kB
    VmallocUsed:      3944 kB
    VmallocChunk:  1009688 kB
    
    The command 'slabtop -o' didn't seem to give an output. Just went to a blank command to enter commands. Here is the from slabtop alone.
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# slabtop
    Active / Total Objects (% used)    : 103619 / 110628 (93.7%)
    Active / Total Slabs (% used)      : 19459 / 19467 (100.0%)
    Active / Total Caches (% used)    : 84 / 122 (68.9%)
    Active / Total Size (% used)      : 77567.85K / 78257.50K (99.1%)
    Minimum / Average / Maximum Object : 0.01K / 0.71K / 4096.00K
     
      OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
    32100  32100 100%    0.19K  1605      20      6420K skbuff_head_cache
    31786  31786 100%    2.00K  15893        2    63572K size-2048
    17340  17340 100%    0.13K    578      30      2312K size-64
      8319  4447  53%    0.06K    141      59      564K buffer_head
      3930  3821  97%    0.13K    131      30      524K size-32
      3885  3885 100%    0.25K    259      15      1036K ip_dst_cache
      1891  1122  59%    0.12K    61      31      244K dentry
      1872  1815  96%    0.05K    24      78        96K sysfs_dir_cache
      966    846  87%    0.08K    21      46        84K vm_area_struct
      678    388  57%    0.01K      2      339        8K anon_vma
      672    549  81%    0.30K    56      12      224K nf_nat:help
      648    432  66%    0.16K    27      24      108K filp
      496    482  97%    0.50K    62        8      248K size-512
      480    453  94%    0.13K    16      30        64K size-96
      432    432 100%    0.23K    27      16      108K nf_conntrack:basic
      429    299  69%    0.29K    33      13      132K radix_tree_node
      377    356  94%    0.29K    29      13      116K inode_cache
      372    244  65%    0.30K    31      12      124K proc_inode_cache
      297    289  97%    0.34K    27      11      108K squashfs_inode_cache
      250    250 100%    0.38K    25      10      100K shmem_inode_cache
      203    63  31%    0.02K      1      203        4K biovec-1
      203    29  14%    0.02K      1      203        4K tcp_bind_bucket
      180    180 100%    0.13K      6      30        24K size-128
      160    121  75%    0.09K      4      40        16K kmem_cache
      150    150 100%    0.25K    10      15        40K size-192
      143    143 100%    4.00K    143        1      572K size-4096
      113    93  82%    0.03K      1      113        4K fs_cache
      113    47  41%    0.03K      1      113        4K fib6_nodes
      110    88  80%    0.34K    10      11        40K sock_inode_cache
      105    105 100%    0.25K      7      15        28K size-256
      101    101 100%    0.04K      1      101        4K pid
      101      5  4%    0.04K      1      101        4K eventpoll_pwq
      101    25  24%    0.04K      1      101        4K ip_fib_hash
        80    80 100%    1.00K    20        4        80K size-1024
     
    
    P.S. I've seen the amount of free space for nvram, and keeping a close eye on it.
     
  23. koitsu

    koitsu Network Guru Member

    The -o flag (also known as --once) is supposed to basically do one iteration of slabtop and then exit. It works fine on my setup; possibly there's a terminal emulation problem causing the issue you're having with it.

    Alternately, you can just do cat /proc/slabinfo and look at the output, or this:

    Code:
    cat /proc/slabinfo | awk '/slabdata/ { printf "%-20s%-10u%-10u%-10u%-15u%-15u\n", $1, $2, $3, $4, ($4*$2), ($4*$3) }' | sort -n -r -k5 | head -10
    
    ...which show you the top 10 slabinfo entries which are taking up the most of memory (based on active number of slab objects). You can change that sort command to sort -n -r -k6 if you want to see the top 10 slabinfo entries which are taking up the most of memory (based on the number of slab objects, not necessarily active).

    The last two columns in the output are objsize*activenumberofslabobjs and objsize*totalnumberofslabobjs (should be obvious from the awk command), represented in bytes.

    An example:

    Code:
    # cat /proc/slabinfo | awk '/slabdata/ { printf "%-20s%-10u%-10u%-10u%-15u%-15u\n", $1, $2, $3, $4, ($4*$2), (
    $4*$3) }' | sort -n -r -k5 | head -10
    size-2048          336      348      2048      688128        712704
    size-4096          154      154      4096      630784        630784
    size-65536          7        7        65536    458752        458752
    size-8192          43        43        8192      352256        352256
    size-32            1920      1920      128      245760        245760
    dentry              1612      1612      124      199888        199888
    buffer_head        2960      3009      64        189440        192576
    proc_inode_cache    456      456      312      142272        142272
    size-131072        1        1        131072    131072        131072
    inode_cache        377      377      296      111592        111592
    
    So as you can see on my router, size-2048 is taking up a total of around 7MBytes, with size-4096 taking up a total of 6.3MBytes, etc. etc...

    Anyway, regardless, here is the root cause:

    Code:
      OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
    31786  31786 100%    2.00K  15893        2    63572K size-2048
    
    This is absolutely, 100%, something within the kernel allocating all of your memory. That's 64MBytes of memory right there, and if you break down the rest of the slabcache you'll see that bits/pieces go to other things.

    So your next question is obviously going to be "what is size-2048?" The kernel can allocate memory (for itself, drivers, modules, etc.) in separate sized chunks. Those chunks vary in size. The number you see after the hyphen is the allocation page size, so in this case, 2048-byte allocations. The problem is that you have 31,786 of those allocations. 31786 * 2048 = 65097728 bytes.

    So how do you figure out what's allocating all of those size-2048 pages? Wonderful question. I don't know how to do this on Linux (I do know how to do this on FreeBSD).

    But if I had to take a guess, it would be this (which I have mentioned in this thread already):

    http://www.linksysinfo.org/index.php?threads/shibby-web-usage-history.38908/

    I'm betting this is the root cause, because it's the only thing in the kernel (that shouldn't, IMO, be in the kernel) that can "grow out of control".

    Please disable the Web Usage and IP Traffic features in your configuration, reboot your router (the reboot is absolutely required), and then report back in a few days or a week (whatever).

    Remember that TomatoUSB is based on a very old version of the Linux kernel (almost 6 years old, I think) -- 2.6.22.19. Digging around Google I found lots of examples of people complaining about memory leaks in the kernel and related drivers:

    http://www.gossamer-threads.com/lists/lvs/users/18299
    http://old.nabble.com/Ethernet-bridge-leaking-memory-td1429520.html
    http://serverfault.com/questions/408104/where-is-memory-gone-no-not-buffers-or-cache
    https://forum.openwrt.org/viewtopic.php?pid=62568

    Blah blah blah, you get the idea. Tracking this down will probably require a kernel debugger, and because these are embedded devices, that is extremely hard to accomplish.
     
  24. Cyberian75

    Cyberian75 Network Guru Member

    097 has no problem with identical settings, mind you.
     
  25. koitsu

    koitsu Network Guru Member

    Are you sure? Has Riddlah actually rolled back to 097 and confirmed the problem goes away? There's no indication of that in this thread, so how are you so sure? Your setup may not be 100% identical to his; I'm certain you two have different router configuration settings and so on. Be realistic here -- think outside the box, turn off tunnel vision, etc...

    Otherwise, someone needs to provide a full, very technical + very detailed changelog of what exactly changed between 097 and 100. Because whatever changed is something kernel-level (device drivers, features, etc.) -- not userland programs! -- and therein lies the problem.
     
  26. mbreslin

    mbreslin Networkin' Nut Member

    @koitsu: I understand what you're saying, I just don't think it's a coincidence it happened to 3 of us. Both 99 and 100 forced router reboots within minutes of flashing and then somewhat sporadically thereafter, something like once per hour.

    Rolled back to 97 and now up 31 hours without incident.
     
  27. Cyberian75

    Cyberian75 Network Guru Member

    I thought he was using version 100...
     
  28. koitsu

    koitsu Network Guru Member

    @mbreslin: I wouldn't be surprised if the "router reboot" was actually a kernel panic (you'd have no way of determining this unless you had "modded" your router to have a serial console port + had it hooked up to something that stored all serial I/O) as a result of memory being exhausted.
     
  29. koitsu

    koitsu Network Guru Member

    Can one of you who is experiencing this problem at the present, when memory is low, provide me output from lsmod ? (Riddlah you're probably the best choice for this :) ). Thanks.
     
  30. GhaladReam

    GhaladReam Network Guru Member

    It's pretty clear that whatever is causing these memory leaks / random reboots is something that was added or changed in build 099. Looking at the 099 changelog, I see (amongst other things) IPSec support was added, DNSCrypt was updated, IPP2P was updated, and Transmission was updates. I don't think it's Transmission since you guys are still getting the reboots / memory leaks even when it's disabled.

    Here is my overview section currently on build 100 AIO:

    Name RT-N66U
    Model Asus RT-N66U
    Chipset Broadcom BCM5300 chip rev 1 pkg 0
    CPU Freq 600MHz
    Flash Size 32MB

    Time Thu, 16 Aug 2012 11:05:57 -0300
    Uptime 2 days, 22:07:55
    CPU Load (1 / 5 / 15 mins) 0.04 / 0.05 / 0.00
    Total / Free Memory 249.74 MB / 237.39 MB (95.05%)
    Total / Free NVRAM: 64.00 KB / 35.95 KB (56.17%)

    As you can see, I'm not experiencing the memory leak, nor the crashes. I am certain they are being caused by IPsec, DNScrypt-proxy, or IPP2P. I am not using IPsec or DNS-proxy. Hope this helps.
     
  31. Cyberian75

    Cyberian75 Network Guru Member

    I've tried turning DNSCrypt off before downgrading, but it wasn't the problem.
     
  32. koitsu

    koitsu Network Guru Member

    And all of this is assuming the changelog is absolutely 100% accurate and absolutely nothing else was modified/changed other than those things.

    Every person who's experiencing this will need to go through the recommendations I listed off earlier in the thread + run the commands I provided (many require Entware) to try and figure out what's taking up all the memory. I imagine the situation may be the same for some people but one cannot assume that is the case for everyone. For example, remember that earlier in the thread, on Riddlah's system for some reason dnsmasq was taking up 13MBytes of memory. The daemon was obviously restarted by him (or maybe by init if someone killed the daemon off) and the memory bloat for that process specifically disappeared. Then later, he provided more evidence of memory bloat, this time in what appears to be the kernel, and I speculate that IP Traffic or Web Usage is what's causing the problem (it's very difficult for me to determine this without a kernel debugger however).

    For all I know, your situation may be purely with bloated userland processes/daemons. For all I know Cyberian75's issue may be something completely different. Every case needs to be handled separately/individually.

    I've done about as much as I can with this thread -- I'm going to unsubscribe/unwatch it now. I can't do anything more, and juggling 5 or 6 sets of balls while people are doing jigs and cartwheels is simply not feasible for me to do.
     
  33. Cyberian75

    Cyberian75 Network Guru Member

  34. Cyberian75

    Cyberian75 Network Guru Member

    I just upgraded to 100, and the problem goes away when IPv6 is disabled.
     
  35. Riddlah

    Riddlah Networkin' Nut Member

    Here is an update after roughly 23-hours. Web Usage and Logging have been disabled. The display from the main page shows roughly 63mb free.

    [​IMG]

    Code:
    root@MIRAGE-ASUS:/tmp/home/root# /opt/bin/ps auxw | sort -n -k6 -r
    nobody    4225  1.5 10.7  14104 13588 ?        S    04:01  17:13 dnsmasq --conf-file=/tmp/gen
    root      980  0.0  1.5  3160  1920 ?        S    Aug15  0:05 /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
    root      696  0.0  1.1  3356  1496 ?        SNs  Aug15  0:00 smbd -D
    root      670  0.0  1.0  2516  1284 ?        Ss  Aug15  0:02 nmbd -D
    root      575  0.1  0.8  1840  1060 ?        S    Aug15  1:59 ntfs-3g -o iocharset=utf8,noatime,nodev /dev/sda1 /tmp/mnt/Cruzer
    root      671  0.0  0.6  2460  800 ?        S    Aug15  0:00 nmbd -D
    root        1  0.0  0.5  1380  736 ?        Ss  Aug15  0:01 /sbin/init noinitrd
    root    20853  6.6  0.4  1260  556 ?        Ss  22:49  0:01 dropbear -p 22 -p 2222 -a
    root    20868  0.0  0.4  1520  528 pts/0    R+  22:50  0:00 /opt/bin/ps auxw
    root    20857  0.0  0.3  1732  500 pts/0    Ss+  22:50  0:00 -sh
    root      957  0.0  0.3  2704  488 ?        S    Aug15  0:03 httpd
    root      553  0.0  0.3  1196  464 ?        S    Aug15  0:00 nas
    root      334  0.0  0.3  1728  432 ttyS0    Ss+  Aug15  0:00 /bin/sh
    root      958  0.0  0.3    944  428 ?        S    Aug15  0:26 miniupnpd -f /etc/upnp/config
    root      593  0.0  0.3  1740  404 ?        Ss  Aug15  0:00 crond -l 9
    root      595  0.0  0.3    996  392 ?        S    Aug15  0:00 rstats
    root      976  0.0  0.2  1736  376 ?        Ss  Aug15  0:00 udhcpc -i vlan2 -b -s dhcpc-event -H MIRAGE-ASUS -m
    root      536  0.0  0.2  1196  344 ?        S    Aug15  0:00 dropbear -p 22 -p 2222 -a
    root      336  0.0  0.2  1720  332 ?        Ss  Aug15  0:17 syslogd -L -s 1024 -O /tmp/mnt/Cruzer/Logging/Messages -b 1
    root    20869  0.0  0.2  1720  308 pts/0    S+  22:50  0:00 sort -n -k6 -r
    root      601  0.9  0.2    764  308 ?        S    Aug15  12:40 /usr/sbin/bcrelay -i ppp[4-9].* -o br0 -n
    root      606  0.0  0.2    816  304 ?        S    Aug15  0:03 radvd
    root      338  0.0  0.2  1720  300 ?        Ss  Aug15  0:12 klogd
    root      600  0.0  0.2    756  292 ?        S    Aug15  0:00 pptpd -c /tmp/pptpd/pptpd.conf -o /tmp/pptpd/options.pptpd -C 6
    nobody    1063  0.0  0.2    736  292 ?        S    Aug15  0:00 /tmp/mnt/Cruzer/pixelserv 192.168.1.100 -n br0
    root      857  0.0  0.2    808  276 ?        S    Aug15  0:00 udpxy -S -p 4022 -c 10 -m vlan2
    root      540  0.0  0.2  1052  264 ?        S    Aug15  0:00 eapd
    root      533  0.0  0.1  1720  244 ?        Ss  Aug15  0:00 telnetd -p 23
    root      333  0.0  0.1  1364  244 ?        Ss  Aug15  0:00 console
    root      290  0.0  0.1    752  240 ?        Ss  Aug15  0:00 hotplug2 --persistent --no-coldplug
    root      842  0.0  0.1    776  228 ?        S    Aug15  0:00 igmpproxy /etc/igmp.conf
    root      332  0.0  0.1  1364  216 ?        Ss  Aug15  0:00 buttons
    root      406  0.0  0.0      0    0 ?        S<  Aug15  0:00 [usb-storage]
    root      405  0.0  0.0      0    0 ?        S<  Aug15  0:00 [scsi_eh_0]
    root        96  0.0  0.0      0    0 ?        S<  Aug15  0:01 [mtdblockd]
    root        54  0.0  0.0      0    0 ?        S<  Aug15  0:00 [aio/0]
    root        53  0.0  0.0      0    0 ?        S<  Aug15  0:00 [kswapd0]
    root        52  0.0  0.0      0    0 ?        S    Aug15  0:00 [pdflush]
    root        51  0.0  0.0      0    0 ?        S    Aug15  0:00 [pdflush]
    root        24  0.0  0.0      0    0 ?        S<  Aug15  0:00 [khubd]
    root        21  0.0  0.0      0    0 ?        S<  Aug15  0:00 [kblockd/0]
    root        5  0.0  0.0      0    0 ?        S<  Aug15  0:00 [khelper]
    root        4  0.0  0.0      0    0 ?        S<  Aug15  0:00 [events/0]
    root        3  0.0  0.0      0    0 ?        S<  Aug15  0:00 [ksoftirqd/0]
    root        2  0.0  0.0      0    0 ?        S<  Aug15  0:00 [kthreadd]
    USER      PID %CPU %MEM    VSZ  RSS TTY      STAT START  TIME COMMAND
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# free
                total        used        free      shared      buffers
    Mem:        126760        96920        29840            0        17200
    -/+ buffers:              79720        47040
    Swap:            0            0            0
    root@MIRAGE-ASUS:/tmp/home/root#
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# cat /proc/meminfo
    MemTotal:      126760 kB
    MemFree:        29784 kB
    Buffers:        17204 kB
    Cached:          17532 kB
    SwapCached:          0 kB
    Active:          38128 kB
    Inactive:        14080 kB
    HighTotal:          0 kB
    HighFree:            0 kB
    LowTotal:      126760 kB
    LowFree:        29784 kB
    SwapTotal:          0 kB
    SwapFree:            0 kB
    Dirty:              0 kB
    Writeback:          0 kB
    AnonPages:      17480 kB
    Mapped:          4236 kB
    Slab:            40216 kB
    SReclaimable:    1312 kB
    SUnreclaim:      38904 kB
    PageTables:        416 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    WritebackTmp:        0 kB
    CommitLimit:    63380 kB
    Committed_AS:    22412 kB
    VmallocTotal:  1015800 kB
    VmallocUsed:      3808 kB
    VmallocChunk:  1009688 kB
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# cat /proc/slabinfo | awk '/slabdata/ { printf "
    %-20s%-10u%-10u%-10u%-15u%-15u\n", $1, $2, $3, $4, ($4*$2), ($4*$3) }' | sort -n
    -r -k5 | head -10
    size-2048          15040    15052    2048      30801920      30826496
    skbuff_head_cache  15380    15380    192      2952960        2952960
    size-64            8250      8250      128      1056000        1056000
    ip_dst_cache        3060      3420      256      783360        875520
    size-4096          140      140      4096      573440        573440
    buffer_head        7850      7906      64        502400        505984
    size-8192          57        57        8192      466944        466944
    size-65536          7        7        65536    458752        458752
    size-32            2581      2670      128      330368        341760
    size-512            464      464      512      237568        237568
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# lsmod
    Module                  Size  Used by    Tainted: P
    tun                    6464  1
    sit                    9200  0
    tunnel4                1808  1 sit
    nls_cp437              4416  0
    fuse                  38128  2
    ip6table_filter          704  1
    ip6table_mangle          992  0
    xt_length                704  2
    xt_recent              6800  4
    ip6t_LOG                7248  3
    ip6t_REJECT            3008  1
    nf_conntrack_ipv6      14176  8
    ehci_hcd              34640  0
    vfat                    9216  0
    fat                    46000  1 vfat
    ext2                  55520  0
    ext3                  113440  0
    jbd                    48352  1 ext3
    mbcache                4528  2 ext2,ext3
    usb_storage            33120  1
    sd_mod                21376  2
    scsi_wait_scan          384  0
    scsi_mod              75488  3 usb_storage,sd_mod,scsi_wait_scan
    nf_nat_pptp            1440  0
    nf_conntrack_pptp      3808  1 nf_nat_pptp
    nf_nat_proto_gre        1072  1 nf_nat_pptp
    nf_conntrack_proto_gre    2464  1 nf_conntrack_pptp
    nf_nat_ftp              1568  0
    nf_conntrack_ftp        5792  1 nf_nat_ftp
    nf_nat_sip              5920  0
    nf_conntrack_sip      19008  1 nf_nat_sip
    nf_nat_h323            5504  0
    nf_conntrack_h323      37152  1 nf_nat_h323
    wl                  2610544  0
    dnsmq                  2032  1 wl
    et                    37312  0
    igs                    13584  1 wl
    emf                    17568  2 wl,igs
     
  36. Cyberian75

    Cyberian75 Network Guru Member

    Riddlah, what version are you using?
     
  37. Gitsum

    Gitsum LI Guru Member

    Uptime with 099 VPN 64 on ASUS RTN66U, 5+ days. No reboots, no memory leaks. I don't use dnscrypt, vpn, ipv6 or any other "extras".
     
  38. Riddlah

    Riddlah Networkin' Nut Member

    Build 100 and just as an update the following have been disabled DNSCrypt, IPv6, Web Usage, Logging. Also no extras have been installed aside from Entware. OpenVPN is still enabled for the purpose of monitoring any issues and any necessary reboots to make it usable again.


    ---
    Riddlah
     
  39. Cyberian75

    Cyberian75 Network Guru Member

    In my case, it's IPv6 Tunnel that's causing memory leaks / reboots.
     
  40. koitsu

    koitsu Network Guru Member

    (Yes I'm still "watching" this thread manually on occasion)

    Cyberian75, are you absolutely 100% certain? Can you provide actual evidence instead of just replying with one-liners?

    EDIT: Ah, I see, you said this (sigh, more one-liners :p). Okay.

    The reason I say this is because Riddlah is was still experiencing the problem yet turned off "IPv6 Tunnel" entirely. See the link/post (it's in this thread) for verification. Starting to see why I said every person's situation may be different?

    Furthermore, can you explain what "IPv6 Tunnel" means in this context? Maybe in Shibby its labelled different than in Toastman's builds, but the only IPv6 tunnelling feature I've seen is a 6in4 tunnel through places like Hurricane Electric, etc... Is that what you're referring to? It's called "6to4 Static Tunnel" in Toastman, not sure about Shibby. This is different than "6in4 Anycast Relay" (and that behaves quite differently too).

    I myself use native IPv6 (specifically: DHCPv6 with Prefix Delegation), since Comcast delegates IPv6 to customers directly (in my area), and do not have issues.

    Possibly the issue is related to ICMPv6 route announcements coming from the tunnel provider? I don't have a good way to tackle this kind of situation. There are lots of adjustables and useful things in /proc but I'm simply not familiar enough with the IPv6 stack on Linux to use those to troubleshoot. About the only thing I can think of is to look at the number of IPv6 routes you're seeing using route -A inet6 -n | wc -l via CLI. On my Comcast connection, I see between 60 and 400 routes (it varies, as it should). Most of my route table entries are for /128s (e.g. a "single IPv6 address").
     
  41. GhaladReam

    GhaladReam Network Guru Member

    Riddlah,

    Can you wipe your NVRAM and start from scratch (on build 100) and config everything you normally do, except do not configure IPv6 tunnel.. see if you still get memory leaks?
     
  42. Cyberian75

    Cyberian75 Network Guru Member

    Yes, I'm using Hurricane Electric. When it's on, memory takes a dive; when it's off, memory is stable. What more can I say?
     
  43. koitsu

    koitsu Network Guru Member

    It would help if you could run some (or all) of the commands I've asked throughout this thread and provide the output from them here (in code blocks for readability). Some may require you to install Entware, but for example the last one I just provided doesn't.
     
  44. jyavenard

    jyavenard Network Guru Member

    Good luck getting this guy posting anything useful in another form than a one-liner... completely hopeless...
     
  45. Cyberian75

    Cyberian75 Network Guru Member

    Code:
    root@Asus:/tmp/home/root# /opt/bin/ps auxw | sort -n -k6 -r
    root        1  0.5  0.2  1380  760 ?        Ss  14:08  0:01 /sbin/init noin                                                                                                                                                            itrd
    root      1142  2.1  0.2  1260  556 ?        Ss  14:11  0:01 dropbear -p 22                                                                                                                                                              -a
    nobody    683  0.0  0.2  1104  536 ?        S    14:10  0:00 dnsmasq -c 1500                                                                                                                                                              --log-async
    root      1213  0.0  0.2  1524  512 pts/0    R+  14:12  0:00 /opt/bin/ps aux                                                                                                                                                            w
    root      700  0.3  0.1  1372  500 ?        Ss  14:10  0:00 dnscrypt-proxy                                                                                                                                                              -d -P 40
    root      1146  0.0  0.1  1732  496 pts/0    Ss+  14:11  0:00 -sh
    root      1089  0.0  0.1  2700  484 ?        S    14:10  0:00 httpd
    root      354  0.0  0.1  1728  432 ttyS0    Ss+  14:09  0:00 /bin/sh
    root      600  0.0  0.1  1732  400 ?        Ss  14:09  0:00 crond
    root      571  0.3  0.1  1196  400 ?        S    14:09  0:00 nas
    root      602  0.0  0.1    996  392 ?        S    14:09  0:00 rstats
    root      611  0.0  0.1    976  380 ?        S    14:09  0:00 cstats
    root      1086  0.0  0.1    900  352 ?        S    14:10  0:00 miniupnpd -f /e                                                                                                                                                            tc/upnp/config
    root      565  0.0  0.1  1196  336 ?        S    14:09  0:00 dropbear -p 22                                                                                                                                                              -a
    root      355  0.0  0.1  1720  332 ?        Ss  14:09  0:00 syslogd -L -s 5                                                                                                                                                            000 -O /tmp/mnt/sda1/logs/syslog -b 2
    root      1214  0.0  0.1  1720  308 pts/0    S+  14:12  0:00 sort -n -k6 -r
    root      619  0.0  0.1    816  304 ?        S    14:09  0:00 radvd
    root      357  0.0  0.1  1720  300 ?        Ss  14:09  0:00 klogd
    root      352  0.0  0.0  1364  244 ?        Ss  14:09  0:00 console
    root      309  0.0  0.0    752  240 ?        Ss  14:09  0:00 hotplug2 --pers                                                                                                                                                            istent --no-coldplug
    root      1090  0.0  0.0  1736  220 ?        Ss  14:10  0:00 udhcpc -i vlan2                                                                                                                                                              -b -s dhcpc-event -H Asus -m -S
    root      351  0.0  0.0  1364  216 ?        Ss  14:09  0:00 buttons
    root      568  0.0  0.0  1052  200 ?        S    14:09  0:00 eapd
    root      455  0.0  0.0      0    0 ?        S<  14:09  0:00 [usb-storage]
    root      454  0.0  0.0      0    0 ?        S<  14:09  0:00 [scsi_eh_0]
    root      392  0.0  0.0      0    0 ?        S<  14:09  0:00 [kmmcd]
    root      101  0.2  0.0      0    0 ?        S<  14:08  0:00 [mtdblockd]
    root        59  0.0  0.0      0    0 ?        S<  14:08  0:00 [aio/0]
    root        58  0.0  0.0      0    0 ?        S<  14:08  0:00 [kswapd0]
    root        57  0.0  0.0      0    0 ?        S    14:08  0:00 [pdflush]
    root        56  0.0  0.0      0    0 ?        S    14:08  0:00 [pdflush]
    root        24  0.0  0.0      0    0 ?        S<  14:08  0:00 [khubd]
    root        21  0.0  0.0      0    0 ?        S<  14:08  0:00 [kblockd/0]
    root        5  0.0  0.0      0    0 ?        S<  14:08  0:00 [khelper]
    root        4  0.0  0.0      0    0 ?        S<  14:08  0:00 [events/0]
    root        3  0.0  0.0      0    0 ?        S<  14:08  0:00 [ksoftirqd/0]
    root        2  0.0  0.0      0    0 ?        S<  14:08  0:00 [kthreadd]
    USER      PID %CPU %MEM    VSZ  RSS TTY      STAT START  TIME COMMAND
    
    Code:
    root@Asus:/tmp/home/root# cat /proc/meminfo
    MemTotal:       255732 kB
    MemFree:        130092 kB
    Buffers:          3564 kB
    Cached:          10868 kB
    SwapCached:          0 kB
    Active:           6512 kB
    Inactive:         9904 kB
    HighTotal:      131072 kB
    HighFree:       114728 kB
    LowTotal:       124660 kB
    LowFree:         15364 kB
    SwapTotal:           0 kB
    SwapFree:            0 kB
    Dirty:               0 kB
    Writeback:           0 kB
    AnonPages:        1992 kB
    Mapped:           2000 kB
    Slab:           104572 kB
    SReclaimable:      852 kB
    SUnreclaim:     103720 kB
    PageTables:        272 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    WritebackTmp:        0 kB
    CommitLimit:    127864 kB
    Committed_AS:     4988 kB
    VmallocTotal:  1015800 kB
    VmallocUsed:      3844 kB
    VmallocChunk:  1009672 kB
    
    Code:
    root@Asus:/tmp/home/root# lsmod
    Module                  Size  Used by    Tainted: P
    sit                     9200  0
    tunnel4                 1808  1 sit
    ip6table_filter          704  1
    ip6table_mangle          992  0
    xt_webmon              16320  2
    xt_web                  2016  2
    xt_length                704  2
    xt_recent               6800  4
    ip6t_REJECT             3008  1
    nf_conntrack_ipv6      14176  4
    ehci_hcd               34640  0
    sdhci                  12368  0
    mmc_block               6768  0
    mmc_core               17216  2 sdhci,mmc_block
    ext2                   55520  1
    ext3                  113440  0
    jbd                    48352  1 ext3
    mbcache                 4528  2 ext2,ext3
    usb_storage            33120  1
    sd_mod                 21376  2
    scsi_wait_scan           384  0
    scsi_mod               75488  3 usb_storage,sd_mod,scsi_wait_scan
    leds_usb                2128  0
    led_class               1552  1 leds_usb
    ledtrig_usbdev          2464  1 leds_usb
    nf_nat_pptp             1440  0
    nf_conntrack_pptp       3808  1 nf_nat_pptp
    nf_nat_proto_gre        1072  1 nf_nat_pptp
    nf_conntrack_proto_gre     2464  1 nf_conntrack_pptp
    nf_nat_ftp              1568  0
    nf_conntrack_ftp        5792  1 nf_nat_ftp
    nf_nat_sip              5920  0
    nf_conntrack_sip       19008  1 nf_nat_sip
    nf_nat_h323             5504  0
    nf_conntrack_h323      37152  1 nf_nat_h323
    wl                   2610544  0
    dnsmq                   2032  1 wl
    et                     37312  0
    igs                    13584  1 wl
    emf                    17568  2 wl,igs
    
    Code:
    root@Asus:/tmp/home/root# route -A inet6 -n | wc -l
    40
    
     
  46. Cyberian75

    Cyberian75 Network Guru Member

    Typing is difficult for me as I've a physical disability.
     
  47. Cyberian75

    Cyberian75 Network Guru Member

    Code:
    root@Asus:/tmp/home/root# cat /proc/slabinfo
    slabinfo - version: 2.1
    # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
    scsi_sense_cache      1    40    96  40    1 : tunables  120  60    0 : slabdata      1      1      0
    scsi_cmd_cache        1    20    192  20    1 : tunables  120  60    0 : slabdata      1      1      0
    ext2_inode_cache      20    27    424    9    1 : tunables  54  27    0 : slabdata      3      3      0
    ext2_xattr            0      0    52  72    1 : tunables  120  60    0 : slabdata      0      0      0
    ext3_inode_cache      0      0    432    9    1 : tunables  54  27    0 : slabdata      0      0      0
    ext3_xattr            0      0    52  72    1 : tunables  120  60    0 : slabdata      0      0      0
    journal_handle        0      0    20  169    1 : tunables  120  60    0 : slabdata      0      0      0
    journal_head          0      0    52  72    1 : tunables  120  60    0 : slabdata      0      0      0
    revoke_table          0      0    12  254    1 : tunables  120  60    0 : slabdata      0      0      0
    revoke_record          0      0    16  203    1 : tunables  120  60    0 : slabdata      0      0      0
    sgpool-128            2      2  2048    2    1 : tunables  24  12    0 : slabdata      1      1      0
    sgpool-64              2      4  1024    4    1 : tunables  54  27    0 : slabdata      1      1      0
    sgpool-32              2      8    512    8    1 : tunables  54  27    0 : slabdata      1      1      0
    sgpool-16              2    15    256  15    1 : tunables  120  60    0 : slabdata      1      1      0
    sgpool-8              2    30    128  30    1 : tunables  120  60    0 : slabdata      1      1      0
    scsi_io_context        0      0    104  37    1 : tunables  120  60    0 : slabdata      0      0      0
    ip_fib_alias          0      0    16  203    1 : tunables  120  60    0 : slabdata      0      0      0
    ip_fib_hash          15    101    36  101    1 : tunables  120  60    0 : slabdata      1      1      0
    bridge_fdb_cache      4    59    64  59    1 : tunables  120  60    0 : slabdata      1      1      0
    fib6_nodes            33    113    32  113    1 : tunables  120  60    0 : slabdata      1      1      0
    ip6_dst_cache        46    60    256  15    1 : tunables  120  60    0 : slabdata      4      4      0
    ndisc_cache          10    24    160  24    1 : tunables  120  60    0 : slabdata      1      1      0
    ip6_mrt_cache          0      0    128  30    1 : tunables  120  60    0 : slabdata      0      0      0
    RAWv6                  5      6    608    6    1 : tunables  54  27    0 : slabdata      1      1      0
    UDPLITEv6              0      0    608    6    1 : tunables  54  27    0 : slabdata      0      0      0
    UDPv6                  2      6    608    6    1 : tunables  54  27    0 : slabdata      1      1      0
    tw_sock_TCPv6          0      0    128  30    1 : tunables  120  60    0 : slabdata      0      0      0
    request_sock_TCPv6      0      0    128  30    1 : tunables  120  60    0 : slabdata      0      0      0
    TCPv6                  4      6  1184    6    2 : tunables  24  12    0 : slabdata      1      1      0
    UNIX                  11    20    384  10    1 : tunables  54  27    0 : slabdata      2      2      0
    nf_nat:help          48    144    312  12    1 : tunables  54  27    0 : slabdata    12    12      0
    nf_nat:base            0      0    280  14    1 : tunables  54  27    0 : slabdata      0      0      0
    nf_conntrack:help      0      0    268  14    1 : tunables  54  27    0 : slabdata      0      0      0
    nf_conntrack_expect      0      0    152  26    1 : tunables  120  60    0 : slabdata      0      0      0
    nf_conntrack:basic    100    112    236  16    1 : tunables  120  60    0 : slabdata      7      7      0
    flow_cache            0      0    96  40    1 : tunables  120  60    0 : slabdata      0      0      0
    squashfs_inode_cache    213    220    352  11    1 : tunables  54  27    0 : slabdata    20    20      0
    configfs_dir_cache      0      0    48  78    1 : tunables  120  60    0 : slabdata      0      0      0
    dnotify_cache          0      0    20  169    1 : tunables  120  60    0 : slabdata      0      0      0
    inotify_event_cache      0      0    28  127    1 : tunables  120  60    0 : slabdata      0      0      0
    inotify_watch_cache      0      0    40  92    1 : tunables  120  60    0 : slabdata      0      0      0
    kioctx                0      0    160  24    1 : tunables  120  60    0 : slabdata      0      0      0
    kiocb                  0      0    160  24    1 : tunables  120  60    0 : slabdata      0      0      0
    fasync_cache          0      0    16  203    1 : tunables  120  60    0 : slabdata      0      0      0
    shmem_inode_cache    185    190    392  10    1 : tunables  54  27    0 : slabdata    19    19      0
    posix_timers_cache      0      0    96  40    1 : tunables  120  60    0 : slabdata      0      0      0
    uid_cache              1    59    64  59    1 : tunables  120  60    0 : slabdata      1      1      0
    ip_mrt_cache          0      0    96  40    1 : tunables  120  60    0 : slabdata      0      0      0
    UDP-Lite              0      0    480    8    1 : tunables  54  27    0 : slabdata      0      0      0
    tcp_bind_bucket        5    203    16  203    1 : tunables  120  60    0 : slabdata      1      1      0
    inet_peer_cache        3    59    64  59    1 : tunables  120  60    0 : slabdata      1      1      0
    secpath_cache          0      0    32  113    1 : tunables  120  60    0 : slabdata      0      0      0
    xfrm_dst_cache        0      0    288  13    1 : tunables  54  27    0 : slabdata      0      0      0
    ip_dst_cache          41    105    256  15    1 : tunables  120  60    0 : slabdata      7      7      0
    arp_cache              7    30    128  30    1 : tunables  120  60    0 : slabdata      1      1      0
    RAW                    3      8    480    8    1 : tunables  54  27    0 : slabdata      1      1      0
    UDP                  12    16    480    8    1 : tunables  54  27    0 : slabdata      2      2      0
    tw_sock_TCP          25    40    96  40    1 : tunables  120  60    0 : slabdata      1      1      0
    request_sock_TCP      4    40    96  40    1 : tunables  120  60    0 : slabdata      1      1      0
    TCP                    6      7  1088    7    2 : tunables  24  12    0 : slabdata      1      1      0
    eventpoll_pwq          3    101    36  101    1 : tunables  120  60    0 : slabdata      1      1      0
    eventpoll_epi          3    40    96  40    1 : tunables  120  60    0 : slabdata      1      1      0
    blkdev_ioc            0      0    32  113    1 : tunables  120  60    0 : slabdata      0      0      0
    blkdev_queue          2      4  1000    4    1 : tunables  54  27    0 : slabdata      1      1      0
    blkdev_requests        8    20    192  20    1 : tunables  120  60    0 : slabdata      1      1      0
    biovec-256            2      2  3072    2    2 : tunables  24  12    0 : slabdata      1      1      0
    biovec-128            2      5  1536    5    2 : tunables  24  12    0 : slabdata      1      1      0
    biovec-64              2      5    768    5    1 : tunables  54  27    0 : slabdata      1      1      0
    biovec-16              2    20    192  20    1 : tunables  120  60    0 : slabdata      1      1      0
    biovec-4              2    59    64  59    1 : tunables  120  60    0 : slabdata      1      1      0
    biovec-1              2    203    16  203    1 : tunables  120  60    0 : slabdata      1      1      0
    bio                    2    40    96  40    1 : tunables  120  60    0 : slabdata      1      1      0
    sock_inode_cache      55    55    352  11    1 : tunables  54  27    0 : slabdata      5      5      0
    skbuff_fclone_cache    10    10    384  10    1 : tunables  54  27    0 : slabdata      1      1      0
    skbuff_head_cache  45120  45120    192  20    1 : tunables  120  60    0 : slabdata  2256  2256      0
    file_lock_cache        0      0    96  40    1 : tunables  120  60    0 : slabdata      0      0      0
    proc_inode_cache    276    276    312  12    1 : tunables  54  27    0 : slabdata    23    23      0
    sigqueue              0      0    144  27    1 : tunables  120  60    0 : slabdata      0      0      0
    radix_tree_node      204    208    296  13    1 : tunables  54  27    0 : slabdata    16    16      0
    bdev_cache            4      9    416    9    1 : tunables  54  27    0 : slabdata      1      1      0
    sysfs_dir_cache    1819  1872    48  78    1 : tunables  120  60    0 : slabdata    24    24      0
    mnt_cache            20    30    128  30    1 : tunables  120  60    0 : slabdata      1      1      0
    inode_cache          507    507    296  13    1 : tunables  54  27    0 : slabdata    39    39      0
    dentry              1417  1426    124  31    1 : tunables  120  60    0 : slabdata    46    46      0
    filp                300    504    160  24    1 : tunables  120  60    0 : slabdata    21    21      0
    names_cache            1      1  4096    1    1 : tunables  24  12    0 : slabdata      1      1      0
    buffer_head        3522  3540    64  59    1 : tunables  120  60    0 : slabdata    60    60      0
    mm_struct            36    36    416    9    1 : tunables  54  27    0 : slabdata      4      4      0
    vm_area_struct      554    782    84  46    1 : tunables  120  60    0 : slabdata    17    17      0
    fs_cache              26    113    32  113    1 : tunables  120  60    0 : slabdata      1      1      0
    files_cache          24    60    192  20    1 : tunables  120  60    0 : slabdata      3      3      0
    signal_cache          50    50    384  10    1 : tunables  54  27    0 : slabdata      5      5      0
    sighand_cache        42    42  3104    2    2 : tunables  24  12    0 : slabdata    21    21      0
    task_struct          44    44  1024    4    1 : tunables  54  27    0 : slabdata    11    11      0
    anon_vma            253    339      8  339    1 : tunables  120  60    0 : slabdata      1      1      0
    pid                  53    101    36  101    1 : tunables  120  60    0 : slabdata      1      1      0
    idr_layer_cache      66    78    148  26    1 : tunables  120  60    0 : slabdata      3      3      0
    size-4194304          0      0 4194304    1 1024 : tunables    1    1    0 : slabdata      0      0      0
    size-2097152          0      0 2097152    1  512 : tunables    1    1    0 : slabdata      0      0      0
    size-1048576          0      0 1048576    1  256 : tunables    1    1    0 : slabdata      0      0      0
    size-524288            0      0 524288    1  128 : tunables    1    1    0 : slabdata      0      0      0
    size-262144            0      0 262144    1  64 : tunables    1    1    0 : slabdata      0      0      0
    size-131072            1      1 131072    1  32 : tunables    8    4    0 : slabdata      1      1      0
    size-65536            7      7  65536    1  16 : tunables    8    4    0 : slabdata      7      7      0
    size-32768            2      2  32768    1    8 : tunables    8    4    0 : slabdata      2      2      0
    size-16384            0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
    size-8192            51    51  8192    1    2 : tunables    8    4    0 : slabdata    51    51      0
    size-4096            265    265  4096    1    1 : tunables  24  12    0 : slabdata    265    265      0
    size-2048          44934  44934  2048    2    1 : tunables  24  12    0 : slabdata  22467  22467      0
    size-1024            72    72  1024    4    1 : tunables  54  27    0 : slabdata    18    18      0
    size-512            217    232    512    8    1 : tunables  54  27    0 : slabdata    29    29      0
    size-256              90    90    256  15    1 : tunables  120  60    0 : slabdata      6      6      0
    size-192            130    135    256  15    1 : tunables  120  60    0 : slabdata      9      9      0
    size-128            131    150    128  30    1 : tunables  120  60    0 : slabdata      5      5      0
    size-96              403    420    128  30    1 : tunables  120  60    0 : slabdata    14    14      0
    size-64            23160  23160    128  30    1 : tunables  120  60    0 : slabdata    772    772      0
    size-32            2743  2760    128  30    1 : tunables  120  60    0 : slabdata    92    92      0
    kmem_cache          117    120    96  40    1 : tunables  120  60    0 : slabdata      3      3      0
    
     
  48. koitsu

    koitsu Network Guru Member

    Thanks. The problem is the same in your case as Riddlah's:

    Code:
    size-2048          44934  44934  2048    2    1 : tunables  24  12    0 : slabdata  22467  22467      0
    
    There are 44,934 kernel slab objects of 2048 bytes in use. 44934 * 2048 = 92024832 bytes, or roughly 92MBytes. Lucky for you your router is 256MBytes of RAM, but as shown from /proc/meminfo's LowFree column, the lowest amount of memory you've had is 15MBytes. That's pretty rough indeed.

    What's using/allocating these slabs is difficult to determine without a kernel debugger.

    So once again, this is a kernel-level problem (whether it be a network stack, device driver, or some other anomaly).
     
  49. Cyberian75

    Cyberian75 Network Guru Member

    Thanks for confirming.
     
  50. Cyberian75

    Cyberian75 Network Guru Member

    Shibby has identified the problem and will try to fix it.
     
  51. guardian

    guardian Serious Server Member

    Let me add that I have this problem too (latest builds from shibby, toastman, avenard, etc all have the same problem, and it's not a surprise, because all of them has the same code base) and yes, I have 6in4 tunnel enabled (using HE.net). Leak progresses very fast if the connection is actively used (say, torrent is downloading having a number of IPv6 peers).

    Build 1.28.0500.2 by toastman has no leak, and 0500.4 has the leak, so GIT repository may be used to track the problem down.
     
  52. jyavenard

    jyavenard Network Guru Member

    The only difference that could explain this issue for the change between 500.2 and 500.4 is the upgrade of the Broadcom drivers to 5.100.138.20.

    They could have introduced a regression when using ipv6 (is the issue only happening for ipv6 users??).
    To confirm, you can try checking my branch Toastman-RT-jya
    And revert the following commit in the order:
    http://repo.or.cz/w/tomato.git/commit/578a13b3e393c166560218d812c77873165d2494
    http://repo.or.cz/w/tomato.git/commit/4ea28201cf57907194f42bf64ea97a18158e30f1
    http://repo.or.cz/w/tomato.git/commit/2c14085986e7bffd1dc15840a36b3ba690882a57

    Unfortunately, there's nothing I can do until mid-October as I'm away, and I have no wireless routers where I could load tomato to try

    Note that the official Asus firmware also has known issue with IPv6, issues that were supposedly addressed in their latest firmware.. Would need to check at what they have changed that are Broadcom drivers related.
     
  53. Cyberian75

    Cyberian75 Network Guru Member

    Shibby thinks it's IPSec that's causing the problem within the kernel.
     
  54. kthaddock

    kthaddock Network Guru Member

  55. jyavenard

    jyavenard Network Guru Member

  56. zorkmta

    zorkmta LI Guru Member

    Have installed Shibby v 100. and have a little memory leak day by day, started have 95% free and after 5 days have 89%. All stable and dont have reboot... hope fix leak soon ;)

    Model Asus RT-N66U
    Chipset Broadcom BCM5300 chip rev 1 pkg 0
    CPU Freq 600MHz
    Flash Size 32MB

    Time Sat, 18 Aug 2012 15:32:58 -0400
    Uptime 5 days, 00:56:34
    CPU Load (1 / 5 / 15 mins) 0.00 / 0.00 / 0.00
    Total / Free Memory 249.74 MB / 222.98 MB (89.28%)
    Total / Free NVRAM: 64.00 KB / 36.88 KB (57.62%)
     
  57. jyavenard

    jyavenard Network Guru Member

    And he's probably wrong... No one here seems to be using IPSec.

    Why would it matter when the problem is reported with all build using the latest drivers? (Toastman, mine, shibby).
     
  58. dupondje

    dupondje Serious Server Member

    Installed 0500.4, and its working fine, untill I start using IPv6. Then it died in like 30 seconds ... :(
     
  59. jyavenard

    jyavenard Network Guru Member

    And it's all fine with 0500.2?
     
  60. dupondje

    dupondje Serious Server Member

    Was running 0500.2 for ~1 month without any issues (and tons of IPv6 traffic :))
     
  61. Cyberian75

    Cyberian75 Network Guru Member

    Because they -- IPv6 & IPSec -- share the same module called "sit.ko".
     
  62. Cyberian75

    Cyberian75 Network Guru Member

    Sigh... I'm just relaying what Shibby told me in a private conversation.

    You seem to have a superiority complex. I asked you a simple question in another thread, and you overreacted right away. No tolerance at all.
     
  63. Cyberian75

    Cyberian75 Network Guru Member

  64. jyavenard

    jyavenard Network Guru Member

  65. Cyberian75

    Cyberian75 Network Guru Member

    I was referring to this post in all replies thereafter. I'd have quoted...
     
  66. fearz

    fearz Serious Server Member

    Shibby, so when can we expect a fix?
     
  67. Cyberian75

    Cyberian75 Network Guru Member

    It will be out when it is ready... he's not a god. Debugging isn't easy. And besides, this isn't a paying job.
     
  68. leandroong

    leandroong Addicted to LI Member

  69. guardian

    guardian Serious Server Member

    jyavenard:

    Mr. Avenard, can I help you or someone else to debug a problem? I can flash debug versions and test them for you, or even give a shell access to my routers. I have both rt-n66 and RT-N16.
     
  70. shibby20

    shibby20 Network Guru Member

    @jya my conclusions:
    - v097 works correct. No memory leaks issue. (this version have new BCM driver and all works great).
    - Memory leak ONLY when we have ipv6 enable . I and Cyberian75 are using HE.net tunnel. Memory leaks when we have ipv6 static to HE.net enabled and we are (for ie) watching a Youtube. Memory leaks very fast. For about 5-6mins we have a reboot.
    - There is not a problem of 6RD support (this was added in v100 but memory leak issue is also in v99
    - There is not a problem with commits:
    DDNS: fix HE.net Tunnel Broker messages
    Added IPSec support for K26 builds
    Transmission: update to 2.61

    i reverted those commit and the problem is still exist. All tests i make on RT-N66U with tomato Mega-VPN 64k, WAN DHCP (public static ip) and HE.net ipv6 tunnel without DDNS.

    I suppore there may be a problem with Fix virtual wireless (MultiSSID). I`m compiling now 097 (as i said this version hasn`t memory problem) with only yours four commits about MultiSSID and let you know.
     
  71. shibby20

    shibby20 Network Guru Member

    Confirmed!! v097 works correct. v097 with applied only four @jya cherry-picks:
    - Fix virtual wireless (MultiSSID)
    - Add option to force nvram commit when validating a page.
    - Rework virtual wireless configuration page
    - Fix K24 build

    compiled and we have a memory leaks problem. go back to v097 and problem disappear.

    this is why all new builds (my, jya and tomastman) have memory leaks problem.

    @jya can you look on that?
     
  72. Cyberian75

    Cyberian75 Network Guru Member

    What does Virtual Wireless have to to with IPv6 Tunnel?
     
  73. jyavenard

    jyavenard Network Guru Member

    The multi-SSID fix only changes the order in which the wireless interface is initialised. The only change that is relevant is that the mac address of the interface is forced. This occurs only once, right after a boot.
    So there's no way that code introduce a leak. Even if that code had a leak, it would be a one-off.

    If this issue cause a leak, all it does is exposing a bug in the kernel driver... Not much you can do there...
     
    koitsu likes this.
  74. leandroong

    leandroong Addicted to LI Member

    My suggestion: Do elimination approach, add them one by one and retest ...
     
  75. guardian

    guardian Serious Server Member

    It may be a bug in broadcom lan drvier, exposed by changing mac address on the interface. I have seen such things before, on x86 platform. I suggest downgrade broadcom drviers (or upgrade to latest from latest Asus beta firmware) and retest.
     
  76. kthaddock

    kthaddock Network Guru Member

  77. guardian

    guardian Serious Server Member

    Well, are there any chances to upgrade asus broadcom drivers or even just rip them off asuswrt-merlin?
     
  78. kthaddock

    kthaddock Network Guru Member

    I have done some more investigation and it seams that Radvd have some bug.
    Shibby, Jya, Toastman can you try to update and se if memory leaks disappear. latest is: radvd-1.9.1
    http://www.litech.org/radvd/
    Libdeaemon 0.14
    http://0pointer.de/lennart/projects/libdaemon/#news Version 0.14
     
  79. RMerlin

    RMerlin Network Guru Member

    Asus did indeed upgrade radvd between 144 and 157. I did not notice any kernel-level change, so I suspect radvd was the culprit.
     
  80. koitsu

    koitsu Network Guru Member

    If radvd is the source of the problem, then the issue is that the daemon itself does not "behave correctly" with the kernel. The point I'm trying to make: the daemon itself does not grow in RSS/RES. Here are posts with relevant output that prove the daemon itself does not have memory bloat (but of course may still be causing memory bloat within kernel slab space):

    http://www.linksysinfo.org/index.php?threads/memory-leak.38924/#post-187308
    http://www.linksysinfo.org/index.php?threads/memory-leak.38924/page-2#post-187382
    http://www.linksysinfo.org/index.php?threads/memory-leak.38924/page-2#post-187431

    View these posts and search for "radvd" and look at the RSS/RES column.

    I imagine radvd must be doing something bad with related RA (route announcements). There must be some form of ABI used between userland and the kernel for handling RA, and the userland program is expected to tell the kernel "free this", which would explain how there could be a bug in radvd. But the actual memory exhaustion itself happens within the kernel.
     
  81. Cyberian75

    Cyberian75 Network Guru Member

    From a layman's observation, it seems like the router is caching the data coming through the tunnel in its memory and not releasing it.
     
  82. jyavenard

    jyavenard Network Guru Member

  83. Cyberian75

    Cyberian75 Network Guru Member

    So Shibby was right regarding "sit.ko"...
     
  84. koitsu

    koitsu Network Guru Member

  85. shibby20

    shibby20 Network Guru Member

  86. guardian

    guardian Serious Server Member

    JYAvenard:

    me too :)

    Could you please provide a link to the test build?
     
  87. Riddlah

    Riddlah Networkin' Nut Member

    I can give the test build a try if you still need some testing


    ---
    Riddlah
     
  88. RMerlin

    RMerlin Network Guru Member

    When I backported that fix (this one was by me as I was also trying to track down the source of the memory leak at the time) it didn't solve the issue. It's possible however that there were two distinct memory leaks, and while I happened to fix that one, Asus fixed another one with the radvd update (tho I agree, it also sounds doubtful).

    No idea which of these two would be responsible for Tomato's specific case however. I'd start by also implementing the kernel patch, and if it still leaks then also try the radvd upgrade.
     
  89. shibby20

    shibby20 Network Guru Member

    tomato-RT already has this commit included:
    http://repo.or.cz/w/tomato.git/commit/f4e040a9a69fd07c2b9d4f2e8b0382fc7ac25efa

    i`m still searching.
     
  90. jyavenard

    jyavenard Network Guru Member

  91. Cyberian75

    Cyberian75 Network Guru Member

    It still leaks, albeit slower.
     
  92. Riddlah

    Riddlah Networkin' Nut Member

    I can confirm the leak is still there.

    [​IMG]

    Code:
    root@MIRAGE-ASUS:/tmp/home/root# /opt/bin/ps auxw | sort -n -k6 -r
    root      670  0.0  1.1  3356  1468 ?        SNs  19:39  0:00 smbd -D
    root      637  0.0  1.0  2504  1320 ?        Ss  19:39  0:00 nmbd -D
    root      467  0.3  0.8  1844  1068 ?        S    19:38  0:33 ntfs-3g -o iocharset=utf8,noatime,nodev /dev/sda1 /tmp/mnt/Cruzer
    root      605  0.0  0.6  1428  832 ?        S    19:38  0:04 cstats
    root      638  0.0  0.6  2460  800 ?        S    19:39  0:00 nmbd -D
    root      1069  0.0  0.6  2716  784 ?        S    19:39  0:00 /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
    root        1  0.0  0.5  1364  736 ?        Ss  19:37  0:02 /sbin/init noinitrd
    nobody    591  0.0  0.4  1120  616 ?        S    19:38  0:03 dnsmasq -c 1500 --log-async
    root      6447  3.4  0.4  1260  556 ?        Ss  22:35  0:01 dropbear -p 22 -p 2222 -a
    root      6473  0.0  0.4  1520  528 pts/0    R+  22:36  0:00 /opt/bin/ps auxw
    root      1051  0.0  0.3  2660  496 ?        S    19:39  0:02 httpd
    root      6451  0.1  0.3  1720  492 pts/0    Ss+  22:35  0:00 -sh
    root      586  0.0  0.3  1196  464 ?        S    19:38  0:00 nas
    root      1048  0.0  0.3    952  460 ?        S    19:39  0:03 miniupnpd -f /etc/upnp/config
    root      599  0.0  0.3  1732  416 ?        Ss  19:38  0:00 crond
    root      324  0.0  0.3  1716  408 ttyS0    Ss+  19:38  0:00 /bin/sh
    root      601  0.0  0.3    996  392 ?        S    19:38  0:00 rstats
    root      620  0.0  0.2    904  372 ?        S    19:38  0:00 radvd
    root      579  0.0  0.2  1200  340 ?        S    19:38  0:00 dropbear -p 22 -p 2222 -a
    root      326  0.0  0.2  1708  336 ?        Ss  19:38  0:04 syslogd -L -s 1024 -O /tmp/mnt/Cruzer/Logging/Messages -b 1
    root      6474  0.0  0.2  1708  316 pts/0    S+  22:36  0:00 sort -n -k6 -r
    root      616  0.5  0.2    764  308 ?        S    19:38  0:54 /usr/sbin/bcrelay -i br0 -o ppp[4-9].* -n
    root      328  0.0  0.2  1708  304 ?        Ss  19:38  0:03 klogd
    root      615  0.0  0.2    756  296 ?        S    19:38  0:00 pptpd -c /tmp/pptpd/pptpd.conf -o /tmp/pptpd/options.pptpd -C 6
    root      3195  0.0  0.2    808  276 ?        S    20:46  0:00 udpxy -S -p 4022 -c 10 -m vlan2
    root      583  0.0  0.2  1052  264 ?        S    19:38  0:00 eapd
    root      380  0.0  0.1    736  248 ?        Ss  19:38  0:00 p9100d  -f /dev/usb/lp0 0
    root      323  0.0  0.1  1344  244 ?        Ss  19:38  0:00 console
    root      576  0.0  0.1  1708  240 ?        Ss  19:38  0:00 telnetd -p 23
    root      281  0.0  0.1    752  240 ?        Ss  19:38  0:00 hotplug2 --persistent --no-coldplug
    root      618  0.0  0.1    904  236 ?        S    19:38  0:00 radvd
    root      3192  0.0  0.1    776  224 ?        S    20:46  0:00 igmpproxy /etc/igmp.conf
    root      1070  0.0  0.1  1724  224 ?        Ss  19:39  0:00 udhcpc -i vlan2 -b -s dhcpc-event -H MIRAGE-ASUS -m -S
    root      322  0.0  0.1  1344  216 ?        Ss  19:38  0:00 buttons
    root      384  0.0  0.0      0    0 ?        S<  19:38  0:00 [usb-storage]
    root      383  0.0  0.0      0    0 ?        S<  19:38  0:00 [scsi_eh_0]
    root      339  0.0  0.0      0    0 ?        S<  19:38  0:00 [khubd]
    root        89  0.0  0.0      0    0 ?        S<  19:37  0:01 [mtdblockd]
    root        47  0.0  0.0      0    0 ?        S<  19:37  0:00 [aio/0]
    root        46  0.0  0.0      0    0 ?        S<  19:37  0:00 [kswapd0]
    root        45  0.0  0.0      0    0 ?        S    19:37  0:00 [pdflush]
    root        44  0.0  0.0      0    0 ?        S    19:37  0:00 [pdflush]
    root        18  0.0  0.0      0    0 ?        S<  19:37  0:00 [kblockd/0]
    root        5  0.0  0.0      0    0 ?        S<  19:37  0:00 [khelper]
    root        4  0.0  0.0      0    0 ?        S<  19:37  0:00 [events/0]
    root        3  0.0  0.0      0    0 ?        S<  19:37  0:02 [ksoftirqd/0]
    root        2  0.0  0.0      0    0 ?        S<  19:37  0:00 [kthreadd]
    USER      PID %CPU %MEM    VSZ  RSS TTY      STAT START  TIME COMMAND
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# free
                total        used        free      shared      buffers
    Mem:        126992      109320        17672            0        10168
    -/+ buffers:              99152        27840
    Swap:            0            0            0
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# cat /proc/meminfo
    MemTotal:      126992 kB
    MemFree:        17644 kB
    Buffers:        10180 kB
    Cached:          15796 kB
    SwapCached:          0 kB
    Active:          16000 kB
    Inactive:        14728 kB
    HighTotal:          0 kB
    HighFree:            0 kB
    LowTotal:      126992 kB
    LowFree:        17644 kB
    SwapTotal:          0 kB
    SwapFree:            0 kB
    Dirty:              12 kB
    Writeback:          0 kB
    AnonPages:        4760 kB
    Mapped:          3568 kB
    Slab:            73760 kB
    SReclaimable:    1084 kB
    SUnreclaim:      72676 kB
    PageTables:        428 kB
    NFS_Unstable:        0 kB
    Bounce:              0 kB
    WritebackTmp:        0 kB
    CommitLimit:    63496 kB
    Committed_AS:    9820 kB
    VmallocTotal:  1015800 kB
    VmallocUsed:      4100 kB
    VmallocChunk:  1009688 kB
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# cat /proc/slabinfo | awk '/slabdata/ { printf "
    %-20s%-10u%-10u%-10u%-15u%-15u\n", $1, $2, $3, $4, ($4*$2), ($4*$3) }' | sort -n
    -r -k5 | head -10
    size-2048          30190    30190    2048      61829120      61829120
    skbuff_head_cache  30460    30460    192      5848320        5848320
    size-64            16110    16110    128      2062080        2062080
    size-4096          158      158      4096      647168        647168
    size-8192          59        59        8192      483328        483328
    size-65536          7        7        65536    458752        458752
    size-32            3126      3150      128      400128        403200
    buffer_head        5942      5959      64        380288        381376
    size-512            431      472      512      220672        241664
    dentry              1588      1612      124      196912        199888
    Code:
    root@MIRAGE-ASUS:/tmp/home/root# lsmod
    Module                  Size  Used by    Tainted: P
    tun                    6464  1
    sit                    9040  0
    tunnel4                1808  1 sit
    nls_cp437              4416  0
    ip6table_filter          704  1
    ip6table_mangle          992  0
    xt_webmon              16320  2
    xt_length                704  2
    xt_recent              6800  4
    ip6t_LOG                7248  3
    xt_IMQ                  736  0
    imq                    2320  0
    ip6t_REJECT            2944  1
    nf_conntrack_ipv6      14176  8
    fuse                  38128  2
    usblp                  11312  0
    ehci_hcd              34640  0
    vfat                    9216  0
    fat                    46000  1 vfat
    ext2                  55520  0
    ext3                  113440  0
    jbd                    48352  1 ext3
    mbcache                4528  2 ext2,ext3
    usb_storage            33120  1
    sd_mod                21376  2
    scsi_wait_scan          384  0
    scsi_mod              75488  3 usb_storage,sd_mod,scsi_wait_scan
    usbcore              114736  4 usblp,ehci_hcd,usb_storage
    nf_nat_pptp            1440  0
    nf_conntrack_pptp      3808  1 nf_nat_pptp
    nf_nat_proto_gre        1072  1 nf_nat_pptp
    nf_conntrack_proto_gre    2464  1 nf_conntrack_pptp
    nf_nat_ftp              1568  0
    nf_conntrack_ftp        5792  1 nf_nat_ftp
    nf_nat_sip              5920  0
    nf_conntrack_sip      19008  1 nf_nat_sip
    nf_nat_h323            5504  0
    nf_conntrack_h323      37152  1 nf_nat_h323
    wl                  2610544  0
    dnsmq                  2032  1 wl
    et                    37312  0
    igs                    13584  1 wl
    emf                    17568  2 wl,igs
     
  93. Cyberian75

    Cyberian75 Network Guru Member

    Riddlah, at what % of RAM does your router reboot? Mine reboots at around 50%.
     
  94. jyavenard

    jyavenard Network Guru Member

    Well.. Then the issue isn't with radvd as that's what I updated to be identical with the asus source code...
     
  95. jyavenard

    jyavenard Network Guru Member

  96. shibby20

    shibby20 Network Guru Member

  97. shibby20

    shibby20 Network Guru Member

    because a second half of memory is mounted as tmpfs on /tmp :)
     
  98. jyavenard

    jyavenard Network Guru Member

  99. Riddlah

    Riddlah Networkin' Nut Member

    I haven't actually seen a reboot. When the memory drops to around 20mb it drops all connections and windows shows the connection as limited. I let it sit for about 5 minutes and it did not reboot. I had to manually pull the power on the router and reboot it.


    ---
    Riddlah
     
  100. shibby20

    shibby20 Network Guru Member

Share This Page