1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Where did my memory go?

Discussion in 'Tomato Firmware' started by BassKozz, Nov 20, 2007.

  1. BassKozz

    BassKozz Network Guru Member

    My WHR is running extremely low on free memory
    Is this normal? What is taking up all the memory, it's not like I am running the VPN version or anything.
     
  2. mstombs

    mstombs Network Guru Member

    Click on "Count cache memory as free memory" in the web gui
    http://lampiweb.com/tomato/admin-debug.htm
    and several more MB should appear! The Linux OS should make use of free ram for cache if it can, and give it back if needed.
     
  3. szfong

    szfong Network Guru Member

    This happened to me once before, try 30s reset, shut down then reflash with tftp, just to be safe.

    It may be a slightly corrupt flash. Also, after flash perform 30s reset.
     
  4. Macskeeball

    Macskeeball LI Guru Member

    Back up your config before resetting.
     
  5. BassKozz

    BassKozz Network Guru Member

    Did that and now I am showing little over 4,000.00KB FREE :)
    Thanks
    Is this necessary after doing mstomb's fix ("Count cache memory as free memory") ?
     
  6. mstombs

    mstombs Network Guru Member

    Note the tick box just changes presentation of the free memory - to stop you worrying!

    Also have a look at the output of the Linux command "top" from ssh or telnet session (CRTL-C to end) - should tell you what is using what memory and cpu cycles.
     
  7. BassKozz

    BassKozz Network Guru Member

    [​IMG]

    Does this help diagnose?
     
  8. szfong

    szfong Network Guru Member

    My whr-hp-g54, running about a month shows about 2,900 KB free memory or 20% free memory. Oftentimes 30 sec reset and a reflash takes care of everything. You're fine now.
     
  9. mraneri

    mraneri LI Guru Member

    A reflash is a completely unnecessary risk to resolve this kind of memory issue. RAM doesn't generally disappear because of any problem which would be fixed by flashing with the same firmware version. 30 sec reset also, shouldn't help unless you're constantly changing settings in your router, and you set stuff up differently after a reset.

    A reboot is a different story, but I think most will find, that when they don't count cache as "used" memory, they have plenty to spare, and reboots aren't necessary to address RAM concerns.

    My router's been up for 20+ days and has 7MB (49%) Free.
     
  10. dewx

    dewx LI Guru Member

    This happens to my WHR-G54S too. The free memory starts at 3000-6000 KB after a reboot and drops to less than 500 KB after 7-10 days. Afterwards, I lose almost all network connection because "out of memory" errors.
    Syslog.txt shows:
    Nov 20 10:46:52 user.warn kernel: nvram_commit(): init
    Nov 20 10:46:52 user.warn kernel: nvram_commit: out of memory
    .
    .
    .
    Nov 20 10:47:24 user.err kernel: Out of Memory: Killed process 2419 (dnsmasq).

    Will the "Count cache memory as free memory" prevent actual "out of memory" errors"? I don't like having to reboot every week.
     
  11. BassKozz

    BassKozz Network Guru Member

    From the sounds of it, It might be smart to setup the "Reboot Scheduler"...
    I now recall having to do this back when I was using dd-wrt, maybe it's just a necessity to keep these third-party firmwares up and running smoothly?
     
  12. mstombs

    mstombs Network Guru Member

    Looks absolutely fine to me - only comment you have 2 ssh sessions running (dropbear) - maybe you are using winscp and putty?
     
  13. mraneri

    mraneri LI Guru Member

    No. As someone else mentioned previously, this is just a display option. If the router is killing dnsmasq, not only are you out of memory for cache, but likely the cache has been flushed and you are STILL out of memory.

    The reboot scheduler is definitely a band-aid that will work for you, but something else is going on to cause your router's memory to become depleted. Maybe you can login via telnet and type "top" after a couple of days and see what's hogging all of your RAM?
     
  14. namaste

    namaste LI Guru Member

    Can u plz tell me steps how to do that.. my router uptime is only 6 hrs and i have seen most of the time its out of memory. If you can help me that would be gr8!
     
  15. mraneri

    mraneri LI Guru Member

    How to what?
    Log in and see what "top" shows you?

    or

    Setup the reboot scheduler?

    or something else?
     
  16. BassKozz

    BassKozz Network Guru Member

    Ok it's been 4 days, and I have the reboot scheduler setup to reboot every night, and I am still seeing these memory issues, also I still have 2 ssh (dropbear) sessions running and i am only using putty to connect (once)...

    [​IMG]

    What gives?
     
  17. mraneri

    mraneri LI Guru Member

    Your "top" screen shot shows you have no memory problem.

    2736K plus 5200k in flushable cache means basically you have nearly 8MB available.

    I think you don't need to worry about your memory. You started this thread because Tomato was showing only 712k free. Of course, you know now that's not really a relevant number. So you checked Count cache memory as free memory and you really have not seen a memory problem since...

    Unless you show less than 1MB with that config option checked, you really don't need to worry about memory.
     
  18. PeterT

    PeterT Network Guru Member

    I believe that dropbear (like many daemons) "forks" itself when a new connection occurs; so basically you have the originally spawned daemon listening for connections on port 22 (pid 110) and then the instance handling your current ssh session (pid 399).
     
  19. BassKozz

    BassKozz Network Guru Member

    Thanks for clearing that up guys :D
     
  20. pfoomer

    pfoomer LI Guru Member

    Where's my memory gone

    Hi

    Just to share my observations...


    I have switched on the 'Count cache memory as free memory'
    today, and it reports about 8mb available, however after
    an uptime of about 6 hours its dropped to about 5mb.

    Yesterday I had some insuficent memory messages, and deleting
    of dnsmasq processes.

    I am running this script

    while sleep 2; do
    if [ -e /tmp/ppp/link ] ; then
    led white off
    led amber on
    else
    led amber off
    led white on
    fi
    done

    to show Wan Up/Downs, in the WAN script section
    and I notice in top its running multiple times,
    like every connection its initiated, this may
    be my stupidity with the script, any clues?

    However the script doesnt appeat to take much resource.

    I am logging via syslog, not internally, which is how I spotted the
    process deletion's





    2533 root R 376 2526 11.8 2.5 top
    2516 root S 644 141 9.2 4.4 dropbear
    5663 root S 400 1 6.5 2.7 script_wanup.sh
    29992 root S 400 1 3.9 2.7 script_wanup.sh
    14091 root R 400 1 2.6 2.7 script_wanup.sh
    13330 root S 400 1 2.6 2.7 script_wanup.sh
    5599 nobody S 476 1 1.3 3.2 dnsmasq
    5655 root S 416 1 1.3 2.8 upnp
    167 root S 400 1 1.3 2.7 script_wanup.sh
    2727 root S 400 1 1.3 2.7 script_wanup.sh
    13598 root S 400 1 1.3 2.7 script_wanup.sh
    1343 root R 400 1 1.3 2.7 script_wanup.sh
    26989 root S 400 1 1.3 2.7 script_wanup.sh
    23981 root R 396 1 1.3 2.7 script_wanup.sh
    20632 root S 396 1 1.3 2.7 script_wanup.sh
    137 root S 304 1 1.3 2.0 httpd
    1 root S 512 0 0.0 3.5 init
    113 root S 480 1 0.0 3.2 nas
    23893 root S 440 1 0.0 3.0 pppoecd
    2526 root S 440 2516 0.0 3.0 sh
    141 root S 412 1 0.0 2.8 dropbear
    14904 root R 400 29992 0.0 2.7 script_wanup.sh
    14902 root R 400 167 0.0 2.7 script_wanup.sh
    14905 root R 396 20632 0.0 2.7 script_wanup.sh
    127 root S 368 1 0.0 2.5 crond
    119 root S 356 1 0.0 2.4 syslogd
    131 root S 336 1 0.0 2.2 rstats
    114 root S 312 1 0.0 2.1 klogd
    39 root S 264 1 0.0 1.8 buttons
    129 root S 264 1 0.0 1.8 telnetd
    14892 root R 244 13598 0.0 1.6 led
    14903 root R 240 5663 0.0 1.6 led
    14887 root S 232 2727 0.0 1.5 sleep
    14888 root S 232 13330 0.0 1.5 sleep
    14886 root S 232 26989 0.0 1.5 sleep
    3 root SWN 0 1 0.0 0.0 ksoftirqd_CPU0
    7 root SW 0 1 0.0 0.0 mtdblockd
    14900 root Z 0 23981 0.0 0.0 led
    14901 root Z 0 1343 0.0 0.0 led
    14897 root Z 0 14091 0.0 0.0 led
    6 root SW 0 1 0.0 0.0 kupdated
    2 root SW 0 1 0.0 0.0 keventd
    5 root SW 0 1 0.0 0.0 bdflush
    4 root SW 0 1 0.0 0.0 kswapd
     
  21. mstombs

    mstombs Network Guru Member

    Haha yes, if you add up the percentages of memory used by all those wanup scripts (and the sleeps that belong to them) its a signifcant percentage more than 50% of total ram if I read it correctly. Each instance needs a bash shell and starting/stopping processes. The wanup script runs every time the wan comes up so starts a new process!

    I've never fully understood the difference between wanup and firewall scripts - they both run every time the wan comes up! Suggest you use the init script with an initial "sleep 10" to create and execute a small script on its own with some code [untested] like:-

    Code:
    sleep 10
    msh=/var/tmp/myled.sh
    cat >$msh<<EOF
    #!/bin/sh
    while sleep 2; do
    if [ -e /tmp/ppp/link ] ; then
    led white off
    led amber on
    else
    led amber off
    led white on
    fi
    done
    EOF
    chmod +x $msh
    $msh &
     
  22. mraneri

    mraneri LI Guru Member

    No need to get fancy particularly with creating a script file (although that's probably good practice). Just move the script from WANUP to INIT, and it should only execute once...
     
  23. pfoomer

    pfoomer LI Guru Member

    Hi

    That appears to be working, will watch the memory and see if it drops after 5-6 hours.

    Thanks
     
  24. mstombs

    mstombs Network Guru Member

    My script above launches the led script as a separate process, allowing the init script and anything that calls it to complete and terminate. May not be necessary with Tomato but has been a problem for me elsewhere!

    Another alternative would be to create the executable script on /jffs and just call it from init with

    /jffs/myscript.sh &
     
  25. PeterT

    PeterT Network Guru Member

    As a safety measure it might be an idea to ensure within any script that there are no other instances of it running... Something along the lines of

    Code:
    # Count instances of script executing
    count=`ps | grep script_wanup.sh | grep -c -v grep`
    if [ $count -eq 1 ] ; then
      while sleep 2; do
        if [ -e /tmp/ppp/link ] ; then 
          led white off
          led amber on
        else
          led amber off
          led white on
        fi
      done
    fi
    
    The ps lists all executing processes, then the | grep script_wanup.sh finds all lines that contain script_wanup.sh (or whatever the script name is), next the grep -c -v grep ignores any of the lines that contained grep and returns the count of the remaining lines.

    As long as there is only one instance running, then the rest of the script executes.

    Of course, one could also kill any additional instances of the script.
     
  26. dewx

    dewx LI Guru Member

    I am starting to get out of memory errors again.

    Tomato:
    Total / Free Memory 14.20 MB / 276.00 KB (1.90%)

    Top:
    Mem: 14208K used, 336K free, 0K shrd, 572K buff, 2996K cached
    Load average: 0.01 0.00 0.00 (Status: S=sleeping R=running, W=waiting)
    PID USER STATUS RSS PPID %CPU %MEM COMMAND
    14864 root R 372 14862 1.3 2.5 top
    14601 nobody S 532 1 0.0 3.6 dnsmasq
    1 root S 480 0 0.0 3.2 init
    14862 root S 440 89 0.0 3.0 sh
    14641 root S 404 1 0.0 2.7 upnp
    14567 root S 368 1 0.0 2.5 crond
    14707 root S 360 1 0.0 2.4 udhcpc
    14566 root S 336 1 0.0 2.2 httpd
    14572 root S 336 1 0.0 2.2 rstats
    14554 root S 332 1 0.0 2.2 syslogd
    14559 root S 312 1 0.0 2.1 klogd
    39 root S 280 1 0.0 1.9 buttons
    105 root S 268 1 0.0 1.8 dropbear
    89 root S 260 1 0.0 1.7 telnetd
    3 root SWN 0 1 0.0 0.0 ksoftirqd_CPU0
    7 root SW 0 1 0.0 0.0 mtdblockd
    4 root SW 0 1 0.0 0.0 kswapd
    2 root SW 0 1 0.0 0.0 keventd
    5 root SW 0 1 0.0 0.0 bdflush
    6 root SW 0 1 0.0 0.0 kupdated

    Syslog:
    Dec 3 08:57:49 user.warn kernel: nvram_commit(): init
    Dec 3 08:57:49 user.warn kernel: nvram_commit: out of memory
    Dec 3 08:58:18 user.warn kernel: nvram_commit(): init
    Dec 3 08:58:18 user.warn kernel: nvram_commit: out of memory
    Dec 3 08:59:10 user.warn kernel: nvram_commit(): init
    Dec 3 08:59:10 user.warn kernel: nvram_commit: out of memory


    So I checked and unchecked "Count cache memory as free memory" a few times.
    Each time I got a little free memory back.

    Tomato:
    Total / Free Memory 14.20 MB / 2,672.00 KB (18.37%)

    Top:
    Mem: 11964K used, 2580K free, 0K shrd, 432K buff, 2024K cached
    Load average: 0.00 0.00 0.00 (Status: S=sleeping R=running, W=waiting)
    PID USER STATUS RSS PPID %CPU %MEM COMMAND
    15177 root R 372 15173 2.3 2.5 top
    15173 root S 440 89 0.0 3.0 sh
    14601 nobody S 420 1 0.0 2.8 dnsmasq
    1 root S 364 0 0.0 2.4 init
    14567 root S 356 1 0.0 2.4 crond
    14641 root S 348 1 0.0 2.3 upnp
    14572 root S 332 1 0.0 2.2 rstats
    14554 root S 328 1 0.0 2.2 syslogd
    14707 root S 324 1 0.0 2.2 udhcpc
    14566 root S 312 1 0.0 2.1 httpd
    14559 root S 296 1 0.0 2.0 klogd
    39 root S 280 1 0.0 1.9 buttons
    105 root S 268 1 0.0 1.8 dropbear
    89 root S 260 1 0.0 1.7 telnetd
    3 root SWN 0 1 0.0 0.0 ksoftirqd_CPU0
    7 root SW 0 1 0.0 0.0 mtdblockd
    4 root SW 0 1 0.0 0.0 kswapd
    2 root SW 0 1 0.0 0.0 keventd
    15178 root Z 0 14567 0.0 0.0 sh
    5 root SW 0 1 0.0 0.0 bdflush
    6 root SW 0 1 0.0 0.0 kupdated

    What was using up the memory? Was the cache not getting flushed?
     
  27. rcordorica

    rcordorica Network Guru Member

    I recently had this happen to me the first time on Tomato... "out of memory" and then Dnsmasq get's killed (and DHCP dies)

    It turns out a Steam update, www.steampowered.com, caused the client to open up thousands of connections in a short period of time. Obviously the 16MB of router RAM couldn't handle it.

    The solution was to lower my max connections to around 1400 connections (the point where the RAM is almost full), so that Dnsmasq doesn't get killed.

    At least that way once a connection expires, or gets dropped by iptables, another one can be started.
     
  28. dewx

    dewx LI Guru Member

    Still having out of memory problems. What happened to the memory?

    Model Buffalo WHR-G54S

    Time Sun, 23 Dec 2007 18:34:03 -0800
    Uptime 16 days, 05:30:56
    CPU Load (1 / 5 / 15 mins) 0.00 / 0.04 / 0.09
    Total / Free Memory 14.20 MB / 620.00 KB (4.26%)

    Syslog:
    Dec 23 18:39:44 user.err kernel: Out of Memory: Killed process 12329 (udhcpc).
    Dec 23 18:39:49 user.err kernel: Out of Memory: Killed process 12329 (udhcpc).
    Dec 23 18:39:55 user.err kernel: Out of Memory: Killed process 12329 (udhcpc).
    Dec 23 18:39:56 daemon.info dnsmasq[12384]: started, version 2.40 cachesize 150
    Dec 23 18:39:59 daemon.info dnsmasq[12384]: compile time options: no-IPv6 GNU-getopt no-RTC no-ISC-leasefile no-DBus no-I18N no-TFTP
    Dec 23 18:40:00 user.err kernel: Out of Memory: Killed process 12384 (dnsmasq).
    Dec 23 18:40:05 user.err kernel: Out of Memory: Killed process 12329 (udhcpc).

    [​IMG]

    [​IMG]
     

Share This Page