Plot CPU Usage (vs. time)

Discussion in 'Tomato Firmware' started by arrmo, Jul 22, 2012.

  1. arrmo

    arrmo LI Guru Member


    I'm thinking about trying to add a plot for this to my local build (as a starting point at least), but before I do ... is anyone else already working on this? My thinking was to plot CPU usage (and RAM usage?) as a function of time, sort of like the bandwidth plots vs. time.


  2. koitsu

    koitsu Network Guru Member

    This is actually already done for you -- minus the graphing part. "CPU Load" (not CPU Usage) is already "implemented as a function over time", meaning the 3 floating-point numbers shown are averages over 1 minute, 5 minute, and 15 minute intervals. This data can be easily obtained by using the /proc/loadavg file on Linux (very simple format, easy to parse; you can find docs on if you wish).

    root@gw:/tmp/home/root# cat /proc/loadavg
    0.00 0.00 0.00 1/33 7945
    (Not much going on on my router. :) )

    My advice would to graph the 1-minute-average loadavg value, and do not apply any averaging logic to it (again: that's already done for you. You don't want an average of an average -- the value will always be lower than expected in that case).

    I can show you some graphs of a FreeBSD machine which graphs these values if you want an example picture/indication of what to do.

    As for the graphing itself, it should be done using the exact same SVG-based method as what the bandwidth graphs etc. already use on Tomato/TomatoUSB. Please don't go looking at rrdtool or anything else; please please please don't. Just stick with the existing SVG model and it'll be great.

    Finally, about graphing RAM usage: we can talk about that if/when you have the CPU load graphing stuff working. Accurately graphing RAM usage is a little bit more tricky because on *IX systems there are different "kinds" of RAM usage to graph (VSZ vs. RSS, certain kernel buffers, etc.). It's not as simple as "this amount of RAM is used" and "this amount of RAM is free". For example, especially on FreeBSD, the VFS and UFS filesystem layers tend to cache a lot of I/O data, so you end up with graphs that show "basically no free memory" when in fact there's tons of free memory -- it's just not labelled "free", it's labelled "inactive". Here's what I'm talking about:

    last pid: 46572;  load averages:  0.00,  0.00,  0.00                                                       up 12+20:33:35  17:05:28
    37 processes:  1 running, 36 sleeping
    CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
    Mem: 71M Active, 463M Inact, 5846M Wired, 825M Buf, 1514M Free
    Swap: 16G Total, 16G Free

    I can point you to some example Linux memory graphs people use if you want, but like I said, it's not as simple as "memory used vs. memory available". Baby steps, right?

    DEXTER Reformed Router Member

    So, did you manage to do this?

    I was searching for a method to watch cpu load/mem usage over time like the bandwith usage, but somehow it seems none of the tomato firmwares have this feature.
  4. arrmo

    arrmo LI Guru Member

    Unfortunately haven't gotten back to this ... :(. Do you want to take a run at it?

  5. remlei

    remlei Networkin' Nut Member

  6. arrmo

    arrmo LI Guru Member

  7. rs232

    rs232 Network Guru Member

    I think a clever developement in this direction would add a "report" page to tomato displaying CPU/RAM/"number of sessions" using the same plot methos currently used for bandwidth real-time/last 24 hours (which actually should be called throughput really as bandwidth refers to maximum capacity)
  8. cloneman

    cloneman Addicted to LI Member

    I still don't understand how load works on this router - so a graph would be even less helpful to me; although a properly working one would be nice.

    My Sirq is at 40% when downloading @ 20mbit, and meanwhile my load is at 0.01. Doesn't make sense to me. IIRC htop was more accurate, fwiw.
  9. arrmo

    arrmo LI Guru Member


    Did a bit of digging in to this, and I don't think it's all that hard ... except that I can't understand the code that is currently used for plotting (not easy to follow, limited documentation ... :(). Can anyone explain it?

    I do have a thought, a few easy steps to make this work (if I can't figure out the current code),
    - write data (CPU load, later memory, etc.) to a csv file
    - use some JS based tool to plot (for example, HighCharts or dygraph ... see
    - of course, include this in the overall menu structure of Tomato, and refresh regularly

    Here is the only hitch - the size of the JS code to do this (~ 125 kB for dygraph). It may be that this is OK, in which case this is pretty easy ... or is this an issue? Could this be referenced, available elsewhere? Not sure if HighCharts my preference, have used it before) is available as a link or not.


    And another question ... any thoughts as to why the JS on Tomato is not minimized already (to save space)? That's an easy thing to do. One more thought too ... does anyone know how to set up the http server to allow the JS file to be zipped on the router side? It's not about download time, rather about "disk" space.

    Sorry for the rambling, want to hear folks thoughts.

  10. mstombs

    mstombs Network Guru Member

    Take a look at Asuswrt-merlin that has graphs of cpu and ram on front page (and chipset temperature elsewhere!).
    The readonly filesystem is heavily compressed with squashfs-lzma so compressing the web pages first wont reduce filesize - but would be good for ram/page load I guess. I recommend adding something to the build system to trim the web pages - that could leave them pretty formatted in the source?
  11. jerrm

    jerrm Network Guru Member

    I'd vote against this. The net savings on the compressed volume would be trivial. I'd much prefer things stay human readable.
  12. arrmo

    arrmo LI Guru Member

    FYI, I have some basics up and running, with HighCharts. You can zoom in and out, show data labels, export to a file (like the attached). Are folks interested?
    chart (1).jpeg chart.jpeg
    Ragtag and rs232 like this.
  13. Ragtag

    Ragtag Reformed Router Member

    Hi arrmo,

    Could you let us know the steps to seetiing this up?

    I really like what you have done and I want to set it up on my Tomato router too...
  14. arrmo

    arrmo LI Guru Member


    It's not quite ready for "prime time" yet, as it requires some temporary files and scripts - just seeing if it's feasible (looks to be), and if it's what folks are after.

    I can send you the temporary trial if you want to test it out - just let me know (and let me know what router you're running on).
    Ragtag likes this.
  15. Ragtag

    Ragtag Reformed Router Member

    Yeah sure, I would be happz to test it out and give you some feedback.

    I am currently using the ASUS N-18U router.

  16. arrmo

    arrmo LI Guru Member

    Actually, just realized ... what build are you using?

  17. Ragtag

    Ragtag Reformed Router Member

    Using Tomato Firmware 1.28.0000 -2.7-128 K26ARM USB AIO-64K (Jacky's Advanced Tomato) ;)
  18. rs232

    rs232 Network Guru Member

    Good stuff!
    Once this is delivered it shouldn't be too difficult to add other charts like this for e.g. "RAM" and "number of sessions" which can be very useful if projected against time.

    P.S. Is this fully dependent on Jacky's Advanced Tomato? I hope not...
  19. arrmo

    arrmo LI Guru Member

    FYI, update - getting there ... :).

    Sliders working, navigation, picture download, etc.

    Attached Files:

  20. Ragtag

    Ragtag Reformed Router Member

  21. rs232

    rs232 Network Guru Member

    I really like what you're doing and especially the selectable time scope (bottom). I have always wished such a function was available for throughput (Bandwidth24h) so that placing the starting/end point where needed you can work out how much data was transferred within the given period.

    One tip on the "CPU load" as per image posted, other system I've seen do report on the 1/5/15 min on the very same chart with 3 different colours. if you ask me what you're doing is already more than enough but just an input...

  22. arrmo

    arrmo LI Guru Member


    OK, working, about to push to the repository (so folks can build then for a variety of machines). Final picture.

    Attached Files:

    Monk E. Boy, Ragtag and rs232 like this.
  23. rs232

    rs232 Network Guru Member

    Awesome! What is the process count I see in the picture? Can the chart switch view?
    Finally might I suggest you use exactly the same time frames as per bandwidth? For consistency more than else...

  24. arrmo

    arrmo LI Guru Member

    Yep - you can turn plots on and off by clicking on their name - and zoom in, out, slide along the X-axis. All real-time, and data is added real time also.
    Monk E. Boy and Ragtag like this.
  25. Ragtag

    Ragtag Reformed Router Member

    Great! I should be able to track the load and see if there is enough horsepower in my router for additional enchansment! ;)
    Last edited: May 14, 2015
  26. Ragtag

    Ragtag Reformed Router Member

    How goes the repo work @arrmo ? Eagerly awaiting, let us know when you got everything up and set up.
  27. tomatosoup

    tomatosoup Networkin' Nut Member

    Ragtag likes this.
  28. Ragtag

    Ragtag Reformed Router Member

    @tomatosoup @arrmo thanks for this mod. However, I am still a newbie could either of you suggest how to implement this mod into Tomato Firmware 1.28.0000 -2.7-128 K26ARM USB AIO-64K (Asus RT-N18U)?

    Thanking you in advance.
  29. tomatosoup

    tomatosoup Networkin' Nut Member

    You'll need to download the sources and compile tomato to get an image including this enhancement.
  30. arrmo

    arrmo LI Guru Member

    FYI, pushing another update today - plots / monitoring of wireless noise floor (to see if you have interference going on). It will also be in the git repo!
  31. Elfew

    Elfew Network Guru Member

    Screenshots pkease
  32. arrmo

    arrmo LI Guru Member

    Enjoy! ... :)

    Attached Files:

    rs232 and Elfew like this.
  33. Elfew

    Elfew Network Guru Member

    Amazing, anything else on TODO list? ;)

    Maybe total traffic throughput for income and outcome connections; active users, etc?
  34. arrmo

    arrmo LI Guru Member

    Not that I know of ... suggestions?

    I am interested in making the normal traffic plots this way also, as it's client side then ... so you can interact with the plot (zoom in, rescale, etc.). But not sure I want to make changes that major.
  35. Elfew

    Elfew Network Guru Member

    What about RAM usage; Wireless temperature?
  36. arrmo

    arrmo LI Guru Member

    RAM I was thinking about - but as there are so many values ... any suggestions of what to go with?

    Not sure how to get wireless temp ... do you know?
  37. vincom

    vincom LI Guru Member

  38. rs232

    rs232 Network Guru Member

    This is turning into something not only stylish but but potentially very useful too! There are plenty of things you can monitor, just to mention few not named yet:

    - Number of wireless clients
    - Number of sessions
    - filesystem usage (USB/CIFS/JFFS)
    - uptime

    I must stress that having all the charts into the very same page called e.g. "Reporting" and be able to compare how things coincide would be incredibly useful!
  39. arrmo

    arrmo LI Guru Member

    To be honest, it's not all that difficult - and the output just needs to go to a csv file (easiest). I'm storing them in /www/ext (ram). Then just plot them!

    Wondering if there is an easy way to get the output data to csv files? I am creating commands (accessible from command line), but scripts to run on startup would serve the purpose as well.
  40. rs232

    rs232 Network Guru Member

    Creating .csv is not a major issue at all. Perhaps a modular approach is the one that works out best here. E.g. an interface (Reporting page) where element can be added from a list like CPU, RAM and number of sessions

    So an item is a "template" and the template comes with a "data feed". This way the workload can be split and other users can contribute with the most colourful metrics.

    With this approach you could also decide to allow the display only N key metrics out of the total e.g. 3/20

    It sounds to me that the hardest part, the one you created, is already done which is very reassuring
  41. arrmo

    arrmo LI Guru Member

    That makes a lot of sense! I was thinking something similar. Let me try to explain how it works now - not to bore you, but so we can figure out how to best make this work ... ;)

    I currently store a couple files (in /www/ext, but that could be configurable ... nvram variable?). The two files are,
    - full history: basically, a csv file, with all the data you want plotted. My command line programs (daemons) allow me to set parameters to create this file (sample time, how long of a history to maintain to limit file size, etc.). This file is used when going to the page ... to get all the data and plot it ("startup").
    - last entry: this file is just the last entry (matches the last line of full history). It's so I don't keep re-plotting the full graph, just add a single data point (faster!). The timing of graph refreshing should be faster than the update rate to the file (to avoid missing samples - not a showstopper of course)

    So perhaps we set an nvram variable (chart_root?), of where the files are (for flexibility)? Then, in that directory, one file to list the possible plots (chartList.csv ... each line = "Name of Plot","filename")? And then, two files for each plot ... filename-full.csv (full history), filename-last.csv (last entry). I could then allow items to be selected, and those plotted.


    We may also need a way to set graph parameters (title, scales, axes, etc.). Maybe one more csv file? Need to think about this one.
  42. rs232

    rs232 Network Guru Member

    There are multiple ways to get to the point of course. The one you took appears to me to be a very effective shortcut. Thinking on a wider scale though I believe it would be a good idea to integrate these custom charts with the existing bandwidth/bandwdth24. With such a sentence I mean a 2 way relationship (learn/use but also improve the current charts). There are multiple reasons for this:
    - all the charts will have the same style
    - you can actually improve the existing charts adding e.g. the very helpful slide bar to define the time slot
    - the time frames (1,2,4,6,12,18,24) is set at code level so we can just pick up the very same values

    Also, moving the chart styles into the CSS would be a great idea as TTB could cover this and users can decide what style to apply.

    On the data side, tomato currently takes two approaches, the real-time + last 24 hours are stored in RAM and (unless this changed recently) will not survive a reboot. The daily,weekly,monthly are instead stored periodically (e.g. hourly) and event driven (e.g. reboot) into a non volatile storage e.g. cifs

    The above approach did not happen by chance, you don't want to write on non volatile storage for each and every data point, but I think I'm saying something obvious here.

    So a RAM usage (just to pick up an example) "could" be summarised daily, weekly and monthly into 3 data points: average, 95%ile and Max.

    If you ask me I would nowadays retain the realtime data for at least 7 days and archive/summarise afterwards, but the 10 years old tomato code was developed on limited hardware so the 24 hour did make sense back then... we can definitely change this on modern platforms!

    I support the nvram variable idea; and if you think it's worthy do have some though on the realtime vs historical data handling.

    P.S. on the Reporting page idea... some thoughts are needed on how to integrate (replicate?) the existing throughput charts to make them coincide with the custom metrics (one below the other for troubleshooting purpose). Perhaps adding the custom charts into the existing pages can be an option, if so the Bandwidth menu needs a name change. Just adding meat on the fire...

    I have to say: well done so far! I wish I had more time to actively support his but I'm maxed out for the next few weeks at work unfortunately...

    rs232 :)
  43. arrmo

    arrmo LI Guru Member

    Agreed! I like the approach - my shortcut is because I don't really know how the current plots work, so changing them is more painful ... :(. If anyone can explain the current plots I'm definitely game to be part of this!

    Also here, you're pushing my abilities ... LOL. I agree with the approach, but again need some collaboration to make this work. Thoughts?

    This is sort of what I do, though for now it's all in RAM (tmpfs). Easy to move / write somewhere else, but my daemon(s) just write to a file, and keep it a fixed length (parameters to the executable programs). Not difficult to change.

    Yep, more good ideas, but also why I may start from the shortcut ... ;). Only because it makes for more work.

    Completely understand! I was slow to reply here too - not because of lack of interest, but busy with kids activities the last few days.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice