Edit data in rstats Bandwidth file

Discussion in 'Tomato Firmware' started by bdf0506, Dec 15, 2013.

  1. bdf0506

    bdf0506 Serious Server Member


    Somehow, I have some stray data in my rstats bandwidth file that shows the year 2035.

    I've stumbled upon this thread (http://tomatousb.org/forum/t-312300/anyway-to-view-rstats-bandwidth-file-in-clear-text), and that led me to a python script that allows me to view the data. Question is - how can I edit this data? I just want to remove the data that involves the year 2035. Or, alternatively, I would like to backup the data that doesn't involve anything in the future, and then restore that data.

    Is there a way to accomplish this? Whenever I look at the file, it seems to be encrypted or compressed somehow and is not in clear text.

  2. RMerlin

    RMerlin Network Guru Member

    The data is in binary format, and not easily editable.
  3. bdf0506

    bdf0506 Serious Server Member

    We say it isnt easily editable....is there ANY way to edit it? I really want to remove the bad data from the file.
  4. RMerlin

    RMerlin Network Guru Member

    It's binary data. You will have to study the source code to figure it out, and write your own program to manipulate it.
    koitsu likes this.
  5. kidont

    kidont Reformed Router Member

    2Rmerlin, is there any software for one-click view/decoding cstats and rstats?
  6. RMerlin

    RMerlin Network Guru Member

    No, that's why I said it wasn't easily editable - you will have to write your own software, after having studied its format in cstats/rstats (both are different BTW).
    koitsu likes this.
  7. Prave

    Prave Reformed Router Member

    I wrote a couple of perl scripts to convert the rstats binary file into plain text and re-convert it back to the rstats format file. See:

    convert_rstats_to_txt.pl http://pastebin.com/e96kCYra

    convert_txt_to_rstats.pl http://pastebin.com/XE4TupLC

    Basically the first one converts the rstats binary file into a single column text file with 264 lines, which one can edit in a plain text editor. Save and convert the text file usinfg the second script back to the rstats format. I’ve skipped the gzip portion, which needs to be performed before importing into tomato. Likewise the first script expects a gunzipped rstats binary file. Helped me to restore a few entries earlier this month when someone rebooted the router and I lost my history continuity – manually added them and merged them using this script.
    RMerlin likes this.
  8. Kim K

    Kim K Serious Server Member

    Hi Prave,

    Could you explain how to add entries to the file?

  9. bdf0506

    bdf0506 Serious Server Member

    Hi Prave,

    I'm excited that someone has figured out a way to do this! However, same question at Kim K... once you have the text file that is the 254 lines, what can you do with this? What does it all mean?

  10. Prave

    Prave Reformed Router Member

    Well – since there are a couple of requests, here is how I reverse engineered and jimmied up the Perl script to get to see inside. The unzipped rstats file will always be 2112 bytes – because it is composed of:

    1 ID version header

    (3 * 62) daily triplet values for date, download and upload bytes for 62 days (records)

    1 summary record for all daily records in the file

    (3 * 25) monthly triplet values for month begin date, download and upload bytes for 25 months (records)

    1 summary record for all monthly records in the file

    So 1 + (3*62) + 1 + (3*25) + 1 = 264

    That is the explanation for the 264 lines the raw rstats file which can be dumped into plain text. Also, each entry is forced and written as an 8 byte word, so264 * 8= 2112, the unzipped file size, which is constant regardless of the data values recorded in there.

    I created a spreadsheet with the explanations athttps://app.box.com/s/b2oqyivq60y6446i7j48It is a dump using the perl script [convert_rstats_to_txt.pl] from earlier this month of my rstats file when my router was rebooted – It has these entries:


    2014/02/08 : Upload 125461274 : Download 1969672502
    2014/02/09 : Upload 388040686 : Download 7499982719
    2014/02/10 : Upload 284661722 : Download 1885114513

    2014/02/01 : Upload 798163682 : Download 11354769734

    I noticed the loss of continuity in my tomato page and ended up merging this rtats file with an older backup to get them back in order.

    To merge two rstats raw files – run convert_rstats_to_txt.pl on both the raw inputs, use a spreadsheet to merge the text entries appropriately with the proper date keys, and finally use convert_txt_to_rstats.pl to get the raw 2112 byte tomato_rstats file. Gzip it and upload to tomato – worked for me. Basically save column B in the spreadsheet as a plain text file and regenerate the tomato_rstats file.

    This is a key reference for the data structures: http://repo.or.cz/w/tomato.git/blob/refs/heads/Toastman-RT:/release/src/router/rstats/rstats.c

    Hope this does not sound like a reddit ELI5, let me know if you folks need further help.
  11. Kim K

    Kim K Serious Server Member

    Thanks Prave for the explanation.

    Now if there was something similar for the iptraffic cstats which I never seem to be able to import after an upgrade.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice