1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Virgin Media UK STM Monitoring Script

Discussion in 'Tomato Firmware' started by tievolu, Dec 16, 2008.

  1. tievolu

    tievolu Network Guru Member

    STM Monitor v1.12 is now live.

    Let me know if you have any problems.
     
  2. rs232

    rs232 Network Guru Member

    Hi tievolu,
    thankyou so much to keep this updated knowing you are very busy with other things in your life. Can we just know what's the difference between v1.11 and v1.12?

    Appreciated!
     
  3. tievolu

    tievolu Network Guru Member

    v1.12 takes into account the fact that VM now only cut the bandwidth in the direction that triggered STM, rather than limiting both. See posts 97-100.
     
  4. rs232

    rs232 Network Guru Member

    cool! thanks.
     
  5. rs232

    rs232 Network Guru Member

    Today I breeched my limit in downstream and I can confirm that the limit was applied only to my downloads. Cool!

    Just an insignificant note:

    STM Mitigation: Active - Downstream STM sentence will end at :56


    The ":56" used to be "0:56" not sure this is wanted but just to let you know any ways.

    Thanks again!!
    rs232
     
  6. paulies

    paulies Serious Server Member

    Hi all,

    Any chance of some help with this script I have tried for hours tryig to get it to work but get nothing. I am not the most technical person but can follow instructions ;).

    I have tried the trusted and secure and after each attampe I receive no erros but the web page you goto is just a white page and thats it.

    Hope somebody can help

    Thanks

    Paul
     
  7. rs232

    rs232 Network Guru Member

    1) Do you have QoS enabled on your router?
    2) What contract/speed do you have? -> you should be able to find this on your "my virgin media page"

    Any ways:

    Copy the following in Administrator/Scripts/WAN UP


    NOTE this script above is based on ML20 contract (20Mb down 1Mb up) You should change this according to your VM contract. Possible options are:

    Size: S (5Mb)
    Size: M (10Mb)
    Size: L (10Mb)
    Size: XL (20Mb)
    Size: ML 20 (20Mb)
    Size: L30 (30Mb)
    Size: XL 30 (30Mb)
    Size: XXL 50 (50Mb)
    Size: XL 60 (60Mb)
    Size: XXL 100

    Ref: http://help.virginmedia.com/system/...R_TYPE=Cable&CMD=VIEW_ARTICLE&ARTICLE_ID=2781

    Once saved into WAN UP, reboot the router. After 5 minutes you'll find a page at your router address admin page + "/ext/stm-monitor.htm". e.g.

    https://192.168.1.1:80/ext/stm-monitor.htm
     
  8. paulies

    paulies Serious Server Member

    Thanks it nows seems to work.

    Quick question will changing the QOS settings to none effect this script ?
     
  9. rs232

    rs232 Network Guru Member

    The only think you it needs from QoS (AFAIK) are bandwidth in and bandwidth out.

    e.g. for my ML20 I've set

    in: 18000
    out: 840
     
  10. paulies

    paulies Serious Server Member

    Thanks got it working :)

    the -i works a treat if I select -p it just says disabled, I'd rather not hit the cap and have speed reduced any ideas ?
     
  11. rs232

    rs232 Network Guru Member

    I strongly advice not to use prevention. Breaching the cap is not a big drama especially with the new speeds out there.
    Getting close to your cap limit and using prevention can seriously impact your internet connection and make it literally unusable until the time frame is finished.
    When I say unusable I mean unable to load a any webpage.
    Trust me I've tried them all ;-)
     
  12. paulies

    paulies Serious Server Member

    Hi,

    I have been using this script with some sucess but since the 21st June I was getting stm'd a lot anyway after 5 days on the phone asking why I was constantly getting traffic managed apparenlty on the 21st of June the new STM was sent live. The operater gave me the 30mb details which I will list below could this be incorapated into the script ?

    9am to 3pm 4GB speed reduced to 15mb for 1 hour if you carry on downloading and exceed another 5.5GB you are reduced to 9mb for 2 hours
    3pm til 00:00 2.8GB reduced to 15mb for an 1 hour if you carry on downloading and exceed 3.6GB your speed is reduced to 6mb for 2 hours
    00:00 to 9am Unlimited

    Seems to me Virgin have given free upgrades and need users to cut back to stop network getting to busy.

    I asked why this wasn't on the site he could not answer

    To me this is CRAP should have changed to sky while I had chance.

    Paul
     
  13. rs232

    rs232 Network Guru Member

  14. rs232

    rs232 Network Guru Member

    Tievolu, I've tried to create v1.13beta myself, you can find it here below in attachment:

    You migh or might not like it but things I've modified are:

    1) updated the connections codes, speeds and caps as per VM webpages:
    http://www.virginmedia.com/images/STM_20Mb_below_800pxA.jpg
    http://www.virginmedia.com/images/STM_30Mb_above_800pxA.jpg

    2) The script now displays only:
    • Physical ML20 limit
    • Current QoS values
    • VM STMed QoS values
    3) At the bottom of the page the customisable fields have been removed adding instead a link to your STM homepage where everything is well explained

    4) Within the script the:
    outbound_bandwidth_without_stm
    inbound_bandwidth_without_stm
    have been taken out of the loops and filled in with the NVRAM QoS settings.

    5) The list of speed loops has been modified from a series of
    IF -> THEN -> FI
    into a single
    IF -> THEN -> ELIF -> THEN -> ELIF -> THEN ....

    6) Each loop has two new variables $downs and $ups with the contract speed. I though of this to simplify future management/modification. Everything else within the loop is calculated.

    7) I didn't get the *0.91 in the loop calculation of v1.12. My understanding of QoS is to apply a 15-25% reduction to the nominal speed, but in any case I thought that comes from QoS now so this issue it's dealt with on the QoS page. The way it works in this beta is:
    a) not STMed --> valued coming from the QoS config page
    b) STMed --> Values calculated from the physical link speed and %restriction only.

    The only point I wasn't very sure about about B) is whether apply simply
    physicallinkspeed * %STMrestriction
    or
    physicallinkspeed * %STMrestriction * *somepercentage* (something in between 75-85%) to prevent QoS to use all the STM bandwidth) so I left it physicallinkspeed * %STMrestriction for the time being.

    8) Page status redesign, including a progress bar. Not that there was something wrong with the old one at all. just personal preference.

    After so many changes i would be surprise if there are no bugs, so whoever wants to test it please do.


    NOTE: I couldn't find a way to solve the bug when the STM is triggered and the message displayed is doesn't display the end hour correctly:
    STM Mitigation: Active - Downstream STM sentence will end at :02



    I'm not sure what you think about this version, I've just modified what I thought was worthy. Do with this script whatever your like, but I'm quite happy to maintain it considering you've express various time your concerns on the amount of time you can put on it in the future.

    Cheers
    rs232 :)
     

    Attached Files:

  15. rs232

    rs232 Network Guru Member

    hi Tievolu, I'm still working on my personal version of the script (beta4 in attachment). This version solved the hour missing bug when STMed.

    I wanted to ask you something quick.
    Can you please help me to work out what is the 720 in the total_transferred_bytes variable? Perhaps the last 10 hours of data?


    Code:
                total_transferred_bytes=`echo $data | awk '
                    {
                        values_string = substr($0, index($0, "[") + 1, index($0, "]") - 6);
                        split(values_string, values_array, ",");
                        for (i=(721-number_of_values); i<=720; i++) {
                            total += values_array[i];
                        } printf("%0.f" total);
                    }' number_of_values=$number_of_values total=0`

    many thanks!
     

    Attached Files:

  16. rs232

    rs232 Network Guru Member

    This is my personal version of the script. I'll call you STM-Monitor-rs232_v1 as I don't want to interfere with future Tievolu's versions if any Here the differences:

    0) Based on Tievolou's v1.12. All the credit to him for his excellent script!
    1) removed prevention, now the script accepts no parameters other then the mandatory contract speed
    2) solved a bug with sentence end hour missing
    3) redesign + progress bar
    4) minor optimisation
    5) Use of a new temporary file (/tmp/stm-monitor.var)
    6) simplified the contract definition to speed up future changes if any
    7) added reference to hh:mm left to the end of the sentence
    8) added reference to last STM end time
    9) changes to the crontab definition. As there's no prevention the script is updated every 5min within active periods and less frequently outside.
    10) added reference to transferred data after the period if ended (reset daily after 1am)
    11) added two new customisable variables: $QOS_IN_PERCENTAGE & $QOS_OUT_PERCENTAGE:
    12) added reference to hostname/domain (in case you have multiple routers/connections)


    Installation:
    A) copy the script somewhere safe (e.g. cifs or usb). I'll use cifs1 in this example below, change accordingly.

    B) Add the following lines to Admin/scripts/WAN UP
    Code:
    cp /cifs1/stm-monitor-rs232_v1.sh /tmp/stm-monitor.sh
    chmod 755 /tmp/stm-monitor.sh
    cru a STM-Monitor-day "1-59/5 10-20 * * *  /tmp/stm-monitor.sh [Broadband contract code]"
    cru a STM-Monitor-evening "10,30,50 21-23 * * * /tmp/stm-monitor.sh [Broadband contract code]"
    cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [Broadband contract code]"
    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"
    Replace "Broadband contract code" above with your/ Supported code are:
    "S5", "M10", "L10", "XL20", "ML20", "L30", "XL30", "XXL50", "XL60", "XXL100" or "100"
    If unsure double check these pages:
    http://www.virginmedia.com/images/STM_20Mb_below_800pxA.jpg
    http://www.virginmedia.com/images/STM_30Mb_above_800pxA.jpg

    C) Point your browser to
    Code:
    http(s)://IP:(port)/ext/stm-monitor.htm
    Now either wait until the script is run by the crontable (details above), or for the inpatients go to Tomato's admin interface tools/system and run:
    Code:
    /tmp/stm-monitor.sh [Broadband contract code]
    NOTE: The XL60 speed has an upload of only 3Mb. Reading up in internet this will be put at 6Mb gradually on a area basis. So expect a new code to come out to differentiate between XL60/3 and XL60/6

    Cheers
    rs232
     

    Attached Files:

  17. rs232

    rs232 Network Guru Member

    FYI, is you are on anything less than 30M call virgin media and they'll put you immediately on 30/2 (so called L30). They will not do this automatically unless you ask!
    Mind you you need to have a superhub or they won't upgrade. If you don't have it they'll offer to send a free one for 20 GBP. Unless your old modem is broken and the delivery is free of course...
     
  18. a.fiodorov

    a.fiodorov Serious Server Member

    I have added support for saving data in case somebody restarts the router.

    http://pastebin.com/Wj47ZCLb

    It's based on the original script.
    This might be still buggy, but the usage is: stm-monitor.sh [S5|M|L|XL|ML20|L30|XL30|XXL|XL60|XXL100] [-p|-i] [-f] BACK-UP_FILE
    The back-up file has 3 lines:
    timestamp
    rx_data
    dx_data

    Each run the script pulls as much as possible relevant data from the back-up file to rx_data (tx_data) variable, adds the newest data from rstats and saves the updated rx_data (tx_data) to the back-up file.
     
  19. a.fiodorov

    a.fiodorov Serious Server Member

    I realised the above would exit with "file not writeable" error if the back-up file doesn't exist yet (on its first run), so here's a quick fix: http://pastebin.com/QQ5jvKnG
     
  20. rs232

    rs232 Network Guru Member

    Hi, this sounds interesting.
    Could you:
    1) post a screenshot of what it looks like
    2) post a diff to see what it was added to the original script? I wouldn't mind add this to my version of the script if you don't mind.

    Regards
     
  21. a.fiodorov

    a.fiodorov Serious Server Member

    Hey. Screenshot would be identical to the original, so omitted.

    I created a new Pastebin post, which highlights the difference from the original (v1.12), you should be able to add it easily and feel free to:

    http://pastebin.com/diff.php?i=bacvb4XG
     
  22. a.fiodorov

    a.fiodorov Serious Server Member

    I changed the way we read the parameters, so examine the start of the script.​
    Note the main block regarding back-ups is​
    486
    if [[ $use_backup -eq 1 ]]; then
    ..​
    fi
    567
    for some reason is not highlighted.​
    Important changes are lines​
    471-
    481
    where the data is read from rstats file (I decided to get of the brackets straight away), subsequently​
    633-
    639
    had to be slightly changed.​
    Look out for line 786, which caused me problems, since it had ${1}, which should be just changed to $bandwith_type. I simply replaced $1 with $bandwith_type everywhere else.​
    Hope it helps.​
    P.S. regarding your question what was 720, it's just 24 (hours) * 60 (minutes) / 2 (minutes); 2 - for time period of rstats​
     
  23. rs232

    rs232 Network Guru Member

    Thanks for this! I'll look into this when I get a spare hour or so.

    I guess what I was asking with the screenshot is what is looks like when reporting historical data (if the scripts includes the reporting function at all of course).

    Thanks again
     
  24. a.fiodorov

    a.fiodorov Serious Server Member

    2rs232: I streamlined the main block. The new version is edited in pastebin #119 (http://pastebin.com/QQ5jvKnG). (I decided to use egrep instead of awk, it seems faster).

    I have also included a fix for people who put a script in a different directory than default:
    1. # STM Monitor version
    2. my_path="`dirname \"$0\"`" # relative
    3. my_path="`( cd \"$my_path\" && pwd )`"
    4. stm_monitor_version=`head $my_path/stm-monitor.sh | grep "STM Monitor" | sed 's/# //g'`

    Happy merging!
     
  25. tievolu

    tievolu Network Guru Member

    Haven't been here for a while. Just a quick note to let you all know that I won't be updating the script any more - I need to upgrade my routers and for budget reasons I'm switching to a couple of TP-Link Atheros-based routers which won't run Tomato (I'm looking at the Gargoyle firmware as the next best thing).

    Looks like the script is in safe hands though! Happy tinkering everyone...
     
  26. koitsu

    koitsu Network Guru Member

    Talk about overcomplicating it...

    Code:
    my_path=`dirname "$0"`
    stm_monitor_version=`grep -m 1 "STM Monitor" "$my_path/stm-monitor.sh" | sed 's/# //g'`
    
    1. There is no point to the use of double-quotes during the variable assignment; issues with spaces in paths happen during variable expansion, not assignment,
    2. As such, there is no need for the excessive escaping/backslashing of double-quotes,
    3. The cd && pwd is silly; if this is an attempt at error checking (i.e. if you can't cd to $my_path then bail out), then it's done wrong, because the script will continue on regardless if the cd failed or not; if you want error checking for this, add an if [ -f "$my_path/stm-monitor.sh" ]; then ... fi statement around the $stm_monitor_version assignment, then put your error handler in an else,
    4. Get rid of the call to head; just use grep -m 1, which limits the match count to 1, i.e. stop reading from stdin once the first match is encountered (starting at the top of the file), exit, and thus issuing SIGPIPE,
    5. Note that I'm using double-quotes around the variable and path in the grep (re: variable expansion), in case $my_path contains spaces,
    6. I'm sure there's a way to do all this with just sed, but can't be bothered to remember sed syntax these days. :)

    Code:
    $ /home/jdc/blah\ blah/x
    STM Monitor 1.0
    
    $ ls -l /home/jdc/blah\ blah/*
    -rw-------    1 jdc       users      59 Feb 13 06:16 /home/jdc/blah blah/stm-monitor.sh
    -rwx------    1 jdc       users     149 Feb 13 06:16 /home/jdc/blah blah/x
    
    $ cat /home/jdc/blah\ blah/stm-monitor.sh
    #
    # stuff we don't care about
    #
    #
    #
    # STM Monitor 1.0
    #
    #
    
     
  27. rs232

    rs232 Network Guru Member

    VirginMedia UK has released today the new fair-usage policies.
    http://www.virginmedia.com/images/STM20MblargeA.jpg
    http://www.virginmedia.com/images/STM30MblargeA.jpg

    I have a updated the mod script from v1 to v2.

    The main change are:

    1) the time-frame for the evening down period moved from 15-21 to 16-22
    2) added reference to the P2P/newsgroup time based limitation (usage independent).
    3) added the new 120 connection
    4) updated percentage of limitation during STM
    5) added link to VM fair usage policy tables
    6) cosmetic adjustments

    The installation procedure is also slightly different, I suggest you remove the old script and use either the online or offline version of the script here below in your wan up:

    online autoupdating version: (will change when I change the source file)
    Code:
    wget -O /tmp/stm-monitor.sh http://tomatothemebase.eu/wp-content/uploads/stm-monitor-rs232.zip
    unzip -o /tmp/stm-monitor.zip
    rm /tmp/stm-monitor.zip
    chmod 755 /tmp/stm-monitor.sh
    cru a STM-Monitor-day "1-59/5 10-21 * * * /tmp/stm-monitor.sh [broadband type]"
    cru a STM-Monitor-evening "10,30,50 22-23 * * * /tmp/stm-monitor.sh [broadband type]"
    cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [broadband type]"
    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"
    
    online static version: (will always download v2 and never update)
    Code:
    wget -O /tmp/stm-monitor.sh http://tomatothemebase.eu/wp-content/uploads/stm-monitor-rs232_v2.zip
    unzip -o /tmp/stm-monitor.zip
    rm /tmp/stm-monitor.zip
    chmod 755 /tmp/stm-monitor.sh
    cru a STM-Monitor-day "1-59/5 10-21 * * * /tmp/stm-monitor.sh [broadband type]"
    cru a STM-Monitor-evening "10,30,50 22-23 * * * /tmp/stm-monitor.sh [broadband type]"
    cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [broadband type]"
    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"
    
    Offline version: (will never change)
    Download the archive and extract the script on your local storage of preference (e.g. cifs1)
    Code:
     
    cp /cifs1/stm-monitor-rs232_v2.sh /tmp/stm-monitor.sh
     
    chmod 755 /tmp/stm-monitor.sh
    cru a STM-Monitor-day "1-59/5 10-21 * * * /tmp/stm-monitor.sh [broadband type]"
    cru a STM-Monitor-evening "10,30,50 22-23 * * * /tmp/stm-monitor.sh [broadband type]"
    cru a STM-Monitor-night "55 0-9 * * * /tmp/stm-monitor.sh [broadband type]"
    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"

    Where [broadband type] is the broadband type = "S5", "M10", "L10", "XL20", "ML20", "L30", "XL30", "XXL50", "XL60", "XXL100" , "100" or "120" (no quotes)

    You can find below the screenshot of mod rs232_V2. A zip archive with the script is also in attachment

    ScreenShot002.png

    Let me know if any problems.

    P.S. sorry to the people who gave suggestions lately I admit I haven't properly look into those yet, but I'm planning to sooner rather than later. I'm particularly interested in saving stats.

    Regards
    rs232
     

    Attached Files:

  28. rs232

    rs232 Network Guru Member

  29. DOFFactory

    DOFFactory Network Newbie Member

    bump. any news?
     
  30. rs232

    rs232 Network Guru Member

    I did get an 1/2 working script last year but they keep chenging the fair-usage policy. To be completely fair nowadays 50Mb is the slowest contracts which makes this script totally obsolete.
     
  31. DOFFactory

    DOFFactory Network Newbie Member

    Well, not really. The script was for every type of contract (S, L, XL, etc.). My problem is, that my gf watches Amazon Prime TV a lot. I watch something else. Her stream is unicast through Flash/Silverlight, but mine is UDP. So when she triggers the STM on, she can still happily watch her stream, but I am cut off with always re-buffering stream. Drives me mad ... QoS, Bandwidth Limiting does not help unfortunately :-/ The effect would be the same on XL and other cotracts too.
     
    Last edited: Jun 29, 2015
  32. rs232

    rs232 Network Guru Member

    Current contracts are 50/100/155 unless you're on a legacy one and you can get a free upgrade if you ring them. You can solve this in two ways:
    1) re-address you relationship and you watch the same thing together (just kidding :))
    2) you have a go with QoS as UDP traffic is not spikey (no attention needed) where on demand (youtube/iplayer) uses TCP which by nature tries use as much as possible. If you capture the latter type of traffic into a QoS class and limit inboud to an acceptable maximum you'll resolve nicely.

    P.S. this script would NOT resolve this scenario of yours any ways

    HTH
    rs232
     
  33. DOFFactory

    DOFFactory Network Newbie Member

    Thanks man.

    ad 1) yeah, we do, but I cannot watch all of those series :) and I cannot force her to watch war movies, documentaries, etc.
    ad 2) it seems that if I enable some slight bandwidth limit (I call it rather a cap) for her, with QoS, she is happy with it, everything works. However, I tried to be fair, and I enabled bandwidth limit for myself (higher than hers). When I do this, my Asus RT-N16 starts to suffer with my UDP packets. So I disabled it for myself. I must stress, I do not have such a big traffic like her.

    In general, my issue is only partially solved because she is sitting in the room with the router, however, I am in a 3rd room (2 walls between me and the router). So my lost UDP packets might be also due to this. Seems like I will have to invest to a internet extender through the power socket.

    Thanks again for your help, you encouraged me to test everything.
     
  34. Gagan

    Gagan New Member Member

    Good day,
    I am new in networking,
    It would be great help, if you could advice me below matter.

    I have bought Netgear WNR3500LV2 and flash with Tomato open source firmware,
    It is working perfectly, but we have 30 users,
    Is it possible that i can set data limit (limit quota) 2GB per user in router.
    appreciate ur help in advance .
     
  35. kthaddock

    kthaddock Network Guru Member

    One post is enough ! Don't triple post!
     

Share This Page