1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Virgin Media UK STM Monitoring Script

Discussion in 'Tomato Firmware' started by tievolu, Dec 16, 2008.

  1. tievolu

    tievolu LI Guru Member

    Virgin Media UK cable internet connections are subject to a traffic management policy (commonly referred to as STM) that cuts your bandwidth by 50-75% if you download/upload a certain amount of data between certain hours of the day, as described here. When this happens, your QOS settings become useless and your connection usually slows to a crawl.

    Over the last week or so I've been working on a script to monitor bandwidth usage (via rstats) and automatically reconfigure QOS if I go over my STM limit. It seems to work ok, so I thought I'd share it to see if anyone else was interested.

    I've set up a small web page here describing it and providing instructions.

    The script itself is available here.

    You can enable the script on your router by adding the following to your "WAN Up" script and rebooting:

    Code:
    wget -O /tmp/stm-monitor.sh http://www.tievolu.co.uk/stm/stm-monitor.sh
    chmod 755 /tmp/stm-monitor.sh
    cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh [b][broadband type][/b]"
    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"
    
    Where [broadband type] = M, M+, L or XL

    Here's the first version of the script, just so you can see what's involved:

    Code:
    #!/bin/sh -x
    #
    # Tomato STM Monitor v1.00
    # Written by tievolu
    # http://tievolu.googlepages.com/stm-monitor.html
    #
    # This script is for use on Virgin Media UK cable internet connections,
    # which implement Subscriber Traffic Management (STM) i.e. the WAN
    # bandwidth is limited for a set time when an predefined amount of data
    # is transferred between certain hours of the day.
    #
    # It allows the WAN bandwidth usage to be automatically monitored and
    # suitable QOS settings applied if an STM limit is exceeded. When the
    # STM sentence has been served the QOS settings return to normal.
    #
    # Add the following to your "WAN Up" script to download the script and
    # set it to run every two minutes:
    #
    #    wget -O /tmp/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh
    #    chmod 755 /tmp/stm-monitor.sh
    #    cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh [broadband type]"
    #    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"
    #
    #    Where [broadband type] = "M", "L" or "XL" (no quotes)
    #
    # Events are recorded in the system log, and details are displayed in a
    # small web page located at:
    #
    #    http://[router IP address]/ext/stm-monitor.htm
    #
    # The script assumes that the time on your router is accurate to within
    # a minute or so, so make sure your NTP settings are correct! It also only
    # works correcly if your router is left on throughout each STM period,
    # because rebooting destroys the Tomato bandwidth stats this script relies on.
    # Note that the actions of this script (i.e. setting QOS to deal with an STM
    # sentence) use NVRAM variables so they *will* survive a reboot (but not a
    # power cycle, because the NVRAM variables are not committed to flash).
    
    # Logging command
    LOGGER="logger -t STM-Monitor"
    
    # STM Monitor version
    stm_monitor_version=`head /tmp/stm-monitor.sh | grep "STM Monitor" | sed 's/# //g'`
    
    # Check command line argument
    if [[ "$#" -ne 1 && "$1" != "M" && "$1" != "L" && "$1" != "XL" ]]
    then
    	$LOGGER "Usage: stm-monitor.sh [M|L|XL]"
    	exit
    fi 
    
    # The upload QOS bandwidth values are set at ~91% of the cable modem
    # bandwidth limiter (download set at 100%), which works well for my
    # connection. Cable connections tend to be pretty stable so they should work
    # ok for most users. The STM related variables are based on the information
    # published by VM here: http://allyours.virginmedia.com/html/internet/traffic.html
    
    # STM period definitions (time values are hours of the day e.g. 16 = 16:00 = 4pm)
    stm_period_name_1="Daytime Downstream"
    stm_period_start_1=10
    stm_period_end_1=15
    stm_period_direction_1="RX"
    
    stm_period_name_2="Evening Downstream"
    stm_period_start_2=16
    stm_period_end_2=21
    stm_period_direction_2="RX"
    
    stm_period_name_3="Evening Upstream"
    stm_period_start_3=15
    stm_period_end_3=20
    stm_period_direction_3="TX"
    
    # STM sentence length (i.e. how long STM lasts for once it's triggered)
    stm_sentence_length_hours=5
    stm_sentence_length_seconds=`expr $stm_sentence_length_hours \* 3600`
    
    # Settings specific to XL (20Mb down, 768Kb up)
    if [[ $1 = "XL" ]] 
    then
    	# STM limits (values in MB)
    	stm_period_bw_limit_mb_1=6000
    	stm_period_bw_limit_mb_2=3000
    	stm_period_bw_limit_mb_3=1400
    	# QOS settings (values in Kbits/s)
    	inbound_bandwidth_without_stm=20480
    	inbound_bandwidth_with_stm=5120
    	outbound_bandwidth_without_stm=700
    	outbound_bandwidth_with_stm=174
    fi
    
    # Settings specific to L (10Mb down, 512Kb up)
    if [[ $1 = "L" ]]
    then
    	# STM limits (values in MB)
    	stm_period_bw_limit_mb_1=2400
    	stm_period_bw_limit_mb_2=1200
    	stm_period_bw_limit_mb_3=700
    	# QOS settings (values in Kbits/s)
    	inbound_bandwidth_without_stm=10240
    	inbound_bandwidth_with_stm=2560
    	outbound_bandwidth_without_stm=465
    	outbound_bandwidth_with_stm=116
    fi
    
    # Settings specific to M (2Mb down, 256Kb up)
    if [[ $1 = "M" ]]
    then
    	# STM limits (values in MB)
    	stm_period_bw_limit_mb_1=1000
    	stm_period_bw_limit_mb_2=500
    	stm_period_bw_limit_mb_3=200
    	# QOS settings (values in Kbits/s)
    	inbound_bandwidth_without_stm=2048
    	inbound_bandwidth_with_stm=1024
    	outbound_bandwidth_without_stm=233
    	outbound_bandwidth_with_stm=116
    fi
    
    # Set up web page. Use a string to store web page content and pipe
    # it to /var/wwwext/stm-monitor.htm when the script exits
    if [[ ! -e "/var/wwwext" ]]
    then
    	mkdir /var/wwwext
    fi
    web_page=""
    
    # Grab the current time
    current_hours=`date +%H`
    current_minutes=`date +%M`
    current_unix_time=`date +%s`
    
    # Check whether we are still serving an STM sentence that has completed
    if [[ "`nvram get stm_sentence_active`" = 1  && $current_unix_time -gt "`nvram get stm_sentence_end_unix_time`" ]]
    then
    	# Return QOS configuration to non-STM settings
    	$LOGGER "STM sentence served: applying normal QOS configuration"
    	nvram set qos_obw=$outbound_bandwidth_without_stm
    	nvram set qos_ibw=$inbound_bandwidth_without_stm
    
    	# Restart QOS
    	service qos restart
    
    	# Unset our NVRAM variables
    	nvram unset stm_sentence_active
    	nvram unset stm_sentence_end_time
    	nvram unset stm_sentence_end_unix_time
    fi   
    
    # Grab current statistics from rstats and wait for the file to be written
    killall -USR1 rstats
    sleep 2
    stats_file=/var/spool/rstats-speed.js
    
    # Read the stats file and extract the parts we're interested in (vlan1 RX and TX)
    read_next_rx_tx=0
    while read line
    do
    	case $line in
    		*vlan1*) read_next_rx_tx=1 ;;
    	esac
    
    	if [[ $read_next_rx_tx = 1 ]]
    	then
    		case $line in
    			*rx:*) rx_data=$line ;;
    			*tx:*) tx_data=$line ; read_next_rx_tx=0 ;;
    		esac
    	fi
    done < $stats_file
    rm $stats_file
    
    # Check which STM periods are active, check total TX/RX against
    # STM limits and note if an STM limit was exceeded
    stm_sentence_triggered=0
    web_page="${web_page}<table cellpadding=9 cellspacing=3 border=0>"
    web_page="${web_page}<tr align=\"center\" bgcolor=\"DDDDFF\"><th>STM Description<th>Start</th><th>End</th><th>Bandwidth Limit</th><th>Bandwidth Used</th><th>Average Rate</th></tr>"
    for i in 1 2 3; do
    	stm_period_name=`eval echo "\\$stm_period_name_$i"`
    	stm_period_start=`eval echo "\\$stm_period_start_$i"`
    	stm_period_end=`eval echo "\\$stm_period_end_$i"`
    	stm_period_direction=`eval echo "\\$stm_period_direction_$i"`
    	stm_period_bw_limit_mb=`eval echo "\\$stm_period_bw_limit_mb_$i"`
    	
    	if [[ $current_hours -ge $stm_period_start && $current_hours -lt $stm_period_end ]]
    	then
    		web_page="${web_page}<tr bgcolor=\"EEEEEE\" align=\"center\">"
    		web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td>"
    
    		# Calculate the number of values to take from the rstats data (each value covers two minutes)
    		minutes_since_stm_period_started=`echo "" | awk '{ print((((hours - stm_start) * 60) + minutes)); }' hours=$current_hours minutes=$current_minutes stm_start=${stm_period_start}`
    
    		number_of_values=`expr $minutes_since_stm_period_started / 2`
    		
    		if [[ $stm_period_direction = "RX" ]]
    		then
    			data=$rx_data
    		else
    			data=$tx_data
    		fi
    		
    		if [[ $number_of_values != 0 ]]
    		then
    			# Sum the last XXX values of the 720 in the rstats data to get the
    			# total bytes transferred since the STM period started 
    			total_transferred_bytes=`echo $data | awk '
    				{
    					values_string = substr($0, index($0, "[") + 1, index($0, "]") - 6);
    					split(values_string, values_array, ",");
    					for (i=(721-number_of_values); i<=720; i++) {
    						total += values_array[i];
    					} printf("%0.f" total);
    				}' number_of_values=$number_of_values total=0`
    			total_transferred_mb=`echo "" | awk '{ printf("%.2f", bytes / 1048576) }' bytes=$total_transferred_bytes`	
    			average_rate=`echo "" | awk '{ printf("%.2f", bytes / (number_of_values * 2 * 60 * 1024)) }' number_of_values=$number_of_values bytes=$total_transferred_bytes`
    		else
    			total_transferred_bytes="0"
    			total_transferred_mb="0.00"
    		    everage_rate="0.00"
    		fi	
    	
    		# Calculate maximum average rate (i.e. rate required to trigger STM),
    		# then compare this to our actual rate. If the actual rate is equal or
    		# higher, we paint the cell red, if it's greater that 75% of the max
    		# we paint it amber, and if it's anything else we paint it green.
    		stm_period_max_ave_rate=`echo "" | awk '{printf((limit * 1024) / ((end - start)*3600))}' limit=$stm_period_bw_limit_mb start=$stm_period_start end=$stm_period_end`
    		cell_color=`echo | awk '{ if (rate >= max) {print("FF6633")} else if (rate > (0.75 * max)) {print("FFCC33")} else {print("66FF33")}}' rate=$average_rate max=$stm_period_max_ave_rate`
    	
    		# Add data to web page
    		web_page="${web_page}<td>$total_transferred_mb MB</td><td bgcolor=\"${cell_color}\">$average_rate KB/s</td>"
    	
    		# Check whether STM limit has been exceeded
    		stm_period_bw_limit_bytes=`expr $stm_period_bw_limit_mb \* 1048576`
    		result=`echo "" | awk '{ if (total > limit) {print(1)} else {print(0)}}' total=$total_transferred_bytes limit=$stm_period_bw_limit_bytes`
    		if [[ $result = 1 && "`nvram get stm_sentence_active`" != 1 ]]
    		then
    			stm_sentence_triggered=1
    			nvram set stm_sentence_active=1
    			$LOGGER "STM Triggered: ${stm_period_name} (${stm_period_start}:00 -> ${stm_period_end}:00, ${stm_period_direction}, ${stm_period_bw_limit_mb} MB)"
    		fi
    	else
    		web_page="${web_page}<tr bgcolor=\"EEEEEE\" style=\"color:AAAAAA\" align=\"center\">"
    		web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td><td>N/A</td><td>N/A</td>"	
    	fi
    	web_page="${web_page}</tr>"
    done
    web_page="${web_page}</table>"
    
    # If STM is active, check whether QOS is set accordingly
    if [[ "`nvram get stm_sentence_active`" = 1  && "`nvram get qos_obw`" != $outbound_bandwidth_with_stm ]]
    then
    	if [[ $stm_sentence_triggered = 1 ]]
    	then
    		# We must have triggered an STM sentence on this run.
    		# Work out the hour component of the time the STM sentence will end
    		# (for logging purposes only - we will use unix time to track the
    		# STM sentence accurately)
    		stm_end_hour=`expr $current_hours + $stm_sentence_length_hours`
    		if [[ $stm_end_hour -gt 23 ]]
    		then
    			stm_end_hour="0`expr $stm_end_hour - 24`"
    		fi
    		
    		# Set NVRAM variables to describe the STM sentence being served
    		nvram set stm_sentence_end_time="$stm_end_hour:$current_minutes"
    		nvram set stm_sentence_end_unix_time=`expr $current_unix_time + $stm_sentence_length_seconds`
    
    		# Send information to log
    		$LOGGER "STM sentence started: applying STM QOS configuration (until $stm_end_hour:$current_minutes)"
    	else
    		# An STM sentence is active and we didn't trigger it on this run,
    		# but the QOS settings are wrong. We must have rebooted, or someone
    		# messed with the QOS settings. So we want to set the QOS settings
    		# to deal with the STM, but we don't want to overwrite our NVRAM
    		# variables.
    		# Send information to log
    		$LOGGER "STM sentence resumed: applying STM QOS configuration (until `nvram get stm_sentence_end_time`)" 
    	fi
    	
    	# Set QOS NVRAM variables
    	nvram set qos_obw=$outbound_bandwidth_with_stm
    	nvram set qos_ibw=$inbound_bandwidth_with_stm
    
    	# Restart QOS
    	service qos restart
    fi
    
    # Add QOS settings to web page
    web_page="${web_page}<p>Current QOS settings - <b>`nvram get qos_ibw`</b> kbits/s down, <b>`nvram get qos_obw`</b> kbits/s up</p>"
    
    # Add latest log entries to web page
    web_page="${web_page}<h3>Latest Log Entries</h3>"
    log=""
    while read line
    do
    	case $line in
    		*STM-Monitor*) log="${log}<nobr><tt>${line}</tt></nobr><br>" ;;
    	esac
    done < /tmp/var/log/messages
    web_page="${web_page}$log"
    
    # Publish web page
    # Write the title and status now, so that they accurately relfect the data in the table
    web_page_header="<html><head><title>${stm_monitor_version}</title><meta http-equiv=\"refresh\" content=\"120\"</head>"
    web_page_header="${web_page_header}<body style=\"font-family:verdana\"><h2>${stm_monitor_version}</h2>"
    web_page_header="${web_page_header}<h3>Status at ${current_hours}:${current_minutes} - "
    if [[ "`nvram get stm_sentence_active`" = 1 ]]
    then
    	web_page_header="${web_page_header}<b style=\"color:red\">STM has been triggered - the sentence will end at `nvram get stm_sentence_end_time`</b></h3>"
    else
    	web_page_header="${web_page_header}<b style=\"color:green\">STM has not been triggered</b></h3>"
    fi 
    web_page="${web_page_header}${web_page}</body></html>"
    echo $web_page > /var/wwwext/stm-monitor.htm
    
    Even if you're not interested in this particular functionality, there may well be stuff in the script that you find useful for other scripts. I learned quite a lot while writing it.

    Any feedback or suggestions most welcome.
  2. kardzzz

    kardzzz Networkin' Nut Member

    Tiev, thanks for the script, but I couldn't get it to work. Could you please verify I have the following correct for a VM L service

    wget -O /tmp/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh
    chmod 755 /tmp/stm-monitor.sh
    cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh L"
    logger -t STM-Monitor "Downloaded `head stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"

    I couldn't call the web page either
    http://192.168.1.1/ext/stm-monitor.sh

    I have 'logging' enabled. I could see your script was downloaded when i looked the the log

    thx
  3. tievolu

    tievolu LI Guru Member

    Can you log into the router and run "ls -l /tmp" and paste the output?

    If the script is there, try running it directly and tell me what happens i.e. run:

    /tmp/stm-monitor.sh L

    EDIT: The web page link will be http://192.168.1.1/ext/stm-monitor.htm (not .sh) That's probably your only problem - the WAN Up code looks fine (that was my fault btw - I've just corrected the web page - d'oh!)
  4. sunjon

    sunjon LI Guru Member

    Really nice script! Thanks a lot.
  5. kardzzz

    kardzzz Networkin' Nut Member

    tievolu, I managed to get this working and this script is fantastic, also great web interface. If I read correctly does the STM 5 hour period start from when you overstep the limits?

    Is it possible to script this so that i include the whole file in my wan-up code? (i.e. it doesn't go off to load a file from a server). For my security i'd like to know what is within a file everytime my router boots.
    Great work!!
    Thanks
  6. tievolu

    tievolu LI Guru Member

    Glad you like it :)

    While I promise not to do anything dodgy with the script, the possibility that I could if I was that way inclined is definitely there, so I completely understand your concerns about downloading it automatically.

    You could build the script entirely within the WAN Up script, but it would be a bit ugly - lots of echo commands, with the need to escape all the special characters in the correct way. It would be much easier to mount a JFFS partition and store a copy of the script there permanently.

    So, assuming your router has enough flash, and you like the script the way it is right now, create and mount your JFFS partition using the Tomato GUI (Administration -> JFFS2) and then issue the following commands (i.e. just telnet in and run them - don't add them to your WAN Up script):

    Code:
    wget -O /jffs/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh
    chmod 755 /jffs/stm-monitor.sh
    Then you would just need the following command in your WAN Up or startup script to set up the cron job when the router boots:

    Code:
    cru a STM-Monitor "1-59/2 * * * * /jffs/stm-monitor.sh [M|L|XL]"
    Hope this helps.
  7. tievolu

    tievolu LI Guru Member

    Just realised I didn't answer this before...

    Yes, the 5 hour punishment period begins when you go over the STM threshold.

    So, if you went over the limit for the 16:00 -> 21:00 period at, say, 18:30, you'd have your bandwidth reduced for five hours from that point i.e. until 23:30.

    Some would argue that it's a bit harsh, but that's the way it works.
  8. samuarl

    samuarl Networkin' Nut Member

    This script is excellent. Thank you tievolu.

    Although I do not really use QOS I have found the STM monitor page extremely useful. I live in a student house with 5 other people so we frequently go over the limit, usually within the last hour of the evening downstream period :mad:

    A suggestion I would like to make would be to have the QOS settings become stricter according to the amount of time and bandwidth left to prevent the limit being reached in the last couple of hours. I will expand upon this further if you need me to, but for know I am struggling to turn my thoughts into words :)

    Tanks again
  9. tievolu

    tievolu LI Guru Member

    I've thought about this too - it would be nice to have the option of preventing STM from kicking in, rather than merely coping with it when it does.

    I don't think it would be too hard to implement. I'll have a look into it...
  10. tievolu

    tievolu LI Guru Member

    Ok, here we go. There are now two modes of operation:

    STM Mitigation - This is the default mode. If an STM limit is exceeded, QOS limits are applied in accordance with the bandwidth limits imposed by STM.

    STM Prevention - If bandwidth usage is on target to exceed an STM limit, QOS limits are applied to prevent that from happening. Activated by the "-p" option (see below).

    Usage is as follows:

    Code:
    stm-monitor.sh [broadband type] [-p]
    
    With STM prevention enabled, the script monitors the average rate, checking whether it has exceeded the maximum rate (i.e. when the cell is red and you're on course to trigger STM). If this happens, the script calculates how much bandwidth you have left in this STM period, and limits the inbound or outbound QOS such that you will only be able to transfer 90% of that bandwidth before the STM period ends. The QOS limit returns to normal if the average rate drops below 90% of the maximum, or when the STM period ends.

    The end result of STM prevention is that you may take a more severe bandwidth cut than if you triggered STM, but that cut will usually last for a much shorter period of time. Personally, I'm not really sure whether prevention is better than mitigation, but YMMV.

    I've also modified the status page slightly. See this page for an example.

    Code:
    #!/bin/sh -x
    #
    # Tomato STM Monitor v1.01
    # Written by tievolu
    # http://tievolu.googlepages.com/stm-monitor.html
    #
    # This script is for use on Virgin Media UK cable internet connections,
    # which implement Subscriber Traffic Management (STM) i.e. the WAN
    # bandwidth is limited for a set time when an predefined amount of data
    # is transferred between certain hours of the day.
    #
    # It allows the WAN bandwidth usage to be automatically monitored and
    # suitable QOS settings applied. There are two modes of operation:
    #
    # 1. STM Mitigation
    #
    #    This is the default mode. If an STM limit is exceeded, QOS limits
    #    are applied in accordance with the bandwidth limits imposed by STM.
    #
    # 2. STM Prevention
    #
    #    If bandwidth usage is on target to exceed an STM limit, QOS limits
    #    are applied to prevent that from happening. Activated by the "-p"
    #    option (see below). 
    #
    # Add the following to your "WAN Up" script to download the script and
    # set it to run every two minutes:
    #
    #    wget -O /tmp/stm-monitor.sh http://tievolu.googlepages.com/stm-monitor.sh
    #    chmod 755 /tmp/stm-monitor.sh
    #    cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh [broadband type] [-p]"
    #    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"
    #
    #    Where [broadband type] = "M", "L" or "XL" (no quotes)
    #                      [-p] = Activates STM prevention
    #
    # Events are recorded in the system log, and details are displayed in a
    # small web page located at:
    #
    #    http://[router IP address]/ext/stm-monitor.htm
    #
    # The script assumes that the time on your router is accurate to within
    # a minute or so, so make sure your NTP settings are correct! It also only
    # works correcly if your router is left on throughout each STM period,
    # because rebooting destroys the Tomato bandwidth stats this script relies on.
    
    # Logging command
    LOGGER="logger -t STM-Monitor"
    
    # STM Monitor version
    stm_monitor_version=`head /tmp/stm-monitor.sh | grep "STM Monitor" | sed 's/# //g'`
    
    # Check command line arguments
    if [[ "$#" -eq 1 && "$1" != "M" && "$1" != "L" && "$1" != "XL" ]]
    then
    	$LOGGER "Usage: stm-monitor.sh [M|L|XL] [-p]"
    	exit
    fi
    if [[ "$#" -eq 2 && "$2" != "-p" ]]
    then
    	$LOGGER "Usage: stm-monitor.sh [M|L|XL] [-p]"
    	exit
    fi
    
    # For testing purposes
    if [[ "$2" = "-p" ]]
    then
    	preventative_mode=1
    fi 
    
    # The upload QOS bandwidth values are set at ~91% of the cable modem
    # bandwidth limiter (download set at 100%), which works well for my
    # connection. Cable connections tend to be pretty stable so they should work
    # ok for most users. The STM related variables are based on the information
    # published by VM here: http://allyours.virginmedia.com/html/internet/traffic.html
    
    # STM period definitions (time values are hours of the day e.g. 16 = 16:00 = 4pm)
    stm_period_name_1="Daytime Downstream"
    stm_period_start_1=10
    stm_period_end_1=15
    stm_period_direction_1="RX"
    
    stm_period_name_2="Evening Downstream"
    stm_period_start_2=16
    stm_period_end_2=21
    stm_period_direction_2="RX"
    
    stm_period_name_3="Evening Upstream"
    stm_period_start_3=15
    stm_period_end_3=20
    stm_period_direction_3="TX"
    
    # STM sentence length (i.e. how long STM lasts for once it's triggered)
    stm_sentence_length_hours=5
    stm_sentence_length_seconds=`expr $stm_sentence_length_hours \* 3600`
    
    # Settings specific to XL (20Mb down, 768Kb up)
    if [[ $1 = "XL" ]] 
    then
    	# STM limits (values in MB)
    	stm_period_bw_limit_mb_1=6000
    	stm_period_bw_limit_mb_2=3000
    	stm_period_bw_limit_mb_3=1400
    	# QOS settings (values in Kbits/s)
    	inbound_bandwidth_without_stm=20480
    	inbound_bandwidth_with_stm=5120
    	outbound_bandwidth_without_stm=700
    	outbound_bandwidth_with_stm=174
    fi
    
    # Settings specific to L (10Mb down, 512Kb up)
    if [[ $1 = "L" ]]
    then
    	# STM limits (values in MB)
    	stm_period_bw_limit_mb_1=2400
    	stm_period_bw_limit_mb_2=1200
    	stm_period_bw_limit_mb_3=700
    	# QOS settings (values in Kbits/s)
    	inbound_bandwidth_without_stm=10240
    	inbound_bandwidth_with_stm=2560
    	outbound_bandwidth_without_stm=465
    	outbound_bandwidth_with_stm=116
    fi
    
    # Settings specific to M (2Mb down, 256Kb up)
    if [[ $1 = "M" ]]
    then
    	# STM limits (values in MB)
    	stm_period_bw_limit_mb_1=1000
    	stm_period_bw_limit_mb_2=500
    	stm_period_bw_limit_mb_3=200
    	# QOS settings (values in Kbits/s)
    	inbound_bandwidth_without_stm=2048
    	inbound_bandwidth_with_stm=1024
    	outbound_bandwidth_without_stm=233
    	outbound_bandwidth_with_stm=116
    fi
    
    # Grab the current time
    current_hours=`date +%H`
    current_minutes=`date +%M`
    current_unix_time=`date +%s`
    
    # Set up web page. Use a string to store web page content and pipe
    # it to /var/wwwext/stm-monitor.htm when the script exits
    if [[ ! -e "/var/wwwext" ]]
    then
    	mkdir /var/wwwext
    fi
    web_page="<html><head><title>${stm_monitor_version} - Status at ${current_hours}:${current_minutes}</title><meta http-equiv=\"refresh\" content=\"120\"</head>"
    web_page="${web_page}<body style=\"font-family:verdana\"><h2>${stm_monitor_version}</h2>"
    web_page="${web_page}<h3>Status at ${current_hours}:${current_minutes}</h3>"
    
    # Check whether we are still serving an STM sentence that has completed
    if [[ "`nvram get stm_sentence_active`" = 1  && $current_unix_time -gt "`nvram get stm_sentence_end_unix_time`" ]]
    then
    	# Return QOS configuration to non-STM settings
    	$LOGGER "STM sentence served: applying normal QOS configuration"
    	nvram set qos_obw=$outbound_bandwidth_without_stm
    	nvram set qos_ibw=$inbound_bandwidth_without_stm
    
    	# Restart QOS
    	service qos restart
    
    	# Unset our NVRAM variables
    	nvram unset stm_sentence_active
    	nvram unset stm_sentence_end_time
    	nvram unset stm_sentence_end_unix_time
    fi   
    
    # Grab current statistics from rstats and wait for the file to be written
    killall -USR1 rstats
    sleep 2
    stats_file=/var/spool/rstats-speed.js
    
    # Read the stats file and extract the parts we're interested in (vlan1 RX and TX)
    read_next_rx_tx=0
    while read line
    do
    	case $line in
    		*vlan1*) read_next_rx_tx=1 ;;
    	esac
    
    	if [[ $read_next_rx_tx = 1 ]]
    	then
    		case $line in
    			*rx:*) rx_data=$line ;;
    			*tx:*) tx_data=$line ; read_next_rx_tx=0 ;;
    		esac
    	fi
    done < $stats_file
    rm $stats_file
    
    # Check which STM periods are active, check total TX/RX against
    # STM limits and note if an STM limit was exceeded
    stm_sentence_triggered=0
    web_page="${web_page}<table cellpadding=9 cellspacing=3 border=0>"
    web_page="${web_page}<tr align=\"center\" bgcolor=\"DDDDFF\"><th>STM Description<th>Start</th><th>End</th><th>Bandwidth Limit</th><th>Bandwidth Used</th><th>Average Rate</th></tr>"
    for i in 1 2 3; do
    	stm_period_name=`eval echo "\\$stm_period_name_$i"`
    	stm_period_start=`eval echo "\\$stm_period_start_$i"`
    	stm_period_end=`eval echo "\\$stm_period_end_$i"`
    	stm_period_direction=`eval echo "\\$stm_period_direction_$i"`
    	stm_period_bw_limit_mb=`eval echo "\\$stm_period_bw_limit_mb_$i"`
    	
    	if [[ $current_hours -ge $stm_period_start && $current_hours -lt $stm_period_end ]]
    	then
    		# This STM period is active.	
    		web_page="${web_page}<tr bgcolor=\"EEEEEE\" align=\"center\">"
    		web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td>"
    
    		# Calculate the number of values to take from the rstats data (each value covers two minutes)
    		minutes_since_stm_period_started=`echo "" | awk '{ print((((hours - stm_start) * 60) + minutes)); }' hours=$current_hours minutes=$current_minutes stm_start=${stm_period_start}`
    		number_of_values=`expr $minutes_since_stm_period_started / 2`
    		
    		if [[ $stm_period_direction = "RX" ]]
    		then
    			data=$rx_data
    			stm_period_rx_active=1
    		else
    			data=$tx_data
    			stm_period_tx_active=1
    		fi
    		
    		if [[ $number_of_values != 0 ]]
    		then
    			# Sum the last XXX values of the 720 in the rstats data to get the
    			# total bytes transferred since the STM period started 
    			total_transferred_bytes=`echo $data | awk '
    				{
    					values_string = substr($0, index($0, "[") + 1, index($0, "]") - 6);
    					split(values_string, values_array, ",");
    					for (i=(721-number_of_values); i<=720; i++) {
    						total += values_array[i];
    					} printf("%0.f" total);
    				}' number_of_values=$number_of_values total=0`
    			total_transferred_mb=`echo "" | awk '{ printf("%.2f", bytes / 1048576) }' bytes=$total_transferred_bytes`	
    			average_rate=`echo "" | awk '{ printf("%.2f", bytes / (number_of_values * 2 * 60 * 1024)) }' number_of_values=$number_of_values bytes=$total_transferred_bytes`
    			average_rate_kilobits=`echo "" | awk '{printf("%.2f", rate_kilobytes * 8)}' rate_kilobytes=$average_rate`
    		else
    			total_transferred_bytes="0"
    			total_transferred_mb="0.00"
    		    average_rate="0.00"
    			average_rate_kilobits="0.00"
    		fi	
    	
    		# Calculate maximum average rate (i.e. rate required to trigger STM),
    		# then compare this to our actual rate. If the actual rate is equal or
    		# higher, we paint the cell red, if it's greater than 90% of the max
    		# we paint it amber, and if it's anything else we paint it green.
    		stm_period_max_ave_rate=`echo "" | awk '{printf((limit * 1024) / ((end - start)*3600))}' limit=$stm_period_bw_limit_mb start=$stm_period_start end=$stm_period_end`
    		cell_color=`echo | awk '{ if (rate >= max) {print("FF6633")} else if (rate > (0.9 * max)) {print("FFCC33")} else {print("66FF33")}}' rate=$average_rate max=$stm_period_max_ave_rate`
    		
    		# STM Prevention code
    		if [[ $preventative_mode = 1 ]]
    		then
    			if [[ $cell_color = "FF6633" ]]
    			then
    				# Cell is red => calculate preventative QOS limit
    				# Calculate the number of kilobits we have left before we trigger STM
    				total_kilobits_remaining=`echo "" | awk '{print(((limit*1048576)-total)/128)}' limit=$stm_period_bw_limit_mb total=$total_transferred_bytes`
    							
    				# Calculate time remaining until end of STM period
    				seconds_remaining=`echo "" | awk '{print(((end_hour - current_hour - 1) * 3600) + (3600 - (60 * current_minute)))}' end_hour=$stm_period_end current_hour=$current_hours current_minute=$current_minutes`
    			
    				# Calculate a QOS limit such that we'll download a maxmum of 90% of this in the remaining time
    				preventative_qos_limit=`echo "" | awk '{printf("%.0f", 0.9*(kilobits/secs))}' kilobits=$total_kilobits_remaining secs=$seconds_remaining`
    				
    				if [[ $stm_period_direction = "RX" && "`nvram get qos_ibw`" = $inbound_bandwidth_without_stm ]]
    				then
    					$LOGGER "Setting preventative inbound QOS limit of $preventative_qos_limit kbits/s"
    					nvram set qos_ibw=$preventative_qos_limit
    					# We must set the inbound rate limits to 100% or less for this to work ("None" will not work
    					# because the rate will not be limited). To make sure this does work, we'll change the
    					# settings to 100% across the board and restore the old settings when we remove the
    					# temporary QOS limit.
    					nvram set qos_irates_old=`nvram get qos_irates`
    					nvram set qos_irates=100,100,100,100,100,100,100,100,100,100
    					service qos restart	
    				fi
    				if [[ $stm_period_direction = "TX" && "`nvram get qos_obw`" = $outbound_bandwidth_without_stm ]]
    				then
    					$LOGGER "Setting preventative outbound QOS limit of $preventative_qos_limit kbits/s"
    					nvram set qos_obw=$preventative_qos_limit
    					service qos restart	
    				fi		
    			fi
    		
    			if [[ $cell_color = "66FF33" ]]
    			then
    				# Cell is green => apply normal QOS if necessary
    				if [[ $stm_period_direction = "RX" && "`nvram get qos_ibw`" != $inbound_bandwidth_without_stm ]]
    				then
    					$LOGGER "Removing preventative inbound QOS limit"
    					nvram set qos_ibw=$inbound_bandwidth_without_stm
    					nvram set qos_irates=`nvram get qos_irates_old`
    					nvram unset qos_irates_old
    					service qos restart	
    				fi
    				if [[ $stm_period_direction = "TX" && "`nvram get qos_obw`" != $outbound_bandwidth_without_stm ]]
    				then
    					$LOGGER "Removing preventative outbound QOS limit"
    					nvram set qos_obw=$outbound_bandwidth_without_stm
    					service qos restart	
    				fi			
    			fi
    		fi # End of STM prevention code
    		
    		# Add data to web page
    		web_page="${web_page}<td>$total_transferred_mb MB</td><td bgcolor=\"${cell_color}\">$average_rate KB/s ($average_rate_kilobits kb/s)</td>"
    	
    		# Check whether STM limit has been exceeded
    		stm_period_bw_limit_bytes=`expr $stm_period_bw_limit_mb \* 1048576`
    		result=`echo "" | awk '{ if (total > limit) {print(1)} else {print(0)}}' total=$total_transferred_bytes limit=$stm_period_bw_limit_bytes`
    		if [[ $result = 1 && "`nvram get stm_sentence_active`" != 1 ]]
    		then
    			stm_sentence_triggered=1
    			nvram set stm_sentence_active=1
    			$LOGGER "STM Triggered: ${stm_period_name} (${stm_period_start}:00 -> ${stm_period_end}:00, ${stm_period_direction}, ${stm_period_bw_limit_mb} MB)"
    		fi
    	else
    		# This STM period is not currently active
    		web_page="${web_page}<tr bgcolor=\"EEEEEE\" style=\"color:AAAAAA\" align=\"center\">"
    		web_page="${web_page}<td align=\"left\">${stm_period_name} (${1})</td><td>${stm_period_start}:00</td><td>${stm_period_end}:00</td><td>${stm_period_bw_limit_mb} MB</td><td>N/A</td><td>N/A</td>"	
    	fi
    	web_page="${web_page}</tr>"
    done
    web_page="${web_page}</table>"
    
    # Remove RX/TX STM Prevention QOS limits if necessary. This code makes sure the temporary limits
    # are removed when they are not relevant (i.e. when we're not in an STM period and STM has not
    # been triggered)
    if [[ $preventative_mode = 1 ]]
    then
    	if [[ "`nvram get stm_sentence_active`" != 1 && "$stm_period_rx_active" != 1 && "`nvram get qos_ibw`" != $inbound_bandwidth_without_stm ]]
    	then
    		$LOGGER "Removing preventative inbound QOS limit"
    		nvram set qos_ibw=$inbound_bandwidth_without_stm
    		nvram set qos_irates=`nvram get qos_irates_old`
    		nvram unset qos_irates_old
    		service qos restart	
    	fi
    	if [[ "`nvram get stm_sentence_active`" != 1 && "$stm_period_tx_active" != 1 && "`nvram get qos_obw`" != $outbound_bandwidth_without_stm ]]
    	then
    		$LOGGER "Removing preventative outbound QOS limit"
    		nvram set qos_obw=$outbound_bandwidth_without_stm
    		service qos restart	
    	fi
    fi
    
    # If STM is active, check whether QOS is set accordingly
    if [[ "`nvram get stm_sentence_active`" = 1  && "`nvram get qos_obw`" != $outbound_bandwidth_with_stm ]]
    then
    	if [[ $stm_sentence_triggered = 1 ]]
    	then
    		# We must have triggered an STM sentence on this run.
    		# Work out the hour component of the time the STM sentence will end
    		# (for logging purposes only - we will use unix time to track the
    		# STM sentence accurately)
    		stm_end_hour=`expr $current_hours + $stm_sentence_length_hours`
    		if [[ $stm_end_hour -gt 23 ]]
    		then
    			stm_end_hour="0`expr $stm_end_hour - 24`"
    		fi
    		
    		# Set NVRAM variables to describe the STM sentence being served
    		nvram set stm_sentence_end_time="$stm_end_hour:$current_minutes"
    		nvram set stm_sentence_end_unix_time=`expr $current_unix_time + $stm_sentence_length_seconds`
    
    		# Send information to log
    		$LOGGER "STM sentence started: applying STM QOS configuration (until $stm_end_hour:$current_minutes)"
    	else
    		# An STM sentence is active and we didn't trigger it on this run,
    
    		# but the QOS settings are wrong. We must have rebooted, or someone
    		# messed with the QOS settings. So we want to set the QOS settings
    
    		# to deal with the STM, but we don't want to overwrite our NVRAM
    		# variables.
    		# Send information to log
    		$LOGGER "STM sentence resumed: applying STM QOS configuration (until `nvram get stm_sentence_end_time`)" 
    	fi
    	
    	# Set QOS NVRAM variables
    	nvram set qos_obw=$outbound_bandwidth_with_stm
    	nvram set qos_ibw=$inbound_bandwidth_with_stm
    
    	# Restart QOS
    	service qos restart
    fi
    
    # Add status to the web page
    web_page="${web_page}<h4>STM Prevention: "
    if [[ $preventative_mode = 1 ]]
    then 
    	if [[ "`nvram get qos_ibw`" = $inbound_bandwidth_without_stm ]]
    	then
    		web_page="${web_page}<b style=\"color:green\">Inactive</b></h4>"
    	else
    		web_page="${web_page}<b style=\"color:red\">Active</b></h4>"
    	fi
    else
    	web_page="${web_page}<b style=\"color:CCCCCC\">Not enabled</b></h4>"
    fi
    
    web_page="${web_page}<h4>STM Mitigation: "
    if [[ "`nvram get stm_sentence_active`" = 1 ]]
    then
    	web_page="${web_page}<b style=\"color:red\">Active - STM sentence will end at `nvram get stm_sentence_end_time`</b></h4>"
    else
    	web_page="${web_page}<b style=\"color:green\">Inactive</b></h4>"
    fi
    
    web_page="${web_page}<h4>Current QOS settings: <b style=\"color:blue\">`nvram get qos_ibw`</b> kb/s down, <b style=\"color:blue\">`nvram get qos_obw`</b> kb/s up</h4>"
    
    # Add latest log entries to web page
    web_page="${web_page}<h3>Latest Log Entries</h3>"
    log=""
    while read line
    do
    	case $line in
    		*STM-Monitor*) log="${log}<nobr><tt>${line}</tt></nobr><br>" ;;
    	esac
    done < /tmp/var/log/messages
    web_page="${web_page}$log"
    
    # Publish web page
    web_page="${web_page}</body></html>"
    echo $web_page > /var/wwwext/stm-monitor.htm
    
    I've done as much testing as I had time for but there may still be bugs. Feedback is most welcome as always.
  11. samuarl

    samuarl Networkin' Nut Member

    I have been using version 1.01 for a few days now and not found any problems other then at first I used -P instead of -p but that became apparent when I looked in the log :) . Recently we haven't been STM'd though as I've found that by showing the other house mates how to check the STM monitor page they have curbed their downloads during STM period.

    I'm finding at the moment my bandwidth to be fairly unstable sometimes as low as 365 kB/s during peak but usually between 500 - 980 kB/s (Damn you virginmedia!). This really does make my QOS almost useless so I may have trouble testing the scripts full capabilities, but I should think its beyond this scripts scope to do bandwidth tests so will take it to the tomato feature request thread.

    Thanks for updating!
  12. davemuk

    davemuk Addicted to LI Member

    Hi Tievolu,

    I have a small problem with the bandwidth totals, it seems they're not working. Here's my STM monitor page output and as you can see the values are zero. I'm guessing this might affect the STM triggering too ?

    * I'm using vanilla tomato 1.23.

    TIA,
    Dave

    Tomato STM Monitor v1.01
    Status at 11:07
    STM Description Start End Bandwidth Limit Bandwidth Used Average Rate
    Daytime Downstream (L) 10:00 15:00 2400 MB 0.00 MB 0.00 KB/s (0.00 kb/s)
    Evening Downstream (L) 16:00 21:00 1200 MB N/A N/A
    Evening Upstream (L) 15:00 20:00 700 MB N/A N/A

    STM Prevention: Not enabled
    STM Mitigation: Inactive
    Current QOS settings: 102400 kb/s down, 465 kb/s up
    Latest Log Entries


    * Thanks for taking the time to write this script :)
  13. kardzzz

    kardzzz Networkin' Nut Member

    davemuk, the script uses 'rstats' so please check if you have 'Bandwidth monitoring' enabled under Admin.
    If that fails try enabling loggin aswell.
  14. davemuk

    davemuk Addicted to LI Member

    Thanks Kardzzz I'll give that a go.
  15. Mikey242

    Mikey242 Networkin' Nut Member

    This is amazing, really good idea Tievolu. Will try this out when I get home. Thanks again for sharing your work.
  16. davemuk

    davemuk Addicted to LI Member

    That sorted my problem, thanks again Kardzzz.
  17. Mikey242

    Mikey242 Networkin' Nut Member

    Just tried it out, works great and very useful. Thanks agin for this fantastic tool Tievolu.
  18. kardzzz

    kardzzz Networkin' Nut Member

    Tievolu,
    I moved to version 1.01 this week, previously was happy with version 1. 1.01 works very well and i managed to activate the STM mitigation on purpose to test the logic. Its a much improved version as the user no longer needs to actively monitor the bandwidth used.

    Some feedback and hopefully you can add the following,,,
    I have a virgin 'L' service and try to download/use as much of the given bandwidth. For example between 10am-3pm i get 2400mb for which i may download 2 files of size 999mb in succession.
    I feel the STM prevention kicks in too soon. The STM prevention should only kick-in after a certain limit has been achieved (which may be user specified through the script), i.e. 85% (which is about 2000mb for my service). This would therefore give me a fast download for the 2 files and start STM prevention when bandwidth is almost used up.

    I look forward to any updates you make.
    thanks
  19. tievolu

    tievolu LI Guru Member

    Unfortunately I've had to change the location of this script, as Google are killing off their Google Pages service. The script can now be found here:

    http://host.blucube.co.uk/~tievolu/stm/stm-monitor.sh

    and the (limited) documentation can be found here:

    http://host.blucube.co.uk/~tievolu/stm/stm-monitor.html

    The old Google Pages site will be killed in June, so update your WAN Up scripts as soon as you can...

    On a more postitive note, version 1.02 is now available, which adds an STM prevention threshold to prevent STM prevention kicking in too early. In v1.02, STM prevention won't become active until you've transferred 80% of an STM limit. Thanks to kardzzz for suggesting this feature and testing my bungled attempts at implementing it :)

    In addition you can now customise some of the settings, including the STM prevention threshold. More details here:

    http://host.blucube.co.uk/~tievolu/stm/stm-monitor.html#CUSTOMISE

    Let me know if you have any questions.
  20. amgraham

    amgraham Networkin' Nut Member

    I've got a good feature Idea,
    If STM prevention is on, recalculate the speed you can download every time the script is run.
    EG, when I've downloaded 2.4GB (so have 600mb remaining) and have 2 hours left in the stm period, it may set 200kb/s as my downrate. but if I don't download anything for an hour I've still got the same speed of 200kb/s set but since I still have the same 600mb to use in the 1 hour remaining I should be at 400kb/s.

    I'm loving your work so far.
  21. tievolu

    tievolu LI Guru Member

    Thanks for the feedback!

    You're pretty much describing the way the script currently works, although it's slightly more conservative. Each time it runs, the script checks two things if STM prevention is enabled:

    1. Are we downloading/uploading faster than the maximum permissable rate (i.e. is the Average Rate cell red)? If so, calculate the remaining bandwidth available in this STM period and set the QOS limit so that only 90% of that can be downloaded before the STM period is over.

    2. If a temporary preventative QOS limit has been applied, check whether our upload/download rate has dropped to an acceptable level (i.e. Average Rate is less than 90% of the maximum permissible rate - cell is green). If so, remove the QOS limit.

    So if you trigger STM prevention by satisfying condition 1, but then don't download anything for a while, you should satisfy condition two at some point and your QOS limits will return to the normal non-STM settings.

    So to take your example:

    - Evening downstream (16:00 - 21:00 3000MB)
    - Download 2.4GB in three hours - average rate = 227.56KB/s
    - Maximum permissible rate = 170.67KB/s
    - => STM prevention kicks in and sets rate to 56.89KB/s
    - ** Don't download anything for an hour **
    - => Downloaded 2.4GB in four hours - average rate 170.67KB/s

    So you're actually still at the maximum permissible rate overall. The script will wait until the overall Average Rate for the entire STM period has dropped to less than 90% of the maximum permissable rate before removing the STM prevention limits. For this particular example, that means another half an hour or so would have to pass, at which point the overall Average Rate would drop to ~153KB/s and the QOS limits would be removed.

    It's not perfect, but it works. I think I have to take this conservative approach to ensure that STM prevention actually does its job - i.e. to make sure it definitely prevents STM from kicking in, every time. I could change the script to cut it much finer (e.g. restore the normal QOS settings as soon as your transfer rate has dropped below the maximum permissible rate), but as it only runs every two minutes it might not quite work and you could end up just going over the STM limit in the last few minutes and then being punished for another five hours.

    I could make it slightly less conservative. Maybe remove the STM prevention limit when you drop below 95% of the maximum permissable rate. In the example above this would mean normal QOS settings would be restored after an extra quarter of an hour rather than half an hour (so in the end it doesn't actually make a huge difference).

    I could make a test version available with slightly different settings if you'd be interested in testing it? I'd also be interested in other people's opinions on this.

    T
  22. amgraham

    amgraham Networkin' Nut Member

    Can you explain / show the STM Rate calculation for this example? (I need it to further explain what I mean)
  23. tievolu

    tievolu LI Guru Member

    I may have confused you by getting one of the calculations wrong (the crucial STM prevention calculation!). Here are the correct values:

    - Evening downstream (16:00 - 21:00 3000MB)
    - Download 2.4GB in three hours - average rate = 2400 MB / 180 mins = 227.56KB/s
    - Maximum permissible rate = 3000 MB / 2300 min = 170.67KB/s
    - => STM prevention kicks in and sets rate to 0.9 * (600 MB / 120 min) = 76.8KB/s
    - ** Don't download anything for an hour **
    - => Downloaded 2.4GB in four hours - average rate = 2400 MB / 240 mins = 170.67KB/s

    Let me know if that still doesn't make sense :)
  24. tievolu

    tievolu LI Guru Member

    I just posted version 1.03. This updates the STM limits with the new values introduced by VM on May 12, and also adds the S broadband offering (although I very much doubt anyone here is using "S"!).

    The new limits consist of a slight increase for L and XL. On the downstream side, L gets an extra 600Mb during the day and 300Mb in the evening. The evening upstream limit has also been increased by 100MB.

    XL gets an extra 1000Mb downstream during the day and 500Mb downstream in the evening. Upstream limits are unchanged.
  25. popcornmix

    popcornmix Networkin' Nut Member

    Firstly thanks for this script. I've been using it successfully for months, for managing my QoS settings, and it works great.

    Just thinking through the STM prevention mode. For example, on M+ I am allowed 750MB between 1600-2100 before throttling. If I want to download "lots", I can:

    enable STM prevention. Download 750MB between 1600-2100 (at a pedestrian 0.3Mbit), and then have full 10Mbit from 2100 until 1000. That is 13 hours of 1.25Mbytes/sec .
    Total of 750MB + 58.5GB

    or at 1600, trigger STM after 750MB (only ten minutes!). Then have 2.5Mbit for next five hours. Then at 2110 my STM finishes (is this right?) and I have full speed for only ten minutes less than previous case.
    Total of 5.6GB + 58.5GB

    So the optimal strategy is to trigger STM as soon as possible, meaning it doesn't eat into the evening's download period.
  26. kardzzz

    kardzzz Networkin' Nut Member

    tievolu, thanks for v1.03... you're doing a fantastic job!
    I have a suggestion which may prove useful and user selectable:

    Say I have 2.9gb between 10am-3pm and I quickly use up 2.5gb by 12noon. The script would enforce a limit of perhaps 450kb/s down for 3 hours which is much worst than the 2560kb/s if i were capped and i would rather take 2 additional hours sentencing.
    Therefore I would like to add setup parameters upfront and to say, if we have >2.5hours remaining then down >= 2560 (the sentence rate).
    In summary, we force a setence to get a better download rate.


    What do other users think?
  27. tievolu

    tievolu LI Guru Member

    At the moment STM Prevention kicks in once you've transferred 80% of the bandwidth limit for an STM period and your Average Rate exceeds the maximum permissable rate (i.e. if you kept on downloading at that speed you would trigger STM).

    So, for your example, on M+ in the 1600-2100 evening downstream STM period, with STM Prevention enabled with the default 80% threshold, you could download 600MB (80% of 750MB) at full speed before STM Prevention kicked in. Assuming you downloaded that amount at the full 10Mb, STM Prevention would kick in at 16:08 and limit your bandwidth to 7.9KB/s so that you could only download 90% of the remaining 150MB over the next 4 hours 52 mins. If STM Prevention was not enabled you'd be STM'd two minutes later at 16:10 and receive five hours limited to 320KB/s.

    Clearly STM Prevention is much worse in this scenario! If this is the way you plan to use your connection I wouldn't recommend using STM prevention in its current form. Just run in the default mode to automatically reconfigure your QOS settings when you get STM'd.

    I knew STM Prevention wasn't one-size-fits-all as soon as I started implementing it. Clearly it's only really worthwhile if you don't download constantly and you only rarely get STM'd. It's especially useful if you usually only get STM'd in the last few minutes of an STM period. In that scenario taking the more extreme hit of STM Prevention over a short period is better than taking a five hour hit of the less extreme STM punishment.

    This is exactly what you and kardzzz are getting at.

    I reckon I can automate the decision in the script by comparing the options like this: if STM Prevention is about to be triggered, work out when the STM limit would be exceeded if we carried on as we are going, and calculate how much data would be transferred over the following five hours from that point for both scenarios - i.e. being STM'd or limiting bandwidth to prevent STM. The script could then go with whichever option would allow more data to be transferred.

    Does that sound sensible?
  28. popcornmix

    popcornmix Networkin' Nut Member

    Yes that sounds good.

    If current STM prevention is going to limit download rate, you should do your normal throttling if we are close to the end of the STM period (i.e. 20:45).

    If it's closer to start of the interval and I'm downloading at 10Mbits then let the STM kick in, and carry downloading at maximum rate possible.

    I guess you work out how much data can be downloaded in total in each case from now until STM is lifted (~ next 5 hours), and pick the bigger one.
  29. tievolu

    tievolu LI Guru Member

    The more I look into this the better I think it will work. I think once this is implemented we can do away with the STM Prevention threshold and let the script decide what's best.

    I'll let you know when I have something good enough for you to test.
  30. amgraham

    amgraham Networkin' Nut Member

    I like STM prevention, please don't remove the feature, I don't want to be flagged as a user that exceeds stm every day.

    This is what I was thinking
    So when STM Prevention kicks in
    say we've downloaded 2.4GB and have 600MB remaining.

    1) Store the Value (600MB*0.1) (the amount you DON'T want to use)
    2) each time the script is run work do the following sum:-
    (Total Allowance - Stored Value - used bandwidth) / seconds remaining

    That above will never use more than 90% of the remaining bandwidth,
    and it gets faster each time it's run if they're not downloading at full STM restricted speed.

    - Anthony
  31. tievolu

    tievolu LI Guru Member

    I'm not going to remove STM Prevention. I'm going to modify the script so that when you are on target to trigger STM, it works out which is the best course of action (i.e. which will allow you to transfer more data in the long term) - Prevention or Mitigation. I'll probably make it a new option so that people can carry on using the script exactly the way it is now if they wish.

    In my last post above I was talking about removing the STM Prevention threshold (i.e. the setting that controls when STM Prevention can kick in - default is after 80% of the STM limit has been transferred). I don't think that setting a threshold will be necessary or desirable with the new implementation - the whole point is to let the script decide what's best by forecasting the results of each option. Artificially skewing those results by introducing an arbitrary threshold would defeat the object of the exercise.

    I did consider this type of approach, but it means that QOS will probably be restarted every time the script runs when STM Prevention has been triggered. I don't know if you noticed, but restarting QOS isn't completely free, for me at least. It seems to disrupt my connection for a short time. I for one wouldn't want that happening every two minutes.

    The way STM Prevention works at the moment means that if you don't download/upload at the full restricted rate, prevention will eventually be removed and you will be given full bandwidth again. The temporary STM Prevention QOS limits will be removed if/when your total Average Rate for the STM period drops to below 90% of the maximum permissable rate for that period.

    I implemented it that way to ensure that QOS is restarted as little as possible.
  32. tievolu

    tievolu LI Guru Member

    Ok, v1.04 is now available, with Intelligent STM Prevention/Mitigation implemented as we've been discussing. This new mode is enabled on the command line by using the -i option (instead of -p). This is how it works:

    1. If the average transfer rate indicates that STM will be triggered (i.e. the cell in the table is red), calculate how long it will be before that happens.
    2. Calculate how much data could be transferred over that time at the uncapped rate, plus the amount of data that could be transferred during the STM sentence at the capped rate.
    3. Calculate how much data could be transferred over the same total time period if we enabled STM Prevention to prevent STM from being triggered.
    4. Compare the values from (2) and (3), and proceed with the option that will allow more data to be transferred.
    The details of the algorthim's decisions are written to the system log and can be seen in the Latest Log Entries section of the status page.

    I decided to leave the Prevention threshold alone for now, so none of this is calculated until at least 80% of the STM period's bandwidth limit has been transferred.

    While working on this and testing it I also noticed that inbound STM Prevention wasn't actually working, due to the amount of extra data that arrives on the WAN side of the router over and above the QOS limit we've set. This extra data is counted by rstats (and Virgin Media), even though it is ultimately discarded as a result of the temporary QOS settings. So we need to ensure that the incoming bandwidth before routing is low enough to prevent STM.

    In previous versions, the script worked out how much bandwidth was left before STM would be triggered and set QOS such that only 90% of that could be downloaded in the remaining time. I did some testing and found that the rate actually has to be based on downloading only 50% of the remaining bandwidth. This is obviously quite a big difference, but it has to be done - there's not much point in STM Prevention if it doesn't prevent STM... (Note that outbound STM Prevention is unchanged. 90% is just fine because the router has complete control over how much data is sent out.)

    This means that the Intelligent algorithm is even more important than before, because inbound STM Prevention is now more harsh on your transfer rate.

    Anyway, give it a go and let me know how it goes.

    T
  33. popcornmix

    popcornmix Networkin' Nut Member

    Thanks for that. I've updated and things seem okay. Haven't triggered STM yet.

    I do get this in the logs:
    May 25 01:03:01 unknown cron.err crond[103]: USER root pid 10082 cmd /tmp/stm-monitor.sh M+ -i

    every 2 minutes. I think I've always got that - is it normal?
  34. tievolu

    tievolu LI Guru Member

    I didn't have cron logging enabled so I never saw this before. I'm not sure what the problem is, but this message seems to be printed when any cronjob is executed. Here's an extract from my log:
    Code:
    May 25 09:59:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 516 cmd /tmp/stm-monitor.sh XL -i
    May 25 10:00:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 678 cmd logger -p syslog.info -- -- MARK --
    May 25 10:00:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 679 cmd rcheck --cron
    May 25 10:00:01 xxxxxxxxxxxx syslog.info root: -- MARK --
    May 25 10:01:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 682 cmd /tmp/stm-monitor.sh XL -i
    May 25 10:03:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 875 cmd /tmp/stm-monitor.sh XL -i
    May 25 10:05:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1107 cmd /tmp/stm-monitor.sh XL -i
    May 25 10:07:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1410 cmd /tmp/stm-monitor.sh XL -i
    May 25 10:09:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1700 cmd /tmp/stm-monitor.sh XL -i
    May 25 10:09:01 xxxxxxxxxxxx cron.err crond[114]: USER root pid 1701 cmd logger `date`
    May 25 10:09:01 xxxxxxxxxxxx user.notice root: Mon May 25 10:09:01 BST 2009
    
    As you can see from the last couple of entries in the log, even a simple cronjob command such as "logger `date`" causes an error. I'm not sure what's going on but I don't think it's anything to do with my script.

    The good news is that these weird errors seem to have no impact at all on the commands executed by cron. I'm probably just going to disable cron logging and forget about them.

    EDIT: Just had a thought. Perhaps these messages from cron are simply notifications, to log which commands were executed at what times, and what their PIDs were. In other words, maybe they're not errors at all.
  35. amgraham

    amgraham Networkin' Nut Member

    Sorry to bug you, but prevention was working, but after a while (over a week) it's stopped:-

    STM Description Start End Bandwidth Limit Bandwidth Used Average Rate
    Daytime Downstream (XL) 10:00 15:00 6000 MB 7253.78 MB 423.97 KB/s (3391.76 kb/s)

    However, mitigation has kicked in ok.
    Current QOS bandwidth: 5120 kb/s down, 174 kb/s up

    Any ideas what it could be?

    - Anthony
  36. tievolu

    tievolu LI Guru Member

    Not sure. A couple of questions:

    1. Which version?
    2. What log messages do you see?
  37. samuarl

    samuarl Networkin' Nut Member

    Excellent work tievolu! Working as expected. I will change the 80% threshold though as today we hit that at 6pm so internet has been painfully slow for last couple of hours. Thanks!
  38. Mikey242

    Mikey242 Networkin' Nut Member

    Thanks a lot tievolu, I love this script.
  39. tievolu

    tievolu LI Guru Member

  40. snellgrove

    snellgrove Guest

    Script is excellent, been using it a while now. Thanks a lot Tievolu.

    Had a strange thing happen today though. I'm on M+ so 750MB downstream in the evening. Set iPlayer off on a 1.3 GB file. I have STM monitor in -i mode, so it works out what's best.

    What I see in the logs is something odd!

    Customisable settings:

    STM Prevention Threshold: 70%
    Normal Inbound Bandwidth: 10240 kb/s
    STM Inbound Bandwidth: 2560 kb/s
    Normal Outbound Bandwidth: 467 kb/s
    STM Outbound Bandwidth: 118 kb/s

    Jul 14 20:49:10 router user.notice STM-Monitor: STM (RX) will trigger in 00h05m51s. Potential RX over 05h05m51s with/without STM Prevention: 22118 MB / 6064 MB => STM Prevention is beneficial.
    Jul 14 20:49:10 router user.notice STM-Monitor: Setting preventative inbound QOS limit of 56 kbits/s
    Jul 14 20:51:08 router user.notice STM-Monitor: STM (RX) will trigger in 23h33m52s. Potential RX over 04h33m52s with/without STM Prevention: 19843 MB / 3665 MB => STM Prevention is beneficial.
    Jul 14 20:51:08 router user.notice STM-Monitor: Setting preventative inbound QOS limit of -338 kbits/s
    Jul 14 20:51:13 router user.notice STM-Monitor: STM Triggered: Evening Downstream (16:00 -> 21:00, RX, 750 MB)
    Jul 14 20:51:14 router user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 01:51)


    Any ideas why it thought preventative was better, and to set QOS inbound to -338 kb/s? Obviously a few moments later (5 seconds, instead of 2 minutes) it hit the limit, and that was that!
  41. tievolu

    tievolu LI Guru Member

    (This post ended up as a bit of a brain dump - I hope it all makes sense!)

    I've always known there was at least one loophole in the script, and you've found it :biggrin:

    The problem has two factors. Firstly, the rstats sampling period of two minutes is really a bit too long for our purposes. The situation the script is looking at can change massively in a two minute period, especially when you're close to the end of an STM period, and the effect is enhanced when you're on one of the lower tiers with a small bandwidth limit relative to your maximum download rate.

    Secondly, there is a delay of just over a minute between the rstats data being produced and the script acting on it. The rstats data is produced on the even minutes (19:00, 19:02, 19:04 etc.), while the STM Monitor runs on every odd minute (19:01, 19:03, 19:05 etc.). This, combined with problem 1, means that it usually takes a couple of iterations for STM Prevention to find a suitable value. Basically, it calculates a value every iteration and then undercuts it slightly to make sure it will work. If a subsequent iteration, based on new data, indicates that a lower prevention limit is needed, the script goes with the new, lower value. If the value is low enough, every iteration after that will calculate a progressively higher prevention limit as the triggering of STM is gradually averted...

    With that in mind, you can see that it's possible to trigger STM before the prevention algorithm has sorted itself out - that seems to be exactly what's happened to you.

    You are right at the end of an STM period, so there's not much time for the script to do anything if you start transferring at a high rate (which is exactly what you did!) - especially as you can transfer at maximum speed for three minutes before the script knows that's what you're doing.

    As soon as it does see what you're doing, it tries to set the preventative limit to stop you triggering STM. (Prevention is always going to be beneficial here because we're so close to the end of the STM period - ten mins at a very slow speed is going to be better than the five hour STM punishment.) However, the initial prevention limit is not low enough (due to the second effect described above), and by the next iteration of the script it's too late - the STM limit has already been exceeded.

    Unfortunately the script isn't prepared for this state of affairs - it assumes that STM Prevention is always going to work i.e. that you'll never exceed the limit when STM prevention is activated. So it goes ahead and tries to tweak the prevention limit, and calculates a negative value (because you've already blown the STM limit). This of course doesn't make any sense, but it's a very short lived screw up because the script then immediately notices that the STM limit has been exceeded and applies the STM bandwidth limits instead.

    The only thing that could be done to improve this situation is to change the cron job to run the script on the even minutes, and introduce a short delay at the start of the script (maybe 10 seconds) to ensure that rstats has done its business before we start calculating anything. This would improve things, but it would be a little messier and it wouldn't eliminate the problem completely - there is still going to be a delay between the generation of the rstats data and the running of the script, and the rstats data is still produced only every two minutes.

    So the inherent lag in the script could potentially be reduced from three minutes to two minutes ten seconds, which would clearly help, but I'm not sure it would make a huge difference. The loophole would still exist...

    The only way to prevent this from happening completely is to avoid starting very large downloads within the last few minutes of an STM period. Awkward I know, but it's the only reliable solution :)
  42. kardzzz

    kardzzz Networkin' Nut Member

    tievolu, been a while since i looked at this thread... i'm still running version 4 (intelligent) but at 99% on my large VM.
    What I didn't like was if the threshold is set at 80% and you hit that level at say 6pm, the script will determine what rate is best and maybe go 100 kb/s down 465 kb/s up (up untouched). I would prefer it if the script compares the 'calculated rate' and if its lower than the 'STM active rate', run with the 'STM active rate'. basically I'd rather have a cap at the virgin limit than have a crawling network and if i eventually trigger the STM I won't feel change

    Do you think its do-able for v 1.05? or can i do that already in the current version.

    Thanks.
  43. tievolu

    tievolu LI Guru Member

    Hi kardzzz. The problem with your suggestion is that the harsh bandwidth limits are essential if you want to prevent STM from kicking in. If you only reduce bandwidth to the level of the STM limit, STM will not be prevented. This defeats the object of "STM Prevention", and I don't think it offers any benefit over simple STM Mitigation either (your proposal would just mean you'd be limited to the STM bandwidth for a longer period of time), so it wouldn't make sense for the Intelligent algorithm to limit the bandwidth in this way rather than choosing Prevention or Mitigation.

    For what it's worth, I don't use the Prevention/Intelligent algorithms myself for the same reason :). I hate having the crawling network. I'd rather trigger STM so long as my QOS settings are aware of the cap and adapt to it. So I stick with the original implementation - STM Mitigation.

    But I can see why some people would prefer to take the temporary bandwidth hit and prevent STM from kicking in, because in the long term it means you can download/upload more data.
  44. kardzzz

    kardzzz Networkin' Nut Member

    tievolu, you make perfect sense. I think i'll stick with the original mitigation code or keep it to how I have it now.
    You did a lot of great work on this script and its made a huge difference over the last year that I now take it for granted not having crawling network (unless i trigger it intentionally by rebooting and downloading lots).

    thx
  45. robst3r

    robst3r Serious Server Member

    Hi, great script and it found it works really well. One question i have though is that when STM is in force and the router is rebooted - the preventative qos settings are removed and default qos bandwidth settings are put back. Now i know that rebooting the router resets the data transferred logs so everything shows green again, but when the default settings are applied it sets all of the inbound classes bandwidth back to none.

    I know that some people suggest that it makes no difference if these settings are set to the none value, but i have found that i get serious lag on the Playstation network if i don't allocate bandwidth specifically to my ps3 and am downloading at the same time. Basically i have my ps3 in the highest class with no bandwidth restrictions, and other internet based traffic set to high with 90% inbound allocation. This fixes the lag problem.

    So my question is, is there any way to prevent the STM script from wiping out the inbound class settings and setting them all to none when a router reboot occurs (the outbound settings remain uneffected and dont change.

    Thanks

    Rob
  46. tievolu

    tievolu LI Guru Member

    Hi Rob,

    This is a bug.

    When STM prevention kicks in, the script stores the user-specified inbound rates in a temporary nvram variable, while it temporarily sets the rate to 100 across the board to ensure that the inbound rate limiting works (this is done because if they're set to "None", as in the majority of cases, the rate won't be limited properly and STM prevention won't work).

    However, it doesn't do an nvram commit so the temporary variable is trashed if the router is power cycled while STM prevention is active (it should be preserved through a soft reboot - via the Tomato GUI - though). I've added a simple null check to the code that restores the rates from this variable, to ensure that it doesn't replace the inbound rates when the temporary variable is empty. This should prevent the behaviour you describe, and the rates you specified in the Tomato QOS GUI will be restored from nvram on reboot.

    Try with the latest version of the script (v1.05) and let me know if you still see this problem.

    Thanks for catching this, and I'm glad you're finding the script useful!
  47. robst3r

    robst3r Serious Server Member

    Hi Tievolu,

    Many thanks for this. I have just downloaded the latest version of the script, and will test it. Before i read your post i violated the STM on purpose to do some more testing. When i rebooted the router, and the STM script came back up, the 1.05 script remembered that the STM had been violated, and the sentence was still applied which is very cool! - Something it didn't do before, as it simply put the rates back to standard speed which kind of threw things off as far as QOS went until my 5 hour stint had passed.

    I wont know until later on when the script has re-applied my standard bandwidth settings if it changes my custom inbound settings, but i will post tomorrow and let you know the resuts. One other thing (not sure if its possible), but it would be really nice if the status monitor remembered the amount of bandwidth transferred within the STM period (in case of needing to reboot the router mid monitoring session), so that the bandwidth stats used would be accurate and STM still applied when needed - then this would be perfect :biggrin:

    Thanks so much for your hard work on this - it really is an awesome piece of work, and i am sure that everyone shares my gratitude.

    Regards

    Rob
  48. tievolu

    tievolu LI Guru Member

    This is actually the way it behaved before for STM mitigation. I haven't changed anything in 1.05 apart from adding the null check I talked about in my last post. However, the script only remembers the STM violation for soft reboots - i.e. reboots triggered via the Tomato web interface. If you power cycle the router the state is lost (because the nvram variables recording the state of the STM violation are not committed).

    The script can't "remember" anything about STM prevention because it relies on the live bandwidth data produced by rstats, which is reset after a reboot.

    I have considered this, and it is possible in principal, but to be completely robust the script would need to use persistent nvram variables, which would mean doing an nvram commit every time they are set/unset. This could put quite a lot of extra strain on the router's flash memory, so on balance I don't think it's a good idea.

    I could try to modify the script so that the state would at least survive a soft reboot, but it would require at least a constantly updated record of the Rx/Tx bandwidth totals, and I think it might be tricky to work out the logic to correctly decide when to use this stored data. Moreover, the stored bandwidth totals would have to be combined with the real live data from rstats in the correct way - working this logic out would be tricky too (the more I think about it the more difficult it seems).

    Unfortunately I don't have as much time to spend on this script as I did a few months ago. I have time to fix relatively simple bugs, but not to add features like this. So the bottom line is that this will probably never get implemented, at least not by me anyway! :)
  49. robst3r

    robst3r Serious Server Member

    Hmmm, interesting - i only ever perform soft reboots via the gui, and yesterday it wiped out the STM sentence but with the new script tonight it didn't. In fairness i was doing a lot of testing yesterday and a lot of soft reboots so that maybe explains why - either way it now works for me.

    This was just really a "nice to have" question - the script is great so no great hardship if it can't be easily done, and the way you have explained it - it doesn't sound so easy :biggrin:

    Also an update on the v1.05 script, instead of waiting for my sentence to expire, i manually changed the qos bandwidth settings to different values to see what happened when the script re-applied the STM. My inbound values stay to how i set them, so looks like your fix works. Awesome work, thanks so much :thumbup:
  50. MadFox

    MadFox LI Guru Member

    I can't seem to get this to work with tomato 1.27, not sure what I'm doing wrong, I can run the script manually through ssh fine, and it changes the QoS settings for that one time, but if i put it in the WAN up scripts section it does not appear to run. The web page for monitoring it is blank and the tomato log doesn't mention it loading :(

    Any ideas?

    ps - excellent work tievolu I have been looking for this for ages!
  51. markp1989

    markp1989 Serious Server Member

    hey, thanks for the script, i tried the newest script on your site, it tried to set the stm of on purpose to test your script. im running it with the L option im not using preventition or anything like that.

    It looks like your script set the qos before i had hit the limit . i hit about 80% of the stm limit, and the stm monitor page says i have hit it, but i havnt yet :S.

    thanks for any pointers you can give, mark
  52. markp1989

    markp1989 Serious Server Member

    i just had a problem, i tried to test the stm preventation ( i used the -i flag), and it failed, and i hit the limit at 22.59 , when the limits ended at 21.00 , so now im stuck with 5 hours at throttled speeds:(

    Code:
    Jun 30 01:00:45 unknown user.notice STM-Monitor: Downloaded Tomato STM Monitor v1.05
    Jun 30 20:31:10 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h23m42s. Potential RX over 05h23m42s with/without STM Prevention: 22163 MB / 7403 MB => STM Prevention is beneficial.
    Jun 30 20:31:10 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 285 kbits/s
    Jun 30 20:33:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h18m10s. Potential RX over 05h18m10s with/without STM Prevention: 21884 MB / 6988 MB => STM Prevention is beneficial.
    Jun 30 20:33:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 237 kbits/s
    Jun 30 20:35:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h15m41s. Potential RX over 05h15m41s with/without STM Prevention: 21842 MB / 6801 MB => STM Prevention is beneficial.
    Jun 30 20:35:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 222 kbits/s
    Jun 30 20:37:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h13m04s. Potential RX over 05h13m04s with/without STM Prevention: 21789 MB / 6605 MB => STM Prevention is beneficial.
    Jun 30 20:37:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 201 kbits/s
    Jun 30 20:39:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h11m14s. Potential RX over 05h11m14s with/without STM Prevention: 21797 MB / 6468 MB => STM Prevention is beneficial.
    Jun 30 20:39:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 189 kbits/s
    Jun 30 20:41:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h08m56s. Potential RX over 05h08m56s with/without STM Prevention: 21768 MB / 6295 MB => STM Prevention is beneficial.
    Jun 30 20:41:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 166 kbits/s
    Jun 30 20:43:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h07m12s. Potential RX over 05h07m12s with/without STM Prevention: 21784 MB / 6165 MB => STM Prevention is beneficial.
    Jun 30 20:43:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 150 kbits/s
    Jun 30 20:45:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h06m07s. Potential RX over 05h06m07s with/without STM Prevention: 21850 MB / 6084 MB => STM Prevention is beneficial.
    Jun 30 20:45:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 144 kbits/s
    Jun 30 20:47:08 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h04m06s. Potential RX over 05h04m06s with/without STM Prevention: 21843 MB / 5933 MB => STM Prevention is beneficial.
    Jun 30 20:47:08 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 111 kbits/s
    Jun 30 20:49:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h02m56s. Potential RX over 05h02m56s with/without STM Prevention: 21903 MB / 5845 MB => STM Prevention is beneficial.
    Jun 30 20:49:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 94 kbits/s
    Jun 30 20:51:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h01m50s. Potential RX over 05h01m50s with/without STM Prevention: 21967 MB / 5763 MB => STM Prevention is beneficial.
    Jun 30 20:51:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 71 kbits/s
    Jun 30 20:53:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h01m03s. Potential RX over 05h01m03s with/without STM Prevention: 22056 MB / 5704 MB => STM Prevention is beneficial.
    Jun 30 20:53:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 52 kbits/s
    Jun 30 20:55:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h00m21s. Potential RX over 05h00m21s with/without STM Prevention: 22152 MB / 5651 MB => STM Prevention is beneficial.
    Jun 30 20:55:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 25 kbits/s
    Jun 30 20:57:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 00h00m05s. Potential RX over 05h00m05s with/without STM Prevention: 22281 MB / 5631 MB => STM Prevention is beneficial.
    Jun 30 20:57:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of 9 kbits/s
    Jun 30 20:59:09 unknown user.notice STM-Monitor: STM (RX) will trigger in 23h59m55s. Potential RX over 04h59m55s with/without STM Prevention: 22419 MB / 5619 MB => STM Prevention is beneficial.
    Jun 30 20:59:09 unknown user.notice STM-Monitor: Setting preventative inbound QOS limit of -26 kbits/s
    Jun 30 20:59:13 unknown user.notice STM-Monitor: STM Triggered: Evening Downstream (16:00 -> 21:00, RX, 1500 MB)
    Jun 30 20:59:14 unknown user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 01:59)
    
    edit: just disabled qos and done a speed test, my internet is the same speed as it always is, so right now, the script is limmiting my connections when it doesnt need to.
  53. AsusTomato

    AsusTomato Serious Server Member

    As with last few posters I'm having a strange problem with this script also when its set to L option. During the hours when evening upstream is active downloads are counted as upstream bandwidth being used. During the hours when only downstream STM applies only about 10% of downloads are counted. This results in either the script triggering unnecessarily because it thinks I've gone over the 800MB upload limit or not triggering when I go over my download limit but the script still thinks I haven't.

    Hoping someone familiar with Linux scripting could check this script for errors.
  54. AsusTomato

    AsusTomato Serious Server Member

    Following on from the above it's the part of the script below that doesn't look right to me.

    # Read the stats file and extract the parts we're interested in (vlan1 RX and TX)
    read_next_rx_tx=0
    while read line
    do
    case $line in
    *vlan1*) read_next_rx_tx=1 ;;
    esac
    if [[ $read_next_rx_tx = 1 ]]
    then
    case $line in
    *rx:*) rx_data=$line ;;
    *tx:*) tx_data=$line ; read_next_rx_tx=0 ;;
    esac
    fi
    done < $stats_file
    rm $stats_file

    Many thanks for your help.
  55. tievolu

    tievolu LI Guru Member

    You mention that you hit the limit at 22:59, but the tomato log shows a time of 20:59. It looks like your router time was not set correctly.
  56. tievolu

    tievolu LI Guru Member

    That part of the code is a bit of a fudge - if the structure of the rstats file changes it may well break. I haven't upgraded my router for a long time - I'm still on 1.22.

    Unfortunately I simply don't have the time to work on this script any more. If you figure out what's causing your problem let me know!
  57. tievolu

    tievolu LI Guru Member

    I'm planning to modify the script to add the parameters for the upgraded upload speeds VM are planning to roll out over the next few months. The options will be as follows:

    M - current M offering
    M+ - M offering with upgraded upload (1Mb)
    L - current L offering
    L+ - L offering with upgraded upload (1Mb)
    XL - current XL offering
    XL+ - XL offering with upgraded upload (2Mb)
    XXL - XXL offering with upgraded upload (5Mb) and new STM policy

    Note that anyone currently using the M+ option will need to switch to M until their upload speed is upgraded (M+ was previously used to indicate M accounts that had been upgraded to 10Mb - I assume they have all been upgraded by now?).
  58. nutellajunkie

    nutellajunkie Serious Server Member

  59. robst3r

    robst3r Serious Server Member

    Hi Tievolu,

    I think i may have found a bug with 1.07. I have a NAS on my home network, and i am currently in the process of migrating my media files over to an external hard disk which is attached to the USB port of my router (I plan to use my RT-N16 as a media server). The STM monitor appears to be treating this traffic as WAN traffic, and the STM sentence has been incorrectly applied. Is there any way to get round this?

    Cheers

    Rob
  60. tievolu

    tievolu LI Guru Member

    The script is 100% reliant on the data provided by rstats - if rstats is counting your USB traffic as WAN traffic (i.e. on vlan1) then there's nothing the script can do to separate it out from genuine WAN traffic.

    Have a look at the bandwidth stats in the Tomato GUI, which work off exactly the same data. I suspect they will also be incorrect.
  61. rs232

    rs232 LI Guru Member

    Hello tievolu

    first of all thank you so much for the script. I've been looking for something like this for ages!
    I've just installed it on 2 routers in two different locations and I'm now testing them out. I did this literally 30 minutes ago so I guess I just need to wait to hit the peak hour to see something on the stats page, don't I? (please see image in attachment).

    I wanted to ask 3you questions if you don't mind:

    1) In my understanding virgin-media (btw I used to work for them long time ago, lol!) considers weekend and bank holidays in a different way. For weekends they refer to "midday to midnight" period but they don't actually specify what they do. The helpdesk wouldn't give away details either. Do you have better information in regards? does your script already differentiate between weekday/weekend (if)?

    2) How does your capping integrate with the built in tomato QoS? It is just a parent class for upload and download? I'm asking as I already do some class capping doing layer 7 matching.

    3) Recalling the previous post about NAS traffic... before finding your script I was thinking to write one, and my initial idea was to use a QoS class and use the class counter rather than rstat. In theory:

    totals - USB traffic = what needs to be counted.

    Not sure how difficult this would it be though.


    Thanks you so much again!
    Stef

    Attached Files:

  62. rs232

    rs232 LI Guru Member

    Actually (extending my prev post):

    1) stat page is working

    2) still open to discussion

    3) I have the same problem as I have a NAS on my lan. As I run Victek mod on asus RT-N16 (like many of us on this forum I guess) wan interface is actually vlan2! vlan 1 is counting LAN traffic. So sould you either:
    a) add an option to manually specify the vlan interface linked to the WAN
    b) even better: let the script work out this by itself. e.g. filtering out IP adresses rfc1918 should leave you with 1 IP and 1 interface linked to it. From there picking up the right vlan should be trivial.
    In the meantime I've added this line after the script is downloaded and the chmod:
    sed 's/vlan1/vlan2/g' /tmp/stm-monitor.sh > /tmp/TMPFILE && mv /tmp/TMPFILE /tmp/stm-monitor.sh

    4) Perhaps the stat page could be linked via .css

    5) How about modifying the script behaviour like this:
    a) allow the script to use local sotrage e.g. /cifs1 /cif2/ mnt/usb ....
    a) at boot time verify the script version online (first line?)
    b) if version online is higher download and store locally, if equal run the local copy.
    c) in any case always run the script from local storage.
    This should help a lot in case for whatever reason your repository goes own.

    To be fair I'm mainly concern about point 3, but I'm happy to give ideas away if it can help anyhow, if not never mind :)

    Again thank you so much for the hard work!
    Stef
  63. Toastman

    Toastman Super Moderator Staff Member Member

    You can use "nvram get wan_ifname" instead of specifying the vlan manually.

    For example, used to reach pppoe bridged modem:

    ip addr add 192.168.0.13/24 dev $(nvram get wan_ifname) brd + #Assign address to vlan port
    iptables -I POSTROUTING -t nat -o $(nvram get wan_ifname) -d 192.168.0.0/24 -j MASQUERADE
  64. rs232

    rs232 LI Guru Member

    Good stuff. I still think this should be included in the script itself but the following 2 lines between wget and chmod do the job nicely:

    wan=`nvram get wan_ifname`
    sed s/vlan1/$wan/g /tmp/stm-monitor.sh > /tmp/TMPFILE && mv /tmp/TMPFILE /tmp/stm-monitor.sh


    So my WANUP script now looks like:

    wget -O /tmp/stm-monitor.sh http://www.tievolu.co.uk/stm/stm-monitor.sh
    wan=`nvram get wan_ifname`
    sed s/vlan1/$wan/g /tmp/stm-monitor.sh > /tmp/TMPFILE && mv /tmp/TMPFILE /tmp/stm-monitor.sh
    chmod 755 /tmp/stm-monitor.sh
    cru a STM-Monitor "1-59/2 * * * * /tmp/stm-monitor.sh L -i"
    logger -t STM-Monitor "Downloaded `head /tmp/stm-monitor.sh | grep \"STM Monitor\" | sed 's/# //g'`"

    Thanks :)
  65. rs232

    rs232 LI Guru Member

    Ok, I've been testing the script for 2 days now and I'm very impressed with the result.

    There's only one case I run into that started me mumble about.That goes back to my point #2

    I happen to have an Internet TV and use BBC iplayer on a daily basis.

    Today it wan't usable at all (braking out every 10 seconds) as I left my laptop running bittorrent and eating up the quota made available.

    I've ended up adding a filter blocking out all the layer 7 matched IPP2P during critical hours, but this is of course not optimal.

    I'm not sure how many other users have this very same problem, but an option I thought about would have not just to use quota limitation but also an understanding of the traffic using it e.g. IPP2P can use max x% of what the quota available no matter what.
    This way delay sensitive data can be better used. Yes, you'll still have the same problem if quota is hit with non IPP2P traffic but I think that's rather unlikely as e.g. a BBC video is about 300 Mb/hour

    So in a nutshell e.g.:

    -- 100% cap
    all other traffic
    -- 30% cap
    IPP2P only
    -- 0%

    with an option to allow IPP2P to get to 100% of the quota e.g. 15 minute before the end of the monitored time.

    So yes, this would indeed need an interaction with bit count per QoS class.

    Option b)
    If all this sounds too complicated how about just allowing a timing cap for the manually defined P2P class during monitored time? Timed QoS settings is unfortunately not an option as per today and I guess it would be something rather trivial compared to the current script complexity.


    I'm not sure how helpful this input can be, but please consider this as my 2 cents :)


    Thanks again!
  66. AsusTomato

    AsusTomato Serious Server Member

    RS232's extra lines have fixed the problem top which was being caused by STM Monitor reading vlan1 instead of vlan2.
    However I've found another slight problem, bought some games in the Steam sale and the huge downloads revealed the following.

    Dec 31 12:55:04 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM Triggered: Daytime Downstream (10:00 -> 15:00, RX, 3000 MB)
    Dec 31 12:55:04 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 17:55)

    STM is triggered at 12:55 PM and the router applies the reduced Upload/Download limits, all good so far.
    But at 4 PM the STM Monitor starts to count download traffic even though STM has already been triggered.

    Dec 31 17:57:01 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence served: applying normal QOS configuration
    Dec 31 18:05:04 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM Triggered: Evening Downstream (16:00 -> 21:00, RX, 1500 MB)
    Dec 31 18:05:05 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence started: applying STM QOS configuration (until 23:05)

    Seven minutes after lifting STM it's reapplied due to the STM Monitor having already been counting downloads since 4PM

    Dec 31 23:07:01 ASUS-RT-N16-Tomato-Toastman user.notice STM-Monitor: STM sentence served: applying normal QOS configuration

    STM Monitor applies normal router speeds but STM is still in effect and wasn't lifted by ISP until 1:30am approx. Which means it wasn't actually triggered until about 8:30PM. During the 11:07pm to 1:30am period pings went through the roof as Steam saturated all the download bandwidth. STM Monitor should only start to count bandwidth once any previous STM sentences have been lifted.

    Sorry tievolu I only seem to pop on now and again to complain about something. :rolleyes:
    It's a very useful tool for anyone on Virgin Media Broadband and I really do appreciate all the work you've put into it.
  67. AlyxUK

    AlyxUK Serious Server Member

    I've had to disable Prevention, it just doesn't seem to work in benefit of the users experience. Sure, maybe the user DOES get better possible download if it avoids the STM -- but it would do when its set to 1kbps and all traffic grinds to a crippling halt. I've even seen it apply -14kbps before now (yes, minus).

    So from now on, I'm just having mitigation.
  68. rs232

    rs232 LI Guru Member

    I've been tweaking the script on a daily basis to get the best result.
    May I suggest to give it a go with the following setting ?:

    1) use prevention

    2) (before the cru command in the WAN UP script)
    Code:
    echo "STM_PREVENTION_THRESHOLD=10" >> /tmp/stm-monitor.cfg
    This way the real monitoring/tweaking is started after 10% of the bandwidth consumption, and avoid the "worst case scenario" where you eat up 80% of the bandwidth allowance at e.g. 3:30PM forcing the script to throttle the 20% left over until 9Pm. Actually this would just fail...

    3)
    If you really want to improve user experience try also to reduce the STM_INBOUND_BANDWIDTH_WITH_STM. I have a L connection and use 1 Mbit with STM, but I may also try lower e.g. 800Kbit

    e.g.

    (before the cru command in the WAN UP script)
    Code:
    echo "STM_INBOUND_BANDWIDTH_WITH_STM=1000" >> /tmp/stm-monitor.cfg
    With these settings you'll cap the bandwidth much earlier and have a yes slower service but fast enough to browse internet and even watch videos during STM period.

    4) use tomato access restriction to match and match (layer 7) and totally disable any IPP2P traffic between 10Am->3Pm and 4Pm->9PM

    5) As sometime P2P traffic is not properly matched by the layer 7 filtering set the P2P client(s) to behave as per point 4 (e.g. utorrent has an internal scheduler where you can specify when stop/start the downloads)

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    Beside this I have a feature request:
    As VM applies a different cap for the two periods of the day I see a value in having different settings per different time of the day.

    e.g. 10 to 3 prevention at 90% (as nobody is home in any case)
    4 to 9 prevention at 10%

    But any setting you may want really.

    HTH
    Stef
  69. tievolu

    tievolu LI Guru Member

    Thanks to everyone for the recent discussion! It's nice to see so much interest in the script.

    I've added the code to get the WAN interface name from nvram, rather than assuming it's vlan1, and I've also tried to address the problem highlighted by AsusTomato above. The script will now only count traffic from the end of the last STM sentence, if that was more recent than the beginning of the current STM period.

    A test version of the script, including these changes, is available here:

    [EDIT: The changes are now available in v1.08.]

    As for some of the other questions raised recently... the script is based on the traffic management policy that VM publish here:

    http://shop.virginmedia.com/help/traffic-management/traffic-management-faster-uploads.html

    This is the only information available to me.

    Currently the script mitigates/prevents STM by simply altering the total outbound/inbound max bandwidth values. The suggestions for tighter integration with QoS are interesting, but I'm not going to have enough time to look into that. If anyone else wants to give it a go, please feel free to modify my script any way you like.

    BTW don't apologise for raising problems - it's how we make things better :)
  70. rs232

    rs232 LI Guru Member

    Thanks tievolu!
    I've started to modify the script to see how it reacts. It's well written and the topis itself is rather complex in nature so I guess it will take me sometime to get my head around.

    In the mean time one question on the script as it is now: I've set the threshold to 5% (for a test) and after some downloading my inbound cell eventually got amber (please see picture in attachment). Now, if the box is amber why the default QoS settings are still applied and not the STM ones (I have a L connection and currently running the script in prevention mode)?

    Thanks :)
    rs232

    Attached Files:

  71. tievolu

    tievolu LI Guru Member

    It started out as quite a simple script, but it's evolved into a bit of a mess and could do with being re-factored using functions. Unfortunately I don't have the time. I've tried to comment it as much as possible to keep it vaguely maintainable - with a bit of effort you should be able to work out what it's doing :)

    The QoS max inbound/outbound settings will only be changed if the rate exceeds the maximum allowed - i.e. if the cell turns red. Here's a description of exactly what happens:

    http://www.tievolu.co.uk/stm/stm-monitor.html#PREVENTION
  72. MikeBuzz

    MikeBuzz Reformed Router Member

    do you need to enable QOS for this to work, what happens if you dont have it set??
  73. tievolu

    tievolu LI Guru Member

    Yes, I think so.

    I imagine that your bandwidth would still get counted on the stats page, so it might still be of some use, but the STM mitigation/prevention wouldn't work because they both use QoS.

    I'm not 100% sure though - it depends what the command "service qos restart" does when QoS is not already started. It might start QoS.....
  74. MikeBuzz

    MikeBuzz Reformed Router Member

    I will give it a test and see what happens :)
  75. MikeBuzz

    MikeBuzz Reformed Router Member

    Would there be a way to add a password to the page?
  76. tievolu

    tievolu LI Guru Member

    It's already protected by the router admin password, because it's served by the same http server as the other admin pages.
  77. tievolu

    tievolu LI Guru Member

  78. MikeBuzz

    MikeBuzz Reformed Router Member

    i will have to test again, as i am sure it let me in with no password, will have to check when get home
  79. tievolu

    tievolu LI Guru Member

    If you're already logged into the admin interface you won't get prompted for the password when you access the stats page (just as you won't get prompted each time you access different pages in the admin interface).

    If you explicitly log out of the admin interface (bottom of the menu on the left hand side), then try to access the stats page, you should be prompted for your admin password.
  80. MikeBuzz

    MikeBuzz Reformed Router Member

    Think it must have been Chrome playing up, I deleted all saved password but it still didn't prompt me, tested on different PC prompted for password :)

    have just installed the 1.08 script and will see how it performers
  81. mikechml

    mikechml Serious Server Member

    I think there's a problem with the mitigation logic. If I hit either limit then both download and upload are limited in the QoS settings, but virgin doesn't seem to do the same. After hitting the download limit, I can still reach my usual upload speed (after disabling the STM monitor script).
  82. tievolu

    tievolu LI Guru Member

    This definitely was not the case when I first wrote the script a couple of years ago - if you hit any STM limit your upload and download both got capped. However, it's quite possible that they've updated/improved their methods since then. It would clearly be much more sensible if they limited only the direction for which the STM limit was exceeded.

    Unfortunately I can't find any official Virgin Media documentation stating what they do in this regard, and I hardly ever trigger STM any more myself (the new 30Mb STM limits are pretty generous!), so I need some further confirmation of this before I modify the script to work this way - the changes would be fairly extensive and fundamental.

    I've posted the question on the Virgin Media support forum. Let's see what happens...
  83. robst3r

    robst3r Serious Server Member

    Tievolu,

    I'm getting an issue with v1.10 of this script. I have the following line appearing in my logs, and the STM Monitor does not start

    "Apr 16 11:41:01 RT-485B39E85566 user.notice STM-Monitor: Usage: stm-monitor.sh [S|M|M+|L|L+|XL|XL+|XL++|XXL] [-p|-i]"

    The line in my WAN-UP script reads

    "cru a STM-Monitor "1-59/2 * * * * /jffs/stm-monitor.sh [XL] [-i]"

    Any ideas?

    Thanks

    Rob
  84. tievolu

    tievolu LI Guru Member

    Hi Rob,

    You don't need the square brackets - i.e. the command in your WAN-UP script should read like this:

    Code:
    cru a STM-Monitor "1-59/2 * * * * /jffs/stm-monitor.sh XL -i"
    T
  85. rs232

    rs232 LI Guru Member

  86. nutellajunkie

    nutellajunkie Serious Server Member

    no reboot required

    If like me, you like to keep your up-times going.. (its a Linux user thing). Then I realised you can have this script up and running without rebooting.

    In your routers status web-page.
    Goto Status>Overview then click the >WAN>status>release button

    telnet into your router and issue these commands:-

    Code:
    wget -O /jffs/stm-monitor.sh http://www.tievolu.co.uk/stm/stm-monitor.sh
    chmod 755 /jffs/stm-monitor.sh
    Go back to Status>Overview then click >WAN>Status>Renew
    You may need to click it twice and make sure your status is connected, with an IP address.

    back in your telnet session, issue:-

    Code:
    cru a stm-monitor.sh "1-59/2 * * * * /jffs/stm-monitor.sh [broadband type] [-p|-i]"
    Where [broadband type] = S, M, M+, L, L+, XL, XL+, XL++, or XXL, [-p] activates STM Prevention, and [-i] activates Intelligent STM Prevention/Mitigation.
    You will notice I uncapitalised the "STM" part of the file name, Linux is sensitive about this.

    if you now goto your monitor page http://192.168.1.1/ext/stm-monitor.htm you should see it up and running.

    However, if I any of this is incorrect, I must apologise. It "appears" to work for me though it is 4:4odd am. But im sure someone more intelligent will be able to tweak this a little, and I shall amend and give due credit.
  87. rs232

    rs232 LI Guru Member

    Would this write the stats on jffs every time the script is called? If that's the case I seem to remember that using jffs for frequents writes is not suggested.
    However it's a good finding as tomatousb users can change jffs for usb.
    Cifs is another elegant possibility...

    Thanks for sharing!
  88. rs232

    rs232 LI Guru Member

    Hi Tievolu, I'm sure you're already aware that VM is doubling the connections speeds sometime later on this year: So

    10/1Mb --> 20/2
    30/3Mb --> 60/6
    50/5Mb --> 100/10

    Not sure what will happen to the existing 100 though.

    It should be easy to modify the script accordingly.

    rs232 :)
  89. tievolu

    tievolu LI Guru Member

    To answer the question from last year (!) - the script, stats and HTML are all held in the temporary RAM disk, so there are no issues with wearing out your flash.

    Regarding the recent STM policy changes and upgrades, I'm in the process of updating the script. If someone can tell me the exact download/upload speeds for each tier (from the modem settings) it would be helpful, because they don't always match the headline rate. For instance, from looking at my own modem config pages I know that the 30Mb tier is actually 33330000 b/s download, and 3333000 b/s download, (about 31.8Mb down, 3.18 Mb down). If no-one has this information, I'll just use the headline rates for the tiers that I don't know about.

    The state of VM's offerings is a bit of a mess at the moment, so I'll also be changing the option names to match the tiers described on VM's STM page to try and keep it as simple as possible:

    http://help.virginmedia.com/system/...E=Cable&CMD=VIEW_ARTICLE&ARTICLE_ID=2781#tier

    So the list of options will be:
    • XXL100
    • XL60
    • XXL
    • XL30
    • L30
    • ML20
    • XL
    • L
    • M
    • S5
    This means that existing installations will break when the new script is downloaded. Sorry about that, but keeping the old options plus the new ones would be really messy. I'm also going to base everything on the speeds after the upload speed upgrades. If I tried to handle all the speeds, before and after the upgrade, the script would have 17 options, which is a bit silly. Remember that you can always set your own speeds if necessary:

    http://www.tievolu.co.uk/stm/stm-monitor.html#CUSTOMISE

    EDIT: Updated script available here for testing:

    [EDIT: Changes are available in v1.11]

    Let me know if you spot any problems, otherwise I'll make this v1.11 next week:
  90. rs232

    rs232 LI Guru Member

    I have been upgraded this morning from 10 to 20Mb, however my upload speed will stay at 1Mb until October due to a hub limitation .
    VM explained me that not all the customer can get any speed they want. e.g. with my postcode the hub serves 5/10/20/60/100 only, so 30, 50 and 120 are not an option.
    My contract is a "L".
    I guess in October when the hub will be upgraded my L will go directly from 20/1 to 30/3 as this is the basic contract they're currently selling.
    VM confirmed that on my 20/1 contract my data capacity is7GByte for the morning and 3.5GByte for the afternoon/evening.
    http://www.virginmedia.com/images/STM_20Mb_below_800pxA.jpg

    Considering the number of options available during this upgrade period perhaps it's a good idea to modify the script and allow the user to specify custom speed/capacity as an alternative to contract type.

    Thanks!
    rs232
  91. rs232

    rs232 LI Guru Member

    I think the setting for my "L upgrade" contract should be:​

    # Settings specific to LU (20Mb down, 1Mb up)
    if [[ $1 = "LU" ]]
    then
    # STM limits (values in MB)
    stm_period_bw_limit_mb_1=7000
    stm_period_bw_limit_mb_2=3500
    stm_period_bw_limit_mb_3=3000
    # QOS settings (values in Kbits/s)
    inbound_bandwidth_without_stm=20240
    inbound_bandwidth_with_stm=5120
    outbound_bandwidth_without_stm=932 # 1024 * 0.91
    outbound_bandwidth_with_stm=233 # 932 * 0.25​

    P.S. according to you test script all the "inbound_bandwidth_with_stm" is set to 50% of the inbound_bandwidth_without_stm. Shouldn't this be 25% instead as the level of speed reduction is 75% ?​

    Regards​
    rs232​
  92. rs232

    rs232 LI Guru Member

    Sorry I got the 50% level of reduction now... there are two different tables: one for <20Mb and one for >30Mb.
  93. tievolu

    tievolu LI Guru Member

    Ok, I fixed the reduction percentage for ML20 and below. I think everything else is ok.

    With regard to specifying options on the command line, there is already a way to customise the config so I'll think I'll leave it at that. The alternative is to force everyone to specify six command line options defining all the characteristics of their connection. I've made the STM period bandwidth limits customisable in the same was as the other options, as people without the upgraded upload speeds may have to tweak them.

    Let me know if you spot any other problems. If not, I'll make the current test script live as v1.11 next week.
  94. tievolu

    tievolu LI Guru Member

    rs232 likes this.
  95. rs232

    rs232 LI Guru Member

    About the script version 1.11: So far so good.
    Many thanks for the update!

    I have to admit that this script was a vital piece of software on my tomato when my afternoon cap was set to 1.5Gb, nowadays I get 3.5G for the same period of time and chances are that this will increase again in the future.I can now watch 3 iplayer videos in the afternoon without breaching the cap.
    BTW I've just noticed that on ML20 it's impossible to brake the upload cap as there's not enough bandwidth to make this happen.
    I *hope* this script will be needed less and less (If VM wants) in the future ;-)

    Great work Tievolu!
  96. tievolu

    tievolu LI Guru Member

    Agreed - I think this script will slowly become obsolete, which is a good thing.

    It's only really useful now for households with multiple users, where QOS is essential (whether STM'd or not) to make the link usable for web browsing while someone else is hammering the upstream with torrent traffic, for instance. However, even that scenario is probably not as big a problem as it used to be now that VM apply their own traffic shaping to P2P/Usenet traffic.

    My life is moving on as well. Since I created script I've had two kids, so I now have hardly any time at all to spend on things like this. To be honest, I quite like it that way :) Anyway, the net of this is that this may well be the last update from me. We'll see.
  97. AsusTomato

    AsusTomato Serious Server Member

    The STM rules have changed according to this post FAQ on the Virgin Media Forum about the Traffic Management Changes it says,

    If I go over the downstream threshold while downloading will my upstream be traffic managed?

    No you will only get traffic managed on the traffic that exceeds a threshold, upstream or downstream.

    Does version 1.11 now only apply STM to download or upload alone depend on which has gone over the limit?
    I'm unable to test it myself since I'm still using 1.10 until my area gets the speed upgrades at the end of this year.
  98. rs232

    rs232 LI Guru Member

    This is a news to me, I haven't tried recently but I'm pretty sure that the limit was applied to both upstream and downstream as soon as either cap was overtaken. Worth testing! They may have change something...
  99. tievolu

    tievolu LI Guru Member

    No, v1.11 does not take this into account - this is definitely a recent change. I asked VM about the behaviour last year and was told that in general both upstream and downstream would be limited, regardless of which triggered STM.

    This will require some pretty fundamental changes to the script. As I mentioned before, I don't have much time to devote to this anymore so it might take a while, if I am able to do it at all.
  100. tievolu

    tievolu LI Guru Member

    I don't think this is as complicated as I first thought. I've had an initial stab at implementing the necessary changes, but I don't have time to test them out - I'm relying on you lot to do that :)

    The test version of the script is here:

    [EDIT: Changes are available in v1.12]

    Please give it a go and let me know if you have any problems.

Share This Page