Low throughput across VLANs

Discussion in 'Tomato Firmware' started by Forb.Jok, Oct 28, 2013.

  1. Forb.Jok

    Forb.Jok Reformed Router Member

    I've been running Tomato (previously Toastman, currently Shibby) on a Netgear WNR3500L v2 for several years, mainly as a home router, but I also have a Synology DiskStation NAS and a server running Ubuntu behind it.

    The LAN setup I've had for a while is as follows:
    br0 (main LAN): with DHCP enabled
    br1 (server network): with DHCP disabled

    I have it set up so that the server and NAS are each connected directly to their own ports on the router, which are set to belong to the VLAN used by br1.
    In my LAN access settings, I have it set up so that my main LAN can access the server network, but not the other way around.

    The problem is that I recently noticed when transferring a large amount of data between my workstation and the NAS, that the throughput is nowhere near what I would expect from a gigabit connection. I quickly noticed that when running a transfer (copying to/from a CIFS share on the NAS from my workstation), the CPU usage on the router would skyrocket to near 100%, causing internet traffic to almost grind to a halt, and sometimes even time out.

    After installing iperf on the NAS and the server, and running some tests "iperf -s -i 1 -f m" on the server and NAS, and "iperf -c -t 15 -i 1 -f m" on my workstation, I found that the bandwidth measured around 100Mbit/s, which is pretty awful considering it's supposed to be a gigabit connection.
    UDP tests resulted in ~153Mbit/s.

    This was running a Toastman build. Since I recently saw Shibby's firmware recommended, I decided to switch over to that to see if it would improve matters. So I did a full 30/30/30 reset both before and after flashing the firmware, and set it up in the same way as before.

    Throughput actually did improve quite a bit - TCP is now ~155Mbit/s, and UDP ~250Mbit/s. However, that's still pretty bad for a gigabit connection, so I did another experiment. I changed the server and NAS IPs to IPs in my main LAN ( and instead of 172.17.1.x equivalents) and switched the ports they were connected to back to the main VLAN (br0).

    To my surprise, I then get a whopping 750Mbit/s (TCP) and over ~800Mbit/s (UDP), which is much nearer the expected 1Gbit/s. While I could maybe see the additional routing between the two subnets adding a little bit of overhead, this difference is just enormous.

    My question is basically, what's the reason for this minor change making such a huge difference in throughput, and are there any settings that can be changed to improve the throughput between different VLANs and subnets?
    Last edited: Oct 28, 2013
    darkknight93 likes this.
  2. Toastman

    Toastman Super Moderator Staff Member Member

    Presumably, you changed things so that the packets were routed via the switch in the router, which doesn't involve the router's CPU.
    Forb.Jok and koitsu like this.
  3. jerrm

    jerrm Network Guru Member

    Toastman is right. Realize when using VLANs you are routing traffic and all those packets are passing through the Linux IP stack(and 155mbps is actually very good - for tomato).

    Many threads here talk about WAN<->LAN routing performance, and the same limits apply to VLAN<->VLAN. When the ports are on the same VLAN, linux never touches wired connections, it is all handled internally by the switch chip.
    Last edited: Oct 29, 2013
    Forb.Jok and koitsu like this.
  4. Forb.Jok

    Forb.Jok Reformed Router Member

    Thanks for the replies.
    That explains it I guess.

    I suppose the solution is simply to not use VLANs then.
  5. darkknight93

    darkknight93 Networkin' Nut Member

    is there any way to improve CPU based "Rendering" of IP packets? Does Hardware Acceleration for Asus RT 66U do the trick e.g.?
    And what will be the Speeds to be expected on dualcore ARM Broadcom CPUs? :) any guesses?
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice