HowTo: BSD/Win/Linux Throughput Optimization

Posted on Posted in Linux, Windows


Original link:

[HOWTO] BSD/Win/Linux Throughput Optimization


This post is copied from old FreeNAS forum , thanks to n9nu for their work:

I have been asked many times how can bandwidth be increased when transferring very large files across my internal network (LAN). Some of the information here within can also be used when transferring files via the Internet. All of these tweaks are hardware dependent of course.

For most part, the newest kernels (2.6.x) do a rather decent job of controlling the RX and TX window sizes among other things. There are, however, several options that can be tweaked/added that will allow for a substantial increase in throughput when transferring very large files across an internal network (LAN). I will focus on FTP, Samba/CIFS, and NFS transfers between Linux, Windows Vista SP2+, Windows 7 and FreeBSD. This also assumes Linux Kernel 2.6.x is being used and FreeBSD Kernel 2.6.x as well.

My incoming connection is via fiber and I allocate about 45Mbit of that for my own personal use (I own and run a large WISP outside Chicago) and after any media converters and other routing equipment, I have my CAT 6 routed directly into my ClearOS Linux-based router/gateway/server/firewall. From there to several systems behind it (LAN1/LAN2, etc).

The users that will find the biggest gain (specifically in LAN transfers) will be those who running with a quality Gigabit NIC, quality CAT 5e or greater cable, and those who are routing traffic through a quality managed or unmanaged Gigabit switch (not hub). I use a 16 port Hewlett Packard switch along with Intel Pro 1000GT PCI NIC’s. I also find many of the newer (last 3 years+) nVidia controlled on-board chipset NIC’s work very well as well.

In all the testing that I have done, I have maintained the default MTU value of 1500. Despite popular myths and misinterpretations, increasing the MTU past 1500 does NOT yield an increase in throughput when transfering files to and from the INTERNET. Increasing the MTU can indeed increase overall throughput when transfering very large files across two PC’s on the LAN via a dedicated (2nd NIC per PC) static route. This is where performance takes off. All this of course is based on ‘matched’ hardware, settings and file transfer protocols. For most of us, setting up a dedicated static route is not needed, therefore, I concentrate on setups that most users have.

There are a few things you can do to verify your hardware is up to par and that each PC and it’s network configuration are set for optimum performance.

Note: If your Internet connection is directly connected to a PC acting as a Gateway/Firewall/Router, you can use the settings for the LAN side NIC (the interface that you assign to your private LAN – e.g. 10.x.x.x. etc.). The interface that is connected to the Internet (WAN) should always be set to auto-negotiate. This will prevent any mismatches that pop up if the ISP changes their hardware.

1. Ensure that each NIC in each PC is optimized for the highest transfer bandwidth. You can do this by checking each NIC for it’s Duplex and Speed settings. Providing a Gigabit switch is used as well as a Gigabit NIC, the optimum settings would be the following:

NIC Duplex: Full Duplex
Speed: Manually set to 1000 or 1GB (if your switch/NIC supports it) – Otherwise, set to auto-negotiation.

Please note that auto-negotiation under certain conditions can actually hinder performance when using older routers as well as hubs. The best way to find out if your can set it manually to 1000, is to RTFM for the switch and NIC and then manually setting it and try it!After applying the setting, restart the network service and go at it. If working, you should notice nothing different. If not, you will obviously know as you won’t be able to ping anything but the NIC on the maching your using at that time. If so, change it to auto-negotiate.

2. Modification of your Linux and/or BSD sysctl.conf file my direct means (e.g. a terminal)

The following is a copy of my own Linux /etc/sysctl.conf file. I currently am using kernel 2.6.37 via Mandriva 2011 (Cooker) 64 Bit system. The modifications below, your ISP speed limit depending, a substational boost or may yield little to no change whatsoever.

For most part, the kernel tree 2.6 or > has been optimized to allow for on-demand receive window (RWIN) adjustments which expand as necessary to allow for a larger or smaller size. It is the same principal that the Windows kernel uses, which is called ‘auto-tuning’. For most users, it is optimally configured, however, for many users, the TCP/IP stack can be modified (as below) to manually set parameters as well as the addition of a few options that are not present in the default kernel.

Linux Kernel 2.6.37 – /etc/sysctl.conf – modifications

net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1

FreeNAS (BSD 7.x based) Kernel 2.6.x – /etc/sysctl.conf – modifications

FreeBAS Version 7.0 introduced auto-tuning into ther kernel much as Linux did and will, for most users, be just fine. Others can try modifying their sysctl.conf file to reflect the following changes and/or additions shown below.

# set to at least 16MB
# set autotuning maximum to at least 16MB too
# enable send/recv autotuning
# increase autotuning step size
# turn off inflight limiting
# set this on test/measurement hosts

With BSD 8.2, you can change/enable the congestion control method with the following line:

BSD uses ‘inflight limiting’ which provides for maximum effeicenty with good old dial-up modem connections, however, it can hinder performance for those with fat pipes. In the options listed above, I have changed the vaule for the inflight limiting to ‘0’ or disabled. You can switch between the two (0 and 1) and experiment as well.

The last option shown above (net.inet.tcp.hostcache.expire=1) is used for web servers and just ‘remembers’ previous settings for the last hour. Disabling this (a 0) will allow for the caching and when trying to test network throughput, can be a problem, so I suggest you enable this when experimenting (a value of 1).

3. Modification of specific settings within FreeNAS itself.

As in the FreeBSD section above, the settings were made to the /etc/sysctl.conf file and besides modifying the file via directly (e.g. terminal), the same settings can be modified via the FreeNAS Advanced page—>sysctl.conf options page.

In addition, there are a number of things you can add to the ‘SMB/CIFS’ section within FreeNAS to enhance performance substantially in some cases. You need to navigate to the SMB/CIFS section and go to the very bottom where it has a section to add additional commands. You can enter the following two lines to increase your performance:

max xmit = 65535