Recently we noticed weird behavior downloading files from certain sites. The transfer would start out fast (around 10 MB/s), then after a couple of seconds it would plummet to around 9 KB/s. It didn’t happen for every file or every site: downloads from S3 buckets were still particularly fast. But some files that I remember being particularly fast were now showing this weird fast/slow/fast/slow behavior, for example the Sun JDK and ISOs from rit.edu that used to saturate our pipe were now getting all cRAzY.
After some poking around I decided to test HTTP versus FTP to see if it could be an application/protocol-level issue. The easiest way to do this was to find a file available via both FTP and HTTP and download it via both protocols. This is where mirrors.rit.edu came in handy. I used cURL to download it and noticed that via HTTP it was much slower than over FTP:
[evan@boba 16:07:03 ~]$ curl -O ftp://mirrors.rit.edu/pub/centos/6/isos/x86_64/CentOS-6.2-x86_64-netinstall.iso % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 227M 100 227M 0 0 9.8M 0 0:00:22 0:00:22 --:--:-- 7816k [evan@boba 16:07:33 ~]$ rm CentOS-6.2-x86_64-netinstall.iso [evan@boba 16:07:39 ~]$ curl -O http://mirrors.rit.edu/centos/6/isos/x86_64/CentOS-6.2-x86_64-netinstall.iso % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 227M 100 227M 0 0 5686k 0 0:00:40 0:00:40 --:--:-- 6269k
22 seconds via FTP at 9.8MB/s average, 40 seconds over HTTP at 5.6 MB/s average (which was one of the better HTTP runs).
This was affecting all machines on our network, and had nothing to do with the per-machine iptables rules (verified by flushing all rules). The only thing I could think of that might affect all machines, but only HTTP and not FTP would be something like packet inspection. Well, turns out that http packet inspection is on by default on the ASA. So I disabled it as described here:
Zeus(config)# conf t Zeus(config)# policy-map global_policy Zeus(config-pmap)# class inspection_default Zeus(config-pmap-c)# no inspect http Zeus(config-pmap-c)# write mem Building configuration...
Since then HTTP transfers have been consistently fast.