The m3.medium is terrible

I’ve been doing some testing of various instance types in our staging environment, originally just to see if Amazon’s t2.* line of instances is usable in a real-world scenario. In the end, I found that not only are the t2.mediums viable for what I want them to do, but they’re far better suited than the m3.medium, which I wouldn’t use for anything that you ever expect to reach any load.

Here are the conditions for my test:

  • Rails application (unicorn) fronted by nginx.
  • The number of unicorn processes is controlled by chef, currently set to (CPU count * 2), so a 2 CPU instance has 4 unicorn workers.
  • All instances are running Ubuntu 14.04 LTS (AMI ami-864d84ee for HVM, ami-018c9568 for paravirtual) with kernel 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64.
  • The test used loader.io to simulate 65 concurrent clients hitting the API (adding products to cart) as fast as possible for 600 seconds (10 minutes).
  • The instances were all behind an Elastic Load Balancer, which routes traffic based on its own algorithm (supposedly the instances with the lowest CPU always gets the next request).

The below charts summarize the findings.

average nginx $request_time
average nginx $request_time

This chart shows each server’s performance as reported by nginx. The values are the average time to service each request and the standard deviation. While I expected the m3.large to outperform the m3.medium, I didn’t expect the difference to be so dramatic. The performance of the t2.medium is the real surprise, however.

#	_sourcehost	_avg	_stddev
1	m3.large	6.30324	3.84421
2	m3.medium	15.88136	9.29829
3	t2.medium	4.80078	2.71403

These charts show the CPU activity for each instance during the test (data as per CopperEgg).

m3.large
m3.large
t2.medium
t2.medium
m3.medium
m3.medium

The m3.medium has a huge amount of CPU steal, which I’m guessing accounts for its horrible performance. Anecdotally, in my own experience m3.medium far more prone to CPU steal than other instance types. Moving from m3.medium to c3.large (essentially the same instance with 2 cpus) eliminates the CPU steal issue. However, since the t2.medium performs as well as the c3.large or m3.large and costs half of the c3.large (or nearly 1/3 of the m3.large) I’m going to try running most of my backend fleet on t2.medium.

I haven’t mentioned the credits system the t2.* instances use for burstable performance, and that’s because my tests didn’t make much of a dent in the credit balance for these instances. The load test was 100x what I expect to see in normal traffic patterns, so the t2.medium with burstable performance seems like an ideal candidate. I might add a couple c3.large to the mix as a backstop in case the credits were depleted, but I don’t think that’s a major risk – especially not in our staging environment.

Edit: I didn’t include the numbers, but the performance seemed to be the consistent whether on hvm or paravirtual instances.

“You have to install development tools first.” – OSX Mavericks, ruby, chef, and nokogiri

I was trying to get knife ec2 working on my Mac, but even though my system Ruby was at 2.0.0, the embedded Ruby that chef/knife use (in /opt/chef/embedded/bin) was 1.9.1. Installing knife-ec2 should just be a matter of typing “gem install knife-ec2” but due to some weird issues with nokogiri, I burned about 4 hours trying to make it work. I tried everything I could find – installing iconv, libxml2, and libxslt via brew and telling “gem install” to use the custom libs in /usr/local/Cellar was the most common suggestion on StackOverflow – but nothing worked. What ended up fixing it for me was reinstalling chef. 😐

[evan@Evan ~] $ sudo /opt/chef/embedded/bin/gem install nokogiri
Building native extensions.  This could take a while...
Building nokogiri using packaged libraries.
ERROR:  Error installing nokogiri:
	ERROR: Failed to build gem native extension.

        /opt/chef/embedded/bin/ruby extconf.rb
Building nokogiri using packaged libraries.
checking for iconv.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
	--with-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/opt/chef/embedded/bin/ruby
	--help
	--clean
	--use-system-libraries
	--enable-static
	--disable-static
	--with-zlib-dir
	--without-zlib-dir
	--with-zlib-include
	--without-zlib-include=${zlib-dir}/include
	--with-zlib-lib
	--without-zlib-lib=${zlib-dir}/lib
	--enable-cross-build
	--disable-cross-build
/opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:381:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:506:in `try_cpp'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:931:in `block in have_header'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:790:in `block in checking_for'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:284:in `block (2 levels) in postpone'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:254:in `open'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:284:in `block in postpone'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:254:in `open'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:280:in `postpone'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:789:in `checking_for'
	from /opt/chef/embedded/lib/ruby/1.9.1/mkmf.rb:930:in `have_header'
	from extconf.rb:103:in `have_iconv?'
	from extconf.rb:148:in `block (2 levels) in iconv_prefix'
	from extconf.rb:90:in `preserving_globals'
	from extconf.rb:143:in `block in iconv_prefix'
	from extconf.rb:116:in `block in each_iconv_idir'
	from extconf.rb:113:in `each'
	from extconf.rb:113:in `each_iconv_idir'
	from extconf.rb:137:in `iconv_prefix'
	from extconf.rb:428:in `block in '
	from extconf.rb:161:in `block in process_recipe'
	from extconf.rb:154:in `tap'
	from extconf.rb:154:in `process_recipe'
	from extconf.rb:423:in `'


Gem files will remain installed in /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1 for inspection.
Results logged to /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1/ext/nokogiri/gem_make.out
[evan@Evan ~] $

Well, this is apparently an indication that you don’t have the command-line dev tools installed on your computer. However, in Mavericks, according to Apple:

If Xcode is installed on your machine, then there is no need to install them. Xcode comes bundled with all your command-line tools. OS X 10.9 includes shims or wrapper executables. These shims, installed in /usr/bin, can map any tool included in /usr/bin to the corresponding one inside Xcode. xcrun is one of such shims, which allows you to find or run any tool inside Xcode from the command line. Use it to invoke any tool within Xcode from the command line.

wat?

I spent several hours trawling through StackExchange, Googling for every combination of nokogiri, mavericks, chef, xcode. Here are some of my searches from today:

How did I end up fixing it? Two things:

  1. In ~/.bashrc, add export PATH=/opt/chef/embedded/bin:$PATH
  2. Reinstall chef: curl -L https://www.getchef.com/chef/install.sh | sudo bash

After reinstalling chef (which installed an embedded Ruby 1.9.3 – my old version was 1.9.1), this command ran successfully:

$ sudo gem install -V --no-rdoc --no-ri nokogiri

Full output below:
Continue reading ““You have to install development tools first.” – OSX Mavericks, ruby, chef, and nokogiri”

Wasted time with Exchange 2010, SquirrelMail, and IMAP-SSL

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Some of the errors SquirrelMail was reporting were “Error connecting to IMAP server xxxx Server error: (0)” and “Error connecting to IMAP server: tls://xxxx:993. 0: ”

Nothing would actually work on port 993. Telnet to 993 got this:

$ telnet 10.0.20.18 993
Trying 10.0.20.18...
Connected to 10.0.20.18.
Escape character is '^]'.
* BYE Connection is closed. 14
Connection closed by foreign host.

After too much poking, I decided to go down to a lower level and do a simple openssl certificate retrieval and see what came back:


$ openssl s_client -connect 10.0.20.18:993
CONNECTED(00000003)
140281653434184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 113 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

That didn’t look right, so I ran it against the same server on port 443 and got back a real certificate. Same for port 995 (pop3s):

$ openssl s_client -connect 10.0.20.18:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = info@valicert.com
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1

(snip)

So there’s just something wrong with SSL on port 993. To make a long story short, I had to use the Enable-ExchangeCertificate to apply the SSL certificate to port 993. First, run “Get-ExchangeCertificate” to list the available certificates and retrieve the Thumbprint.

[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  .P....     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  I..W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

Copy & paste the thumbprint for whichever cert you want to use into Enable-ExchangeCertificate:

[PS] C:\Windows\system32>Enable-ExchangeCertificate -ThumbPrint xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -Services IIS,P
OP,IMAP -DoNotRequireSSL
[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  ......     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  IP.W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

After running that, imaps on port 993 worked perfectly. I can connect to it with both SquirrelMail and Thunderbird.

The SquirrelMail config looks like this:

IMAP Settings
--------------
4.  IMAP Server            : webmail.example.com
5.  IMAP Port              : 993
6.  Authentication type    : login
7.  Secure IMAP (TLS)      : true
8.  Server software        : exchange
9.  Delimiter              : detect

Edit Feb 15, 2011: I just renewed the SSL cert and ran into a problem with a Ruby script that was suddenly unable to check a mailbox over IMAPS. The error received was:

/usr/lib/ruby/1.8/net/imap.rb:898:in `connect': unknown protocol (OpenSSL::SSL::SSLError)
        from /usr/lib/ruby/1.8/net/imap.rb:898:in `initialize'

After a few minutes, I remembered this blog post and ran Enable-ExchangeCertificate and it worked again. Glad I wrote it down.

CONNECTED(00000003) 26831:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

Ruby/Sinatra part 2

Following up on my previous post, I decided to go the easier route and just install an older version of Ruby that didn’t have the problem with Sinatra, since I wanted a setup I could replicate easily, and editing the server.rb each time I installed wasn’t what I was going for. I downloaded 1.9.2-rc2 from http://ftp.ruby-lang.org//pub/ruby/1.9/, compiled it and it works:

[evan@evanfc12 ~]$ ruby --version
ruby 1.9.2dev (2010-07-11 revision 28618) [x86_64-linux]
[evan@evanfc12 ~]$ ruby hi.rb ^C
[evan@evanfc12 ~]$ cat hi.rb

require 'sinatra'

get '/' do
	"Hello World!"
end
[evan@evanfc12 ~]$ ruby hi.rb
== Sinatra/1.0 has taken the stage on 4567 for development with backup from WEBrick
[2010-10-08 15:32:23] INFO  WEBrick 1.3.1
[2010-10-08 15:32:23] INFO  ruby 1.9.2 (2010-07-11) [x86_64-linux]
[2010-10-08 15:32:23] INFO  WEBrick::HTTPServer#start: pid=21758 port=4567

Speed bumps like this are so much fun!

Trying to teach myself Ruby (again)

Now that I have a real web project to work on I figured it’s a good time to try and teach myself Ruby again. Sinatra seems to be the new hotness so that’s what I’m trying, but so far things aren’t going quite as I expected. Everything appears to be installed, but when I run the “Hello, World!” script, the webserver doesn’t start as it’s apparently supposed to do:

[evan@ehoffman 16:59:34 ~]$ ruby --version
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]
[evan@ehoffman 16:59:39 ~]$ irb
irb(main):001:0> require 'sinatra'
=> true
irb(main):002:0>
You have new mail in /var/spool/mail/evan
[evan@ehoffman 17:00:20 ~]$ cat hi.rb
#require 'rubygems'
require 'sinatra'

get '/' do
	"Hello World"
end
[evan@ehoffman 17:00:24 ~]$ ruby hi.rb
[evan@ehoffman 17:00:33 ~]$

I’ve tried this on two different systems with the same result. I’ve seen this same sample code in 20 different places, including sinatrarb.com, but for some reason it’s not working for me. Rather than firing up a webserver on port 4567, running “ruby hi.rb” simply exits immediately.

Edit: According to a post on comp.lang.ruby entitled 1.9.2-rc2 -> 1.9.2-p0 breaks Sinatra, server.rb needs to be edited:

Sinatra’s behaviour has appeared to change in the 1.9 compatible
versions.
I need to add the following line in my server.rb:
set :run, true

I’m debating whether it’s worth doing this (I have 8 different server.rbs in my /usr dir) or just reinstalling an older version of ruby that doesn’t have this issue.

Edit 2: Resolved by installing Ruby 1.9.2-rc2.