Integrating Amazon Simple Email Service with postfix for SMTP smarthost relaying.

So, we’ve outgrown the 500 outbound messages/day limit imposed by Google Apps’s Standard tier. A wise friend suggested SendGrid, but I figured it was worth looking into what options Amazon provides. I found SES and am in the process of setting it up. Hopefully I can set it up as a drop-in replacement, obviating the need for code changes to use it. SES is attractive for us because:

Free Tier
If you are an Amazon EC2 user, you can get started with Amazon SES for free. You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk. Many applications are able to operate entirely within this free tier limit.

Note: Data transfer fees still apply. For new AWS customers eligible for the AWS free usage tier, you receive 15 GB of data transfer in and 15 GB of data transfer out aggregated across all AWS services, which should cover your Amazon SES data transfer costs. In addition, all AWS customers receive 1GB of free data transfer per month.

Free to try? Sounds good.

After signing up, the first thing I did was download the Perl scripts. Create a credentials file with your AWS access key ID and Secret Key (credentials can be found here when logged in). The credentials file (aws-credentials) should look like this:

AWSAccessKeyId=022QF06E7MXBSH9DHM02
AWSSecretKey=kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct

Make sure to chmod 0600 aws-credentials. To ensure it’s working, run:

$ ./ses-get-stats.pl -k aws-credentials -s

If it doesn’t return anything it should be working correctly.

Next, you need to add at least one verified email address:

$ ./ses-verify-email-address.pl -k aws-credentials --verbose -v support@example.com

Amazon will send a verification message to support@example.com with a link you need to click to verify the address. Once you click, it’s verified. It’s important to note that initially your account will only be able to send email to verified addresses. According to this thread, you need to submit a production access request to send to unverified To: addresses. I did this and got my “approval” email about 30 minutes later.

To send a test email:

$ ./ses-send-email.pl --verbose -k aws-credentials -s "Test from SES" -f support@example.com evan@example.com
This is a test message from SES.

(Press ctrl-D to send.)

The next step is integrating the script with sendmail/postfix. The first thing I did was move my scripts to /opt/ (out of /root/) and attempt to run them with absolute pathnames (rather than ./ses-send-email.pl) and I got perl @INC errors:

[root@web2 ~]$ mv amazon-email/ /opt/
[root@web2 ~]$ /opt/ses-get-stats.pl -k aws-credentials -s
-bash: /opt/ses-get-stats.pl: No such file or directory
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s
Can't locate SES.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /opt/amazon-email/ses-get-stats.pl line 23.
BEGIN failed--compilation aborted at /opt/amazon-email/ses-get-stats.pl line 23.

The problem is that SES.pm isn’t in perl’s include path. To solve this, I tried adding the directory to the PERL5LIB environment var:

[root@web2 amazon-email]$ PERL5LIB=/opt/amazon-email/
[root@web2 amazon-email]$ echo $PERL5LIB
/opt/amazon-email/
[root@web2 amazon-email]$ cd
[root@web2 ~]$ export PERL5LIB
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s
Cannot open credentials file . at /opt/amazon-email//SES.pm line 54.
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -s
Timestamp               DeliveryAttempts        Rejects Bounces Complaints
2011-04-27T20:27:00Z    1                       0       0       0
[root@web2 ~]$

This worked for setting all users’ PERL5LIB … but didn’t allow postfix to send the message. After a couple more attempts at doing this “the right way,” I just ended up dropping a symlink to SES.pm in /usr/lib/perl5/site_perl and the @INC error went away.

After following Amazon’s instructions for editing main.cf and master.cf, I still was unable to send mail through Postfix, even though I could send directly through the perl scripts. I kept getting this error:

Apr 28 11:26:32 web2 postfix/pipe[27226]: A2AD33C9A6: to=, relay=aws-email, delay=0.35, delays=0.01/0/0/0.34, dsn=5.3.0, status=bounced (Command died with status 1: "/opt/amazon-email/ses-send-email.pl". Command output: Missing final '@domain' )

Google led me to this blog post which led me to this other blog post which illuminated the problem: apparently the Postfix pipe macro ${sender} uses the user@hostname of the mail sender. Since the hostname of an EC2 machine is usually something crazy like dom11-22-33-44.internal, this is not likely a validated sending email address. So the solution proposed by Ben Simon was to create a regex to map user@internal to user@realdomain.com and have postfix map everything. This didn’t work for me or the bashbang.com guys, who changed it to map from user@internal to validuser@realdomain.com. I found that you can eliminate the need for the mapping entirely by changing the master.cf entry to this:

  flags=R user=mailuser argv=/opt/amazon-email/ses-send-email.pl -r -k /opt/amazon-email/aws-credentials -e https://email.us-east-1.amazonaws.com -f support@example.com ${recipient}

The only difference between the above line and Amazon’s suggestion is that this replaces “-f ${sender}” with “support@example.com” which is a validated email address.

After this I was able to relay email successfully through SES. Whew!

Update 5/26/2011: We’ve been relaying through SES without issues for a few weeks now. I recently ran ses-get-stats.pl to see how many messages we’re actually sending and it’s a lot lower than expected. I’m still glad we moved to SES though, since it has no hard cap like Google Apps does:

$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -q
SentLast24Hours Max24HourSend   MaxSendRate
317             10000           5

JavaScript: The Good Parts

I just finished reading JavaScript: The Good Parts, one of the best programming books I’ve read. The ending is fantastic:

We see a lot of feature-driven product design in which the cost of features is not properly accounted. Features can have a negative value to consumers because they make the products more difficult to understand and use. We are finding that people like products that just work. It turns out that designs that just work are much harder to produce than designs that assemble long lists of features.

Features have a specification cost, a design cost, and a development cost. There is a testing cost and a reliability cost. The more features there are, the more likely one will develop problems or will interact badly with another. In software systems, there is a storage cost, which was becoming negligible, but in mobile applications is becoming significant again. There are ascending performance costs because Moore’s Law doesn’t apply to batteries.

Features have a documentation cost. Every feature adds pages to the manual, increasing training costs. Features that offer value to a minority of users impose a cost on all users. So, in designing products and programming languages, we want to get the core features—the good parts—right because that is where we create most of the value.

We all find the good parts in the products that we use. We value simplicity, and when simplicity isn’t offered to us, we make it ourselves. My microwave oven has tons of features, but the only ones I use are cook and the clock. And setting the clock is a struggle. We cope with the complexity of feature-driven design by finding and sticking with the good parts.

It would be nice if products and programming languages were designed to have only good parts.

Continue reading JavaScript: The Good Parts

Going back to FiOS

I’m not sure why these guys operate this way – they’re more than happy to lose me as a customer and then throw huge discounts at me to get me back. If they’d just give me a good price I’d love not to have to go through this rigmarole. But after being with Cablevision for 2 months I checked Verizon’s pricing and it beat my current deal with Cablevision.

FiOS digital voice with number ported for free; 25/25 Mbps internet; HMDVR free “forever” plus a second HD STB, Showtime, Movie Channel and Flix. Since I already had the battery thing installed last time I had FiOS they gave me a fair discount. Basically the whole package for $87/month + tax, price locked for 2 years, no contract. Not as great of a deal as I’d had with FiOS originally, but it’s pretty good, and FiOS’s service is definitely better than Cablevision’s. I’ve heard Cablevision was rolling out their “DVR plus” service with all programs recorded “in the cloud” rather than on the actual box, but it’s been two months and I haven’t heard of it coming to Long Island. So basically 2 years later Cablevision’s service is exactly the same while Verizon has iPhone apps to control the DVR and use the phone as a remote, plus DVR that’s much faster and just generally better service.

On a side note, I noticed tonight I was having problems trying to stream Netflix to my Wii. I tried loading netflix.com on my laptop and that also didn’t work, it said “couldn’t find server movies.netflix.com.” I tested this via dig on my linux box and sure enough, movies.netflix.com isn’t resolving against the default Cablevision nameserver (167.206.3.206) – getting a SERVFAIL:

[evan@lunix ~]$ dig movies.netflix.com

; <> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_5.3 <> movies.netflix.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17569
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;movies.netflix.com.            IN      A

;; ANSWER SECTION:
movies.netflix.com.     232     IN      CNAME   merchweb-frontend-1502974957.us-east-1.elb.amazonaws.com.

;; Query time: 2129 msec
;; SERVER: 167.206.3.206#53(167.206.3.206)
;; WHEN: Sun Apr 24 01:23:58 2011
;; MSG SIZE  rcvd: 103

I tried the same query against Google’s nameserver (8.8.8.8) and it resolves correctly:

[evan@lunix ~]$ dig movies.netflix.com @8.8.8.8

; <> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_5.3 <> movies.netflix.com @8.8.8.8
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43718
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;movies.netflix.com.            IN      A

;; ANSWER SECTION:
movies.netflix.com.     300     IN      CNAME   merchweb-frontend-1502974957.us-east-1.elb.amazonaws.com.
merchweb-frontend-1502974957.us-east-1.elb.amazonaws.com. 39 IN A 174.129.220.6

;; Query time: 34 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Apr 24 01:37:26 2011
;; MSG SIZE  rcvd: 119

I set my router to resolve against 8.8.8.8 rather than whatever Cablevision provides and now it works. I’m not sure if this is related to the big EC2 disaster of the past few days but it looks more like Cablevision’s fault than Amazon’s or Netflix’s.

Chaos theory and Google’s crawler

I’ve been moderately perplexed by the recent spike in traffic on basically unrelated keywords. Apparently this site is currently the #5 result for “fedora 15 beta download” despite my having never written about Fedora 15. In an attempt to funnel people to a useful page I created the previous post with links to the FC 15 ISOs. I feel bad if people come here looking for an answer that’s not to be found.

In looking into this issue I searched Google for the keywords and saw this:

evanhoffman.com is blockable
evanhoffman.com is blockable

There’s a “Block all evanhoffman.com results” link under my site, but there’s none under any of the other sites. What the hell? Does my site somehow qualify as a spammer or content farm? Why do I get this dubious distinction? Ugh.

Chaos theory and Google's crawler

I’ve been moderately perplexed by the recent spike in traffic on basically unrelated keywords. Apparently this site is currently the #5 result for “fedora 15 beta download” despite my having never written about Fedora 15. In an attempt to funnel people to a useful page I created the previous post with links to the FC 15 ISOs. I feel bad if people come here looking for an answer that’s not to be found.

In looking into this issue I searched Google for the keywords and saw this:

evanhoffman.com is blockable
evanhoffman.com is blockable

There’s a “Block all evanhoffman.com results” link under my site, but there’s none under any of the other sites. What the hell? Does my site somehow qualify as a spammer or content farm? Why do I get this dubious distinction? Ugh.