A few people have emailed me asking me to integrate the perl code snippet into I wrote to strip illegal headers when sending email via Amazon SES into something actually usable. I’ve done so! I haven’t really tested this beyond sending some test emails, but here it is. Use this at your own risk, I make no warranty, blah blah blah.
Continue reading “Amazon SES: “illegal headers” with ses-send-email.pl (followup)”
Amazon SES – “Illegal Header” errors
Update 8/2/2011: please see the followup post for a possible workaround.
A few people have inquired about the “Illegal header” error when attempting to relay email through SES. “Oncle Tom” pointed to a thread on Amazon’s forums about a similar problem which led to a list of headers Amazon will accept. The list is below; if you need to add headers outside this list, you can do so using “X-Headers.”
Some people have posted sample code in the thread with modifications to ses-send-email.pl to replace “illegal” headers read from STDIN with an equivalent X-Header. I wrote something generic that takes the list of legal headers and replaces anything not on that list with its X-Header equivalent. Disclaimer: I haven’t tested it, but it seems like this type of solution would be more adaptable since you don’t have to keep fixing it every time a new application attempts to send email with a new header. I haven’t worked this into ses-send-email.pl (since I’m not having any problems, I don’t want to touch it 🙂 ) but I fed in an email header and the output looked correct.
https://gist.github.com/1068310.js
Input email:
MIME-Version: 1.0 Received: by 10.142.136.15 with HTTP; Mon, 16 May 2011 09:32:44 -0700 (PDT) Date: Mon, 16 May 2011 12:32:44 -0400 Delivered-To: evandhoffman@gmail.com Message-ID: Subject: Hi friend. From: "Evan D. Hoffman" To: "Evan D. Hoffman (Personal)" Content-Type: text/plain; charset=ISO-8859-1 Email Test.
Output email:
[evan@EvanMBP ~]$ perl replace-headers.pl test-email.txt MIME-Version: 1.0 Received: by 10.142.136.15 with HTTP; Mon, 16 May 2011 09:32:44 -0700 (PDT) Date: Mon, 16 May 2011 12:32:44 -0400 X-Delivered-To: evandhoffman@gmail.com X-Message-ID: Subject: Hi friend. From: "Evan D. Hoffman" To: "Evan D. Hoffman (Personal)" Content-Type: text/plain; charset=ISO-8859-1 Email Test.
Hope this helps someone.
Here’s the legal header list, from http://docs.amazonwebservices.com/ses/2010-12-01/DeveloperGuide/index.html?AppendixHeaders.html
-
Accept-Language
-
Bcc
-
Cc
-
Comments
-
Content-Type
-
Content-Transfer-Encoding
-
Content-ID
-
Content-Description
-
Content-Disposition
-
Content-Language
-
Date
-
DKIM-Signature
-
DomainKey-Signature
-
From
-
In-Reply-To
-
Keywords
-
List-Archive
-
List-Help
-
List-Id
-
List-Owner
-
List-Post
-
List-Subscribe
-
List-Unsubscribe
-
Message-Id
-
MIME-Version
-
Received
-
References
-
Reply-To
-
Return-Path
-
Sender
-
Subject
-
Thread-Index
-
Thread-Topic
-
To
-
User-Agent
Integrating Amazon Simple Email Service with postfix for SMTP smarthost relaying.
So, we’ve outgrown the 500 outbound messages/day limit imposed by Google Apps’s Standard tier. A wise friend suggested SendGrid, but I figured it was worth looking into what options Amazon provides. I found SES and am in the process of setting it up. Hopefully I can set it up as a drop-in replacement, obviating the need for code changes to use it. SES is attractive for us because:
Free Tier
If you are an Amazon EC2 user, you can get started with Amazon SES for free. You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk. Many applications are able to operate entirely within this free tier limit.Note: Data transfer fees still apply. For new AWS customers eligible for the AWS free usage tier, you receive 15 GB of data transfer in and 15 GB of data transfer out aggregated across all AWS services, which should cover your Amazon SES data transfer costs. In addition, all AWS customers receive 1GB of free data transfer per month.
Free to try? Sounds good.
After signing up, the first thing I did was download the Perl scripts. Create a credentials file with your AWS access key ID and Secret Key (credentials can be found here when logged in). The credentials file (aws-credentials) should look like this:
AWSAccessKeyId=022QF06E7MXBSH9DHM02 AWSSecretKey=kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct
Make sure to chmod 0600 aws-credentials. To ensure it’s working, run:
$ ./ses-get-stats.pl -k aws-credentials -s
If it doesn’t return anything it should be working correctly.
Next, you need to add at least one verified email address:
$ ./ses-verify-email-address.pl -k aws-credentials --verbose -v support@example.com
Amazon will send a verification message to support@example.com with a link you need to click to verify the address. Once you click, it’s verified. It’s important to note that initially your account will only be able to send email to verified addresses. According to this thread, you need to submit a production access request to send to unverified To: addresses. I did this and got my “approval” email about 30 minutes later.
To send a test email:
$ ./ses-send-email.pl --verbose -k aws-credentials -s "Test from SES" -f support@example.com evan@example.com This is a test message from SES.
(Press ctrl-D to send.)
The next step is integrating the script with sendmail/postfix. The first thing I did was move my scripts to /opt/ (out of /root/) and attempt to run them with absolute pathnames (rather than ./ses-send-email.pl) and I got perl @INC errors:
[root@web2 ~]$ mv amazon-email/ /opt/ [root@web2 ~]$ /opt/ses-get-stats.pl -k aws-credentials -s -bash: /opt/ses-get-stats.pl: No such file or directory [root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s Can't locate SES.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /opt/amazon-email/ses-get-stats.pl line 23. BEGIN failed--compilation aborted at /opt/amazon-email/ses-get-stats.pl line 23.
The problem is that SES.pm isn’t in perl’s include path. To solve this, I tried adding the directory to the PERL5LIB environment var:
[root@web2 amazon-email]$ PERL5LIB=/opt/amazon-email/ [root@web2 amazon-email]$ echo $PERL5LIB /opt/amazon-email/ [root@web2 amazon-email]$ cd [root@web2 ~]$ export PERL5LIB [root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s Cannot open credentials file . at /opt/amazon-email//SES.pm line 54. [root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -s Timestamp DeliveryAttempts Rejects Bounces Complaints 2011-04-27T20:27:00Z 1 0 0 0 [root@web2 ~]$
This worked for setting all users’ PERL5LIB … but didn’t allow postfix to send the message. After a couple more attempts at doing this “the right way,” I just ended up dropping a symlink to SES.pm in /usr/lib/perl5/site_perl and the @INC error went away.
After following Amazon’s instructions for editing main.cf and master.cf, I still was unable to send mail through Postfix, even though I could send directly through the perl scripts. I kept getting this error:
Apr 28 11:26:32 web2 postfix/pipe[27226]: A2AD33C9A6: to=, relay=aws-email, delay=0.35, delays=0.01/0/0/0.34, dsn=5.3.0, status=bounced (Command died with status 1: "/opt/amazon-email/ses-send-email.pl". Command output: Missing final '@domain' )
Google led me to this blog post which led me to this other blog post which illuminated the problem: apparently the Postfix pipe macro ${sender} uses the user@hostname of the mail sender. Since the hostname of an EC2 machine is usually something crazy like dom11-22-33-44.internal, this is not likely a validated sending email address. So the solution proposed by Ben Simon was to create a regex to map user@internal to user@realdomain.com and have postfix map everything. This didn’t work for me or the bashbang.com guys, who changed it to map from user@internal to validuser@realdomain.com. I found that you can eliminate the need for the mapping entirely by changing the master.cf entry to this:
flags=R user=mailuser argv=/opt/amazon-email/ses-send-email.pl -r -k /opt/amazon-email/aws-credentials -e https://email.us-east-1.amazonaws.com -f support@example.com ${recipient}
The only difference between the above line and Amazon’s suggestion is that this replaces “-f ${sender}” with “support@example.com” which is a validated email address.
After this I was able to relay email successfully through SES. Whew!
Update 5/26/2011: We’ve been relaying through SES without issues for a few weeks now. I recently ran ses-get-stats.pl to see how many messages we’re actually sending and it’s a lot lower than expected. I’m still glad we moved to SES though, since it has no hard cap like Google Apps does:
$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -q SentLast24Hours Max24HourSend MaxSendRate 317 10000 5
Evan Hoffman's Blog 