Tag: email

Posted on

Amazon SES – “Illegal Header” errors

Update 8/2/2011: please see the followup post for a possible workaround.

A few people have inquired about the “Illegal header” error when attempting to relay email through SES. “Oncle Tom” pointed to a thread on Amazon’s forums about a similar problem which led to a list of headers Amazon will accept. The list is below; if you need to add headers outside this list, you can do so using “X-Headers.”

Some people have posted sample code in the thread with modifications to ses-send-email.pl to replace “illegal” headers read from STDIN with an equivalent X-Header. I wrote something generic that takes the list of legal headers and replaces anything not on that list with its X-Header equivalent. Disclaimer: I haven’t tested it, but it seems like this type of solution would be more adaptable since you don’t have to keep fixing it every time a new application attempts to send email with a new header. I haven’t worked this into ses-send-email.pl (since I’m not having any problems, I don’t want to touch it 🙂 ) but I fed in an email header and the output looked correct.

https://gist.github.com/1068310.js

Input email:

MIME-Version: 1.0
Received: by 10.142.136.15 with HTTP; Mon, 16 May 2011 09:32:44 -0700 (PDT)
Date: Mon, 16 May 2011 12:32:44 -0400
Delivered-To: evandhoffman@gmail.com
Message-ID: 
Subject: Hi friend.
From: "Evan D. Hoffman" 
To: "Evan D. Hoffman (Personal)" 
Content-Type: text/plain; charset=ISO-8859-1

Email Test.

Output email:

[evan@EvanMBP ~]$ perl replace-headers.pl test-email.txt 
MIME-Version: 1.0
Received: by 10.142.136.15 with HTTP; Mon, 16 May 2011 09:32:44 -0700 (PDT)
Date: Mon, 16 May 2011 12:32:44 -0400
X-Delivered-To: evandhoffman@gmail.com
X-Message-ID: 
Subject: Hi friend.
From: "Evan D. Hoffman" 
To: "Evan D. Hoffman (Personal)" 
Content-Type: text/plain; charset=ISO-8859-1

Email Test.

Hope this helps someone.

Here’s the legal header list, from http://docs.amazonwebservices.com/ses/2010-12-01/DeveloperGuide/index.html?AppendixHeaders.html

  • Accept-Language

  • Bcc

  • Cc

  • Comments

  • Content-Type

  • Content-Transfer-Encoding

  • Content-ID

  • Content-Description

  • Content-Disposition

  • Content-Language

  • Date

  • DKIM-Signature

  • DomainKey-Signature

  • From

  • In-Reply-To

  • Keywords

  • List-Archive

  • List-Help

  • List-Id

  • List-Owner

  • List-Post

  • List-Subscribe

  • List-Unsubscribe

  • Message-Id

  • MIME-Version

  • Received

  • References

  • Reply-To

  • Return-Path

  • Sender

  • Subject

  • Thread-Index

  • Thread-Topic

  • To

  • User-Agent

Posted on

Integrating Amazon Simple Email Service with postfix for SMTP smarthost relaying.

So, we’ve outgrown the 500 outbound messages/day limit imposed by Google Apps’s Standard tier. A wise friend suggested SendGrid, but I figured it was worth looking into what options Amazon provides. I found SES and am in the process of setting it up. Hopefully I can set it up as a drop-in replacement, obviating the need for code changes to use it. SES is attractive for us because:

Free Tier
If you are an Amazon EC2 user, you can get started with Amazon SES for free. You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk. Many applications are able to operate entirely within this free tier limit.

Note: Data transfer fees still apply. For new AWS customers eligible for the AWS free usage tier, you receive 15 GB of data transfer in and 15 GB of data transfer out aggregated across all AWS services, which should cover your Amazon SES data transfer costs. In addition, all AWS customers receive 1GB of free data transfer per month.

Free to try? Sounds good.

After signing up, the first thing I did was download the Perl scripts. Create a credentials file with your AWS access key ID and Secret Key (credentials can be found here when logged in). The credentials file (aws-credentials) should look like this:

AWSAccessKeyId=022QF06E7MXBSH9DHM02
AWSSecretKey=kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct

Make sure to chmod 0600 aws-credentials. To ensure it’s working, run:

$ ./ses-get-stats.pl -k aws-credentials -s

If it doesn’t return anything it should be working correctly.

Next, you need to add at least one verified email address:

$ ./ses-verify-email-address.pl -k aws-credentials --verbose -v support@example.com

Amazon will send a verification message to support@example.com with a link you need to click to verify the address. Once you click, it’s verified. It’s important to note that initially your account will only be able to send email to verified addresses. According to this thread, you need to submit a production access request to send to unverified To: addresses. I did this and got my “approval” email about 30 minutes later.

To send a test email:

$ ./ses-send-email.pl --verbose -k aws-credentials -s "Test from SES" -f support@example.com evan@example.com
This is a test message from SES.

(Press ctrl-D to send.)

The next step is integrating the script with sendmail/postfix. The first thing I did was move my scripts to /opt/ (out of /root/) and attempt to run them with absolute pathnames (rather than ./ses-send-email.pl) and I got perl @INC errors:

[root@web2 ~]$ mv amazon-email/ /opt/
[root@web2 ~]$ /opt/ses-get-stats.pl -k aws-credentials -s
-bash: /opt/ses-get-stats.pl: No such file or directory
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s
Can't locate SES.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /opt/amazon-email/ses-get-stats.pl line 23.
BEGIN failed--compilation aborted at /opt/amazon-email/ses-get-stats.pl line 23.

The problem is that SES.pm isn’t in perl’s include path. To solve this, I tried adding the directory to the PERL5LIB environment var:

[root@web2 amazon-email]$ PERL5LIB=/opt/amazon-email/
[root@web2 amazon-email]$ echo $PERL5LIB
/opt/amazon-email/
[root@web2 amazon-email]$ cd
[root@web2 ~]$ export PERL5LIB
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s
Cannot open credentials file . at /opt/amazon-email//SES.pm line 54.
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -s
Timestamp               DeliveryAttempts        Rejects Bounces Complaints
2011-04-27T20:27:00Z    1                       0       0       0
[root@web2 ~]$

This worked for setting all users’ PERL5LIB … but didn’t allow postfix to send the message. After a couple more attempts at doing this “the right way,” I just ended up dropping a symlink to SES.pm in /usr/lib/perl5/site_perl and the @INC error went away.

After following Amazon’s instructions for editing main.cf and master.cf, I still was unable to send mail through Postfix, even though I could send directly through the perl scripts. I kept getting this error:

Apr 28 11:26:32 web2 postfix/pipe[27226]: A2AD33C9A6: to=, relay=aws-email, delay=0.35, delays=0.01/0/0/0.34, dsn=5.3.0, status=bounced (Command died with status 1: "/opt/amazon-email/ses-send-email.pl". Command output: Missing final '@domain' )

Google led me to this blog post which led me to this other blog post which illuminated the problem: apparently the Postfix pipe macro ${sender} uses the user@hostname of the mail sender. Since the hostname of an EC2 machine is usually something crazy like dom11-22-33-44.internal, this is not likely a validated sending email address. So the solution proposed by Ben Simon was to create a regex to map user@internal to user@realdomain.com and have postfix map everything. This didn’t work for me or the bashbang.com guys, who changed it to map from user@internal to validuser@realdomain.com. I found that you can eliminate the need for the mapping entirely by changing the master.cf entry to this:

  flags=R user=mailuser argv=/opt/amazon-email/ses-send-email.pl -r -k /opt/amazon-email/aws-credentials -e https://email.us-east-1.amazonaws.com -f support@example.com ${recipient}

The only difference between the above line and Amazon’s suggestion is that this replaces “-f ${sender}” with “support@example.com” which is a validated email address.

After this I was able to relay email successfully through SES. Whew!

Update 5/26/2011: We’ve been relaying through SES without issues for a few weeks now. I recently ran ses-get-stats.pl to see how many messages we’re actually sending and it’s a lot lower than expected. I’m still glad we moved to SES though, since it has no hard cap like Google Apps does:

$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -q
SentLast24Hours Max24HourSend   MaxSendRate
317             10000           5
Posted on

Autodiscover mysteriously stopped working (Exchange 2010)

I had Autodiscover working for months but recently it just stopped. I’m not sure why, but it may be related to removing the last Exchange 2003 servers from service recently. Maybe some setting got wiped from AD when I uninstalled Exchange 2003 (as per the procedure Microsoft gives). Basically what was happening was that the email address field was being autopopulated by the user’s UPN rather than their email address. Since we have a single label domain, the UPN isn’t a valid email address, and autodiscovery fails.

Anyway, I ran Get-AutodiscoverVirtualDirectory and it looks like the autodiscover URL isn’t set. Pretty sure I set this at some point.

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl

InternalUrl :
ExternalUrl :

[PS] C:\Windows\system32>

I just piped this to Set-AutodiscoverVirtualDirectory to correct the problem:

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | Set-AutodiscoverVirtualDirectory -ExternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml' -InternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml'
[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl


InternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml
ExternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml


[PS] C:\Windows\system32>

After resetting the InternalURL and ExternalURL, autodiscover works again (we have SRV records that tell Outlook to look at webmail.example.com for the Autodiscover service).

Hooray!

Posted on

Relaying through Google Apps using Sendmail to bypass EC2 spam blockage

Update 3 May 2011: I’ve subsequently modified our EC2 systems to relay SMTP mail through Amazon’s SES which doesn’t have the 500 messages per day limit that Google Apps does.

A few months ago I moved a site into EC2. I didn’t want to move the existing IMAP server (ugh) so I moved the email to Google Apps. There are only about 10 mailboxes so we went with “Standard” edition (free). Once we completed the move to EC2 we discovered that emails from our webserver were bouncing due to our EC2 IP address being listed in a spam RBL. This sucked, so I looked into relaying the mail from the EC2 webserver through our Google Apps account. Fortunately this turned out to be pretty easy.

This wiki page on scalix.com has a procedure for setting up SMTP relaying in Ubuntu with TLS & auth. I’m not running Ubuntu so the paths were different but it was basically the same procedure:

  • Create the file /etc/mail/client-info with these contents: AuthInfo:smtp.gmail.com "U:bounces@example.com" "I:bounces@example.com" "P:superpassword", where “example.com” is your Google Apps domain, “bounces” is a valid account, and the password is the account’s password. Mail relayed with these credentials will show “bounces@example.com” in the From: field of the message.
  • In /etc/mail, run makemap hash client-info < client-info
  • Edit /etc/mail/sendmail.mc, adding or uncommenting these lines: 
    define(`SMART_HOST', `smtp.gmail.com')dnl
define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
FEATURE(`authinfo', `hash /etc/mail/client-info')dnl
  • Recompile sendmail.cf: m4 sendmail.mc > sendmail.cf . I got this error: “/etc/mail/sendmail.mc:10: m4: Cannot open /usr/share/sendmail-cf/m4/cf.m4” when running the command, but I resolved it by doing yum install sendmail-cf
  • Restart sendmail.

Once this was done I sent myself a test message from the command line and received it; I checked the SMTP headers and sure enough it went through Google’s mail server. One nice side effect is that all the mail sent by the webserver appears in the “Sent” folder for the Google Apps username provided in the client-info file. Hopefully this will resolve the spam issues, since the mail is now coming from Google’s IP block.

Posted on

Wasted time with Exchange 2010, SquirrelMail, and IMAP-SSL

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Some of the errors SquirrelMail was reporting were “Error connecting to IMAP server xxxx Server error: (0)” and “Error connecting to IMAP server: tls://xxxx:993. 0: ”

Nothing would actually work on port 993. Telnet to 993 got this:

$ telnet 10.0.20.18 993
Trying 10.0.20.18...
Connected to 10.0.20.18.
Escape character is '^]'.
* BYE Connection is closed. 14
Connection closed by foreign host.

After too much poking, I decided to go down to a lower level and do a simple openssl certificate retrieval and see what came back:


$ openssl s_client -connect 10.0.20.18:993
CONNECTED(00000003)
140281653434184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 113 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

That didn’t look right, so I ran it against the same server on port 443 and got back a real certificate. Same for port 995 (pop3s):

$ openssl s_client -connect 10.0.20.18:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = info@valicert.com
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1

(snip)

So there’s just something wrong with SSL on port 993. To make a long story short, I had to use the Enable-ExchangeCertificate to apply the SSL certificate to port 993. First, run “Get-ExchangeCertificate” to list the available certificates and retrieve the Thumbprint.

[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  .P....     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  I..W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

Copy & paste the thumbprint for whichever cert you want to use into Enable-ExchangeCertificate:

[PS] C:\Windows\system32>Enable-ExchangeCertificate -ThumbPrint xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -Services IIS,P
OP,IMAP -DoNotRequireSSL
[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  ......     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  IP.W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

After running that, imaps on port 993 worked perfectly. I can connect to it with both SquirrelMail and Thunderbird.

The SquirrelMail config looks like this:

IMAP Settings
--------------
4.  IMAP Server            : webmail.example.com
5.  IMAP Port              : 993
6.  Authentication type    : login
7.  Secure IMAP (TLS)      : true
8.  Server software        : exchange
9.  Delimiter              : detect

Edit Feb 15, 2011: I just renewed the SSL cert and ran into a problem with a Ruby script that was suddenly unable to check a mailbox over IMAPS. The error received was:

/usr/lib/ruby/1.8/net/imap.rb:898:in `connect': unknown protocol (OpenSSL::SSL::SSLError)
        from /usr/lib/ruby/1.8/net/imap.rb:898:in `initialize'

After a few minutes, I remembered this blog post and ran Enable-ExchangeCertificate and it worked again. Glad I wrote it down.

CONNECTED(00000003) 26831:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

Posted on

The Barracuda Spam Firewall VMware Appliance (Vx) finally exists!

When I started at my current company, spam was handled with a separate server running SpamAssassin and a few other services. This sort of got the job done but required babysitting. I wasn’t part of the Sysadmin team at that point but I know they had to restart SpamAssassin relatively frequently, manually clear out the email queue when people noticed they weren’t receiving email, etc.

Continue reading “The Barracuda Spam Firewall VMware Appliance (Vx) finally exists!”

Posted on

Long email signatures amuse me.

Putting a glossary in your emails is a new one for me. Not really a bad idea if you deal with lots of industry-specific terminology and lots of non-industry people.


From: Clickatell SC [noreply@clickatell.com]
Sent: Sunday, September 12, 2010 1:04 AM
To: Evan D. Hoffman
Subject: Clickatell System Alert

Dear Clickatell Client,

(Blah Blah Blah)

Apologies for any inconvenience caused.

Email:
support@clickatell.com

Phone:
+27 21 910 7700 (South Africa)
+1 650 641 0011 (US)
+44 20 7060 0212 (UK)
+61 290 371 951 (Australia)

--
Clickatell

www.clickatell.com

Our Vision
Connecting the world through any message, anywhere.

-------------------------------------------------------------------------------
Terminology:

-Mobile originated (MO): A message sent (originating) from a mobile handset to an application via Clickatell.

-Mobile terminated (MT): A message sent from an application to (terminating on) a mobile handset via Clickatell.

-Premium rated message (MO): A mobile user is charged a premium for the message that they send to a particular short or long code. This service is not available in all regions; please contact an Account Manager for more information.

-Revenue share: This refers to the portion of the premium charge associated with a premium rated message, which is passed on to the content provider.

-Content provider: This is the Clickatell customer who is offering one or more services that are usually premium rated SMS system.

-Customer: A registered Clickatell customer utilising the Clickatell API for message delivery and receipt.

-Sender ID: The “from” address that appears on the user’s handset. This is also known as the message originator or source address. A Sender ID must be registered within your account and approved by us before it may be used.

-Destination address: The mobile number/MSISDN of the handset to which the message must be delivered. The number should be in international number format, e.g. country code + local mobile number, excluding the leading zero (0).

-Source address: The Sender ID or From address of the SMS.

-Short code: A short number which is common across all the operators for a specific region.

-Subscriber: The mobile network subscriber who owns the mobile number (MSISDN) which will send or receive SMSs, or be billed for premium rated services.

-Upstream gateway: A network operator, third party or our own short message service centre (SMSC).

Posted on

Outlook 2007 & Exchange 2010 Autodiscover SSL certificate error annoyance

One of the more annoying side effects of migrating my mailbox to Exchange 2010 has been the nagging of Outlook 2007’s Autodiscovery feature. Now, every time I start Outlook I get hit with a certificate error for autodiscover.domain.com. Now, autodiscover.domain.com is a CNAME to mail.domain.com, which is the OWA URL for the CAS. The SSL certificate is valid – but it’s valid for mail.domain.com. I could buy a SSL certificate from GoDaddy for $12.99 (an insanely great price, btw) for “autodiscover” but that would also require using another IP address on the CAS (since you can can only bind one SSL certificate to an IP:port pair), and that seems like a waste of an IP address.

I found a possible solution in KB 940726. Basically you use this cmdlet to change the Autodiscover URI for internal clients:

Set-ClientAccessServer –AutodiscoverServiceInternalUri https://mail.contoso.com/autodiscover/autodiscover.xml

You’d replace mail.contoso.com with the external URL of your OWA server (in my case, mail.domain.com). I’ve made the changes but I think I need to wait for AD propagation. Hopefully this will resolve it, because I don’t want to move everyone’s mailboxes over until this thing is “perfect,” whatever that means.

Edit: I also needed to add a SRV record so Outlook would know what host to check for autodiscovery when outside the domain.

Edit 2:: Also need to install a hotfix or be running Outlook 2007 SP1 or later for the SRV functionality.

Edit 3: It occurs to me that a simpler fix for this issue may be simply to delete the DNS record for autodiscover entirely. That way, when Outlook attempts to open the SSL connection to autodiscover.domain.com, it gets a NXDOMAIN error (should) silently skip it. Unfortunately we have wildcard DNS active for our domain.

Other useful resources:

Posted on

Moved my email directly to Google

For a few years I’ve been funneling my various inboxes directly to my Gmail account. The massive storage, great web UI, and spam filtering made it a no-brainer. I’d basically been relying on a .forward file to do this until a couple of days ago when I signed up with Google Apps for evanhoffman.com, changed the MX records to point to Google, and … all my email still forwards to my Gmail account. But at least Bluehost is out of the loop now. Really, at this point the only reason for me to stay with them is the massive storage quota I have with them for the gallery; but even that doesn’t make much sense with the Picasa web albums stuff. They own Blogger so I assume I could port this whole thing over. Google’s free so I guess I’d save like $100 per year moving everything there, and I mostly only use it for email anyway. I think I’m paying like $8/month… With thin-provisioned disk space they could drop that to like $3/month probably and still make $.

Anyway, tomorrow’s the closing. I should go to bed. Oh, and WaMu is gone – attempting to login to the Wamu online banking redirects me to Chase.com now.