Exchange CAS crashes, users get 503 when accessing OWA… but ActiveSync works

Since migrating to Exchange 2010, every couple of weeks it looks like IIS crashes on the CAS. Going to the OWA URL in a browser would yield a 503 error. Strangely enough, ActiveSync worked fine (though Outlook Anywhere over HTTPS didn’t). Rebooting the CAS resolved the issue.

This is what the logs show (Website Pulse reported the outage around 11:45):

Information	1/16/2011 11:59	EventLog	6013	None	The system uptime is 1726976 seconds.
Error	1/16/2011 11:43	Microsoft-Windows-WAS	5002	None	Application pool 'MSExchangeOWAAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Warning	1/16/2011 11:43	Microsoft-Windows-WAS	5011	None	A process serving application pool 'MSExchangeOWAAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '13180'. The data field contains the error number.
Warning	1/16/2011 11:43	Microsoft-Windows-WAS	5011	None	A process serving application pool 'MSExchangeOWAAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '12856'. The data field contains the error number.
Warning	1/16/2011 11:42	Microsoft-Windows-WAS	5011	None	A process serving application pool 'MSExchangeOWAAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '15328'. The data field contains the error number.
Warning	1/16/2011 11:41	Microsoft-Windows-WAS	5011	None	A process serving application pool 'MSExchangeOWAAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '11724'. The data field contains the error number.
Warning	1/16/2011 11:41	Microsoft-Windows-WAS	5009	None	A process serving application pool 'MSExchangeOWAAppPool' terminated unexpectedly. The process id was '15868'. The process exit code was '0x800703e9'.
Information	1/16/2011 11:21	Service Control Manager	7036	None	The WinHTTP Web Proxy Auto-Discovery Service service entered the stopped state.

This happened before we had any antivirus on the machine (antivirus has been suggested as a culprit). The machine is Win 2008r2 x64 with all updates running Exchange 2010 with all updates.

This MS Technet article seems to suggest Automatic Updates may be related, but I don’t have Automatic Updates running.

This Experts-Exchange article seems to suggest it’s a COM permissions issue. I actually have no idea what that means:

This seems to be COM permissions.
Command prompt->dcomcnfg->component services->computers->right click on my computer->properties.
Go to COM security
Under launch and activation permissions
Add the SID S-1-5-82-3006700770-424185619-1745488364-794895919-4004696415 and provide local launch permissions.

then try and check if IISRESET still throws same error in eventlog.

I suppose I can try that next time though. I just have no idea why it’s crashing out of nowhere randomly.

From the Application log:

Log Name:      Application
Source:        Application Error
Date:          1/16/2011 11:41:17 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13
Event Xml:

  
    
    1000
    2
    100
    0x80000000000000
    
    31050
    Application
    exch2010fe1
    
  
  
    w3wp.exe
    7.5.7600.16385
    4a5bd0eb
    KERNELBASE.dll
    6.1.7600.16385
    4a5bdfe0
    e053534f
    000000000000aa7d
  

Log Name:      Application
Source:        Windows Error Reporting
Date:          1/16/2011 11:41:18 AM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      exch2010fe1
Description:
Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 7.5.7600.16385
P3: 4a5bd0eb
P4: KERNELBASE.dll
P5: 6.1.7600.16385
P6: 4a5bdfe0
P7: e053534f
P8: 000000000000aa7d
P9:
P10:

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_32ada61695ee2297c903c5b1ee2ccc1df1ba4d8_416794b1

Analysis symbol:
Rechecking for solution: 0
Report Id: 7031214d-218f-11e0-8593-00505697272d
Report Status: 4
Event Xml:

  
    
    1001
    4
    0
    0x80000000000000
    
    31051
    Application
    exch2010fe1
    
  
  
    
    
    0
    APPCRASH
    Not available
    0
    w3wp.exe
    7.5.7600.16385
    4a5bd0eb
    KERNELBASE.dll
    6.1.7600.16385
    4a5bdfe0
    e053534f
    000000000000aa7d
    
    
    
    
    
    
    C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_32ada61695ee2297c903c5b1ee2ccc1df1ba4d8_416794b1
    
    
    0
    7031214d-218f-11e0-8593-00505697272d
    4
  


Microsoft Office 2007's awful user interface

Office 2007 is pretty old by now, and I know much has been written on the move from a “normal” looking app to the “Ribbon” UI. I personally hate the change and feel Microsoft just changed the UI as a way to make the application look “different” so that people will look at it and go “oooh, shiny!” and not feel as bad about being forced into another $400 upgrade of a word processor. Sure, Excel’s row limit was finally raised beyond 64k, and I’m sure there were some other tweaks, but .docx? .xlsx? Yet more file formats, ensuring most businesses will feel compelled to upgrade. If your clients are upgrading, you’re going to have to.

Anyway, that’s all well documented. What may not be is the ridiculous location of the SMTP header info in a message in Outlook. If you want to view this interserver communication, which is invaluable when debugging mail issues, you can either A) right-click the message in the inbox, or B) … Well, in Office 2003, there was a way to do this from within the open message. I didn’t think there was a way to do it from within the message in Outlook 2007, but it turns out there is. It’s just retarded:

WTF?

Dear Microsoft: please drop the “Ribbon” completely and go back to menus, or at least provide that as an option. This UI is awful.

Microsoft Office 2007’s awful user interface

Office 2007 is pretty old by now, and I know much has been written on the move from a “normal” looking app to the “Ribbon” UI. I personally hate the change and feel Microsoft just changed the UI as a way to make the application look “different” so that people will look at it and go “oooh, shiny!” and not feel as bad about being forced into another $400 upgrade of a word processor. Sure, Excel’s row limit was finally raised beyond 64k, and I’m sure there were some other tweaks, but .docx? .xlsx? Yet more file formats, ensuring most businesses will feel compelled to upgrade. If your clients are upgrading, you’re going to have to.

Anyway, that’s all well documented. What may not be is the ridiculous location of the SMTP header info in a message in Outlook. If you want to view this interserver communication, which is invaluable when debugging mail issues, you can either A) right-click the message in the inbox, or B) … Well, in Office 2003, there was a way to do this from within the open message. I didn’t think there was a way to do it from within the message in Outlook 2007, but it turns out there is. It’s just retarded:

WTF?

Dear Microsoft: please drop the “Ribbon” completely and go back to menus, or at least provide that as an option. This UI is awful.

The Joy of Migrating from Exchange 2003 to 2010

I’ve been working on migrating from Exchange 2003 to Exchange 2010 for several weeks. Actually, at this point it feels like several months. Now that I think about it, I guess that’s because it’s actually been several months.

Back in January or February, I got fed up with the Exchange setup I inherited: our Exchange 2003 server was running on a server in the basement of our office, on non-UPS power, with a power company that likes to pull shenanigans (like 3-4 hour outages every few months). In addition, the physical machine itself has some weird bug where it would hang at the POST screen complaining about some USB device, even though there are no USB devices plugged in, and USB is disabled in the BIOS. Meanwhile, in the datacenter, I had recently finished migrating most of our ancient physical servers to virtual machines on beautiful new hardware. It didn’t take long to see the solution that seemed to be obvious: move Exchange to the datacenter, in a VM.

There was a major wrinkle in this plan, however: there were no quota limits enforced in Exchange, and the average mailbox was 6-7 gigabytes, with 4 users over 10 gigs. At the time, we only had a 5 mbit upload connection to the datacenter, and the total size of the mailboxes was around 400 gigs. I didn’t want to spend weeks and weeks moving tons of mail over a slow pipe – and with mailboxes being so big, I wasn’t sure I could even complete some of them overnight.

At this point I brought up the idea of migrating the company to Google Apps. I’m a big fan of Gmail and moving off of Exchange would have certainly simplified some aspects of my job, and nobody would need Outlook (especially not me). I knew it would be a tough sell internally, but the pricing certainly didn’t help; it came out to $83/user/year for Google Apps + document retention. The price came out to about the same as upgrading to Exchange 2010. If it had been half or a third the cost I may have pushed harder, but to make the story (a little) shorter, we ended up sticking with Exchange, and instituting quotas.

We phased in the quotas over the course of a month to give users time to archive and clean up their mailboxes. Once that was done, I setup a new Exchange 2003 frontend server (in a VM) in the datacenter and pointed our webmail (OWA & ActiveSync) there. So we had the frontend in the datacenter and the backend “mailbox” server still in the office. I then setup another VM running Exchange 2003 in the datacenter. This enabled me to move mailboxes over one at a time with almost no interruption in service, except for the user whose mail was in transit. Since we instituted quotas, the mailboxes were all under 2 GB, and I was able to do 6-10 mailboxes each night.

I can’t tell you how happy I was when we lost power yet everyone retained full connectivity to email via their phones (except BlackBerry users, since BES was still in the basement — note to RIM: ActiveSync!).

So phase 1 & 2 (instituting quotas and moving email out of the basement) were complete. Phase 3 was the bigger unknown – moving to Exchange 2010. After lots of reading and planning, installing, configuring and testing, about two weeks ago I setup a Client Access Server to serve as the new webmail “frontend.” Microsoft has some pretty great instructions for setting up 2003 and 2010 in coexistence, but basically you point your “real” webmail URL to the 2010 CAS and move your “old” Exchange 2003 webmail to another url (they suggest legacy.company.com). Then people log in to the 2010 interface, and if their mailbox is housed on the 2003 server, it seamlessly redirects them to https://legacy.company.com/, and they don’t have to log in again. Pretty slick, and I didn’t believe it would work until I saw it for myself (which, btw, it does). So ActiveSync and Outlook Anywhere were working through the 2010 CAS even for the users housed on the 2003 server (which was all of them).

This week I started moving users over to Exchange 2010. So far it’s been mostly positive. We have several Mac users, so the ability for them to have native mail & calendaring is pretty epic. The Outlook Web App in Exchange 2010 is phenomenal. I mean, it almost brings a tear to my eye, it’s so beautiful – especially when compared with 2003. And being able to do server-side searching in OWA & on my iPhone is fabulous.

All is not perfect, though. I keep getting stupid certificate errors for Autodiscover when I open Outlook 2007. I guess I’ll need to buy another SSL certificate and dedicate another IP to this service… ugh. And now that I moved my mailbox to Exchange 2010, Outlook Anywhere appears not to work. Oh well, almost there…