Exchange (OWA) CAS crashes with 503 error – again

This just started happening again, with these errors appearing in the event viewer:

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/17/2011 6:47:07 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' terminated
unexpectedly. The process id was '3108'. The process exit code was
'0x800703e9'.

Log Name: Application
Source: Application Error
Date: 9/17/2011 6:46:30 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time
stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time
stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

After reviewing the IIS logs and the event logs, I think it has to do with the WebReady document viewer – the thing in OWA that renders and lets you view .doc attachments within the browser rather than forcing you to open Word or Excel. I think users were attempting to open corrupted files and that was causing it to crash. I’ve disabled Webready in EMC (Server Config -> CAS) and I’ll see what happens.

Go Daddy $12.99 SSL Sale!

Renaming a single-label domain to a FQDN

Long ago — eons, perhaps — before I had anything to do with the Windows environment here, someone created the AD domain in my company as a single-label domain (e.g. instead of “example.com” our domain is just “example”). Over the years this has led to lots of “fun” on the part of Windows admins who’ve worked here as the implications of this choice became more apparent.

Since I inherited this system about a year ago, I haven’t really bumped up against any problems stemming from the single-label domain issue… until now. I recently attempted to add a new Windows 2008r2 file server to our DFS replication group/namespace. This totally failed for some mysterious reason. Well, I shouldn’t say “totally” failed, as I was able to add it to the DFS replication group, but unable to add it to the DFS namespace. In my attempt to debug the namespace issue, I deleted the namespace and attempted to recreate it, but just kept getting this error: The namespace cannot be queried. The specified domain either does not exist or could not be contacted.. I couldn’t do anything with the namespace – even clicking on it in the DFS Management console brought up an error. After some searching I found that this was likely due to having a single-label domain. I wasn’t sure why the error was happening even on Windows 2003 machines though, maybe joining a 2008r2 box to the domain made some schema changes? I tried a few suggestions like editing the hosts file but nothing seemed to resolve this.

Fortunately, we didn’t really need DFS namespaces and were able to just direct everybody to the fileserver via its DNS name, though as you can imagine this was clumsy. However, since this has been a problem since time immemorial, I figured it was time to see if it was fixable. After some quick searching, I found RENDOM. However, after even more searching I discovered this TechNet article which says:

The domain rename operation is not supported in Microsoft Exchange Server 2007 or Exchange Server 2010. DNS domain rename is supported in Exchange Server 2003. However, renaming of the NetBIOS domain name is not supported in any version of Exchange Server. Other non-Microsoft applications might also not support domain rename.

Well. We’re running Exchange 2010. So now what? I guess we’re going to have to create a second domain and migrate over to it. We’d already discussed this as a likely way of implementing the rename anyway, since it didn’t seem like “RENDOM” had any rollback procedure – it either just works (hahaha) or semi-works and semi-fails, leaving a wake of destruction throughout AD. Building a second domain seems like a lot of work, but at least we can move users over one at a time, and we get the side benefit of starting fresh, outgrowing the 5+ years of crud that’s accumulated in our AD.

Guess we’ll see what happens. Neither option seems like much fun. I guess the alternative is do nothing, but Microsoft clearly doesn’t think very highly of single-label domains, and anyone who asks about them gets looked at funny. At least it gives us something to do!

Autodiscover mysteriously stopped working (Exchange 2010)

I had Autodiscover working for months but recently it just stopped. I’m not sure why, but it may be related to removing the last Exchange 2003 servers from service recently. Maybe some setting got wiped from AD when I uninstalled Exchange 2003 (as per the procedure Microsoft gives). Basically what was happening was that the email address field was being autopopulated by the user’s UPN rather than their email address. Since we have a single label domain, the UPN isn’t a valid email address, and autodiscovery fails.

Anyway, I ran Get-AutodiscoverVirtualDirectory and it looks like the autodiscover URL isn’t set. Pretty sure I set this at some point.

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl

InternalUrl :
ExternalUrl :

[PS] C:\Windows\system32>

I just piped this to Set-AutodiscoverVirtualDirectory to correct the problem:

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | Set-AutodiscoverVirtualDirectory -ExternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml' -InternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml'
[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl


InternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml
ExternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml


[PS] C:\Windows\system32>

After resetting the InternalURL and ExternalURL, autodiscover works again (we have SRV records that tell Outlook to look at webmail.example.com for the Autodiscover service).

Hooray!

Wasted time with Exchange 2010, SquirrelMail, and IMAP-SSL

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Some of the errors SquirrelMail was reporting were “Error connecting to IMAP server xxxx Server error: (0)” and “Error connecting to IMAP server: tls://xxxx:993. 0: ”

Nothing would actually work on port 993. Telnet to 993 got this:

$ telnet 10.0.20.18 993
Trying 10.0.20.18...
Connected to 10.0.20.18.
Escape character is '^]'.
* BYE Connection is closed. 14
Connection closed by foreign host.

After too much poking, I decided to go down to a lower level and do a simple openssl certificate retrieval and see what came back:


$ openssl s_client -connect 10.0.20.18:993
CONNECTED(00000003)
140281653434184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 113 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

That didn’t look right, so I ran it against the same server on port 443 and got back a real certificate. Same for port 995 (pop3s):

$ openssl s_client -connect 10.0.20.18:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = info@valicert.com
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1

(snip)

So there’s just something wrong with SSL on port 993. To make a long story short, I had to use the Enable-ExchangeCertificate to apply the SSL certificate to port 993. First, run “Get-ExchangeCertificate” to list the available certificates and retrieve the Thumbprint.

[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  .P....     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  I..W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

Copy & paste the thumbprint for whichever cert you want to use into Enable-ExchangeCertificate:

[PS] C:\Windows\system32>Enable-ExchangeCertificate -ThumbPrint xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -Services IIS,P
OP,IMAP -DoNotRequireSSL
[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  ......     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  IP.W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

After running that, imaps on port 993 worked perfectly. I can connect to it with both SquirrelMail and Thunderbird.

The SquirrelMail config looks like this:

IMAP Settings
--------------
4.  IMAP Server            : webmail.example.com
5.  IMAP Port              : 993
6.  Authentication type    : login
7.  Secure IMAP (TLS)      : true
8.  Server software        : exchange
9.  Delimiter              : detect

Edit Feb 15, 2011: I just renewed the SSL cert and ran into a problem with a Ruby script that was suddenly unable to check a mailbox over IMAPS. The error received was:

/usr/lib/ruby/1.8/net/imap.rb:898:in `connect': unknown protocol (OpenSSL::SSL::SSLError)
        from /usr/lib/ruby/1.8/net/imap.rb:898:in `initialize'

After a few minutes, I remembered this blog post and ran Enable-ExchangeCertificate and it worked again. Glad I wrote it down.

CONNECTED(00000003) 26831:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

Continue reading “Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group”

Exchange 2010 Post-Upgrade weirdness: can’t edit Mail Non-Universal Group or Security Group

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

Continue reading “Exchange 2010 Post-Upgrade weirdness: can’t edit Mail Non-Universal Group or Security Group”

Changing all Exchange mailboxes to use database defaults for quota limits with PowerShell

Here’s an example of why I’ve come to love PowerShell. Under Exchange 2003, we set users’ mailbox quota limits per-user, since we initially set them ridiculously high (around 20 GB). The reason we did that was that prior to this year we actually didn’t have any quotas on user mailboxes and the average mailbox was about 9 GB. In preparation for moving Exchange to the datacenter (over our measly 5Mbps upload line) we introduced quotas and over the course of about 2 months gradually lowered the limits as people whittled away their old email until they were under the quota we had established (about 2.25 GB for the no-send limit — I forgot the rationale for that number — and 1.75 GB for the warning limit [the no-receive limit was set to about 40GB since I never wanted users to hit that limit]).

I think the quota limits Exchange 2003 would allow you to set through the UI were capped at around 2.0 GB so to set quota limits above this we used ADModify to update the mDBQuota* settings for each user directly in their AD entries.

However, now all those users have been moved to Exchange 2010 which let me easily set the quota limits on the Exchange DB to 2000 MB (warning) and 3000 MB (no-send). The problem with this was that all the users were set to override the database setting. How could I quickly set them all to use the database default once again? With PowerShell, this was simple:

First, get the list of all users not using the database defaults:
[PS] C:\Windows\system32>get-mailbox -filter { usedatabasequotadefaults -eq $false -AND recipientTypeDetails -eq 'usermailbox' }

When doing any bulk operation, always verify the list of objects you’re going to be modifying before actually doing the modification. 🙂 The first filter I created didn’t have the RecipientTypeDetails in it and included a DiscoveryMailbox, which I didn’t want to touch. The above command affects only UserMailboxes. If the list looks ok, set them all back to defaults:

[PS] C:\Windows\system32>get-mailbox -filter { usedatabasequotadefaults -eq $false -AND recipientTypeDetails -eq 'usermailbox' } | set-mailbox -UseDatabaseQuotaDefaults $true

And voilà, everyone’s back to using DB defaults (which I increased to 3000MB to celebrate not having to do any more mailbox moves any time soon).

Finally, all users moved from Exchange 2003 to Exchange 2010.

I’ve been working on migrating our Exchange environment from 2003 to 2010 for several months. My first post about this is from April 14th, when I was just trying to virtualize our existing Exchange 2003 system. Once that was complete, I started playing around with Exchange 2010 around June or July, and had most of the users moved over to 2010 by the end of August. The last holdouts were Blackberry users. I couldn’t move their mailboxes because our BES was hosted on our original Exchange 2003 server.

BES is another product that I inherited that I had no experience with. It’s BES 4.1.x and while I wasn’t a fan of the UI it seemed to do its job. However, when I started moving people to Exchange 2010 I learned that BES 4.1 doesn’t support Exchange 2010. So, to cut the (absurdly long) story short, I setup BES Express on a new VM, pointed it at our Exchange 2010 server, tested it out (and it worked), and just last week was able (finally) to move the last few users over to Exchange 2010. BES users had to have their phones wiped to join them to the BES Express server, which was the major sticking point.

I can’t believe it actually took that long to complete, but we managed to move all user mailboxes twice (Ex2003 physical -> Ex2003 VM, then Ex2003 VM -> Ex2010 VM) with no noticeable interruption to users (we did the moves at night). OWA 2010 alone would make it worth the upgrade, but I’m actually loving the Exchange Management Shell too.

Anyway… nice to have it completed.

Converting Exchange 2003 conference rooms to Exchange 2010

I’m wrapping up moving mailboxes to Exchange 2010. The last ones to be moved (except for BlackBerry users… thanks BES) are the conference rooms. So the first step was to move them using the Local Move tool, which was pretty simple. But I don’t want them in 2010 as user mailboxes if they can be designated as “rooms,” which they can. So here’s how I’m doing it:

Identify the mailboxes to be moved

Once you figure out the syntax for the “-Filter” flag to get-mailbox, this is easy

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*conference*") }

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Conference Room2          ConferenceRoom2      exch2010be1      unlimited
Production Conference ... productionconf       exch2010be1      unlimited
Conference Room 1         conference1          exch2010be1      unlimited
L&D Conference Room       ldconference         exch2010be1      unlimited
Tech Conference Room      techconference       exch2010be1      unlimited
Client Services Confer... csconference         exch2010be1      unlimited
Suite 202 Conference Room 202conf              exch2010be1      unlimited

Convert them to rooms

As Microsoft says in this story about converting mailboxes to rooms, this can only be done via Exchange Management Shell (not EMC), so just pipe the output from the previous command to Set-Mailbox -Type Room:

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*confe
rence*") } | set-mailbox -type room
[PS] C:\Windows\system32>

Done! Now when you create an appointment in Outlook 2007, in Scheduling Assistant, you can click the “Add Room” button to add a room. Hooray.

Exchange 2010 – Out-of-office response (OOF) won’t turn off?

Two users reported the same problem this week: they turned on their out-of-office reply while they were out, then came back and turned it off. Except even after they turned it off, the autoreply was still being sent out. I had them log in to OWA and make sure it was off (maybe some weird bug with Outlook not registering the change in the server), which it was in both cases. I Googled hard and fast and couldn’t find anyone with this same problem.

I went in with Powershell and checked their autoreply status via Get-MailboxAutoReplyConfiguration and it appears that it is, in fact, disabled:

[PS] C:Windowssystem32>Get-mailbox -identity username | Get-MailboxAutoReplyConfiguration
RunspaceId       : 7ad7e9af-cd57-4572-a4fd-c1e999e4b9a5
AutoReplyState   : Disabled
EndTime          : 8/12/2010 12:00:00 PM
ExternalAudience : All
ExternalMessage  :
InternalMessage  :
StartTime        : 8/11/2010 12:00:00 PM
MailboxOwnerId   :  [removed]
Identity         :  [removed]
IsValid          : True

I used Set-MailboxAutoReplyConfiguration to set the messages to “” (empty string) and it’s still sending the user’s autoresponse, from before I blanked it out. My working theory right now is that the out-of-office message was set on both the Exchange 2010 server and the Exchange 2003 server (where the mailboxes were before I migrated them to 2010).

What a fun problem! It’s hard to test whether I’ve fixed it, since each sender only receives the message once, so I have to keep creating new test email addresses to send test messages.

Also, as an aside, why is “out-of-office” abbreviated “OOF” in Microsoft’s docs?

Edit 1: I had one user verify the message was off in OWA and then start Outlook via Start -> Run… “outlook /cleanrules” and this seemed to resolve the issue. Hopefully this isn’t required every time…