Exchange (OWA) CAS crashes with 503 error – again

This just started happening again, with these errors appearing in the event viewer:

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/17/2011 6:47:07 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' terminated
unexpectedly. The process id was '3108'. The process exit code was
'0x800703e9'.

Log Name: Application
Source: Application Error
Date: 9/17/2011 6:46:30 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time
stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time
stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

After reviewing the IIS logs and the event logs, I think it has to do with the WebReady document viewer – the thing in OWA that renders and lets you view .doc attachments within the browser rather than forcing you to open Word or Excel. I think users were attempting to open corrupted files and that was causing it to crash. I’ve disabled Webready in EMC (Server Config -> CAS) and I’ll see what happens.

Go Daddy $12.99 SSL Sale!

Advertisements

Renaming a single-label domain to a FQDN

Long ago — eons, perhaps — before I had anything to do with the Windows environment here, someone created the AD domain in my company as a single-label domain (e.g. instead of “example.com” our domain is just “example”). Over the years this has led to lots of “fun” on the part of Windows admins who’ve worked here as the implications of this choice became more apparent.

Since I inherited this system about a year ago, I haven’t really bumped up against any problems stemming from the single-label domain issue… until now. I recently attempted to add a new Windows 2008r2 file server to our DFS replication group/namespace. This totally failed for some mysterious reason. Well, I shouldn’t say “totally” failed, as I was able to add it to the DFS replication group, but unable to add it to the DFS namespace. In my attempt to debug the namespace issue, I deleted the namespace and attempted to recreate it, but just kept getting this error: The namespace cannot be queried. The specified domain either does not exist or could not be contacted.. I couldn’t do anything with the namespace – even clicking on it in the DFS Management console brought up an error. After some searching I found that this was likely due to having a single-label domain. I wasn’t sure why the error was happening even on Windows 2003 machines though, maybe joining a 2008r2 box to the domain made some schema changes? I tried a few suggestions like editing the hosts file but nothing seemed to resolve this.

Fortunately, we didn’t really need DFS namespaces and were able to just direct everybody to the fileserver via its DNS name, though as you can imagine this was clumsy. However, since this has been a problem since time immemorial, I figured it was time to see if it was fixable. After some quick searching, I found RENDOM. However, after even more searching I discovered this TechNet article which says:

The domain rename operation is not supported in Microsoft Exchange Server 2007 or Exchange Server 2010. DNS domain rename is supported in Exchange Server 2003. However, renaming of the NetBIOS domain name is not supported in any version of Exchange Server. Other non-Microsoft applications might also not support domain rename.

Well. We’re running Exchange 2010. So now what? I guess we’re going to have to create a second domain and migrate over to it. We’d already discussed this as a likely way of implementing the rename anyway, since it didn’t seem like “RENDOM” had any rollback procedure – it either just works (hahaha) or semi-works and semi-fails, leaving a wake of destruction throughout AD. Building a second domain seems like a lot of work, but at least we can move users over one at a time, and we get the side benefit of starting fresh, outgrowing the 5+ years of crud that’s accumulated in our AD.

Guess we’ll see what happens. Neither option seems like much fun. I guess the alternative is do nothing, but Microsoft clearly doesn’t think very highly of single-label domains, and anyone who asks about them gets looked at funny. At least it gives us something to do!

Autodiscover mysteriously stopped working (Exchange 2010)

I had Autodiscover working for months but recently it just stopped. I’m not sure why, but it may be related to removing the last Exchange 2003 servers from service recently. Maybe some setting got wiped from AD when I uninstalled Exchange 2003 (as per the procedure Microsoft gives). Basically what was happening was that the email address field was being autopopulated by the user’s UPN rather than their email address. Since we have a single label domain, the UPN isn’t a valid email address, and autodiscovery fails.

Anyway, I ran Get-AutodiscoverVirtualDirectory and it looks like the autodiscover URL isn’t set. Pretty sure I set this at some point.

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl

InternalUrl :
ExternalUrl :

[PS] C:\Windows\system32>

I just piped this to Set-AutodiscoverVirtualDirectory to correct the problem:

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | Set-AutodiscoverVirtualDirectory -ExternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml' -InternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml'
[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl


InternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml
ExternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml


[PS] C:\Windows\system32>

After resetting the InternalURL and ExternalURL, autodiscover works again (we have SRV records that tell Outlook to look at webmail.example.com for the Autodiscover service).

Hooray!

Wasted time with Exchange 2010, SquirrelMail, and IMAP-SSL

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Some of the errors SquirrelMail was reporting were “Error connecting to IMAP server xxxx Server error: (0)” and “Error connecting to IMAP server: tls://xxxx:993. 0: ”

Nothing would actually work on port 993. Telnet to 993 got this:

$ telnet 10.0.20.18 993
Trying 10.0.20.18...
Connected to 10.0.20.18.
Escape character is '^]'.
* BYE Connection is closed. 14
Connection closed by foreign host.

After too much poking, I decided to go down to a lower level and do a simple openssl certificate retrieval and see what came back:


$ openssl s_client -connect 10.0.20.18:993
CONNECTED(00000003)
140281653434184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 113 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

That didn’t look right, so I ran it against the same server on port 443 and got back a real certificate. Same for port 995 (pop3s):

$ openssl s_client -connect 10.0.20.18:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = info@valicert.com
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1

(snip)

So there’s just something wrong with SSL on port 993. To make a long story short, I had to use the Enable-ExchangeCertificate to apply the SSL certificate to port 993. First, run “Get-ExchangeCertificate” to list the available certificates and retrieve the Thumbprint.

[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  .P....     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  I..W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

Copy & paste the thumbprint for whichever cert you want to use into Enable-ExchangeCertificate:

[PS] C:\Windows\system32>Enable-ExchangeCertificate -ThumbPrint xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -Services IIS,P
OP,IMAP -DoNotRequireSSL
[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  ......     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  IP.W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

After running that, imaps on port 993 worked perfectly. I can connect to it with both SquirrelMail and Thunderbird.

The SquirrelMail config looks like this:

IMAP Settings
--------------
4.  IMAP Server            : webmail.example.com
5.  IMAP Port              : 993
6.  Authentication type    : login
7.  Secure IMAP (TLS)      : true
8.  Server software        : exchange
9.  Delimiter              : detect

Edit Feb 15, 2011: I just renewed the SSL cert and ran into a problem with a Ruby script that was suddenly unable to check a mailbox over IMAPS. The error received was:

/usr/lib/ruby/1.8/net/imap.rb:898:in `connect': unknown protocol (OpenSSL::SSL::SSLError)
        from /usr/lib/ruby/1.8/net/imap.rb:898:in `initialize'

After a few minutes, I remembered this blog post and ran Enable-ExchangeCertificate and it worked again. Glad I wrote it down.

CONNECTED(00000003) 26831:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

Using Zabbix for SNMP monitoring disk usage percent for Windows hosts

A few years ago we moved from Nagios to Zabbix for our server monitoring needs. I wasn’t a big fan of Nagios, finding it a pain to manage with its myriad configuration files. It’s probably gotten better since I last toyed with it but since we moved to Zabbix I haven’t had much reason to look at Nagios again.
Continue reading “Using Zabbix for SNMP monitoring disk usage percent for Windows hosts”

Exchange 2010 Post-Upgrade weirdness: can’t edit Mail Non-Universal Group or Security Group

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

Continue reading “Exchange 2010 Post-Upgrade weirdness: can’t edit Mail Non-Universal Group or Security Group”

Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

Continue reading “Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group”