Tag: windows

Posted on

Exchange (OWA) CAS crashes with 503 error – again

This just started happening again, with these errors appearing in the event viewer:

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/17/2011 6:47:07 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' terminated
unexpectedly. The process id was '3108'. The process exit code was
'0x800703e9'.

Log Name: Application
Source: Application Error
Date: 9/17/2011 6:46:30 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time
stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time
stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

After reviewing the IIS logs and the event logs, I think it has to do with the WebReady document viewer – the thing in OWA that renders and lets you view .doc attachments within the browser rather than forcing you to open Word or Excel. I think users were attempting to open corrupted files and that was causing it to crash. I’ve disabled Webready in EMC (Server Config -> CAS) and I’ll see what happens.

Go Daddy $12.99 SSL Sale!

Posted on

Renaming a single-label domain to a FQDN

Long ago — eons, perhaps — before I had anything to do with the Windows environment here, someone created the AD domain in my company as a single-label domain (e.g. instead of “example.com” our domain is just “example”). Over the years this has led to lots of “fun” on the part of Windows admins who’ve worked here as the implications of this choice became more apparent.

Since I inherited this system about a year ago, I haven’t really bumped up against any problems stemming from the single-label domain issue… until now. I recently attempted to add a new Windows 2008r2 file server to our DFS replication group/namespace. This totally failed for some mysterious reason. Well, I shouldn’t say “totally” failed, as I was able to add it to the DFS replication group, but unable to add it to the DFS namespace. In my attempt to debug the namespace issue, I deleted the namespace and attempted to recreate it, but just kept getting this error: The namespace cannot be queried. The specified domain either does not exist or could not be contacted.. I couldn’t do anything with the namespace – even clicking on it in the DFS Management console brought up an error. After some searching I found that this was likely due to having a single-label domain. I wasn’t sure why the error was happening even on Windows 2003 machines though, maybe joining a 2008r2 box to the domain made some schema changes? I tried a few suggestions like editing the hosts file but nothing seemed to resolve this.

Fortunately, we didn’t really need DFS namespaces and were able to just direct everybody to the fileserver via its DNS name, though as you can imagine this was clumsy. However, since this has been a problem since time immemorial, I figured it was time to see if it was fixable. After some quick searching, I found RENDOM. However, after even more searching I discovered this TechNet article which says:

The domain rename operation is not supported in Microsoft Exchange Server 2007 or Exchange Server 2010. DNS domain rename is supported in Exchange Server 2003. However, renaming of the NetBIOS domain name is not supported in any version of Exchange Server. Other non-Microsoft applications might also not support domain rename.

Well. We’re running Exchange 2010. So now what? I guess we’re going to have to create a second domain and migrate over to it. We’d already discussed this as a likely way of implementing the rename anyway, since it didn’t seem like “RENDOM” had any rollback procedure – it either just works (hahaha) or semi-works and semi-fails, leaving a wake of destruction throughout AD. Building a second domain seems like a lot of work, but at least we can move users over one at a time, and we get the side benefit of starting fresh, outgrowing the 5+ years of crud that’s accumulated in our AD.

Guess we’ll see what happens. Neither option seems like much fun. I guess the alternative is do nothing, but Microsoft clearly doesn’t think very highly of single-label domains, and anyone who asks about them gets looked at funny. At least it gives us something to do!

Posted on

Autodiscover mysteriously stopped working (Exchange 2010)

I had Autodiscover working for months but recently it just stopped. I’m not sure why, but it may be related to removing the last Exchange 2003 servers from service recently. Maybe some setting got wiped from AD when I uninstalled Exchange 2003 (as per the procedure Microsoft gives). Basically what was happening was that the email address field was being autopopulated by the user’s UPN rather than their email address. Since we have a single label domain, the UPN isn’t a valid email address, and autodiscovery fails.

Anyway, I ran Get-AutodiscoverVirtualDirectory and it looks like the autodiscover URL isn’t set. Pretty sure I set this at some point.

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl

InternalUrl :
ExternalUrl :

[PS] C:\Windows\system32>

I just piped this to Set-AutodiscoverVirtualDirectory to correct the problem:

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | Set-AutodiscoverVirtualDirectory -ExternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml' -InternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml'
[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl


InternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml
ExternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml


[PS] C:\Windows\system32>

After resetting the InternalURL and ExternalURL, autodiscover works again (we have SRV records that tell Outlook to look at webmail.example.com for the Autodiscover service).

Hooray!

Posted on

Wasted time with Exchange 2010, SquirrelMail, and IMAP-SSL

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Some of the errors SquirrelMail was reporting were “Error connecting to IMAP server xxxx Server error: (0)” and “Error connecting to IMAP server: tls://xxxx:993. 0: ”

Nothing would actually work on port 993. Telnet to 993 got this:

$ telnet 10.0.20.18 993
Trying 10.0.20.18...
Connected to 10.0.20.18.
Escape character is '^]'.
* BYE Connection is closed. 14
Connection closed by foreign host.

After too much poking, I decided to go down to a lower level and do a simple openssl certificate retrieval and see what came back:


$ openssl s_client -connect 10.0.20.18:993
CONNECTED(00000003)
140281653434184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 113 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

That didn’t look right, so I ran it against the same server on port 443 and got back a real certificate. Same for port 995 (pop3s):

$ openssl s_client -connect 10.0.20.18:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = info@valicert.com
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1

(snip)

So there’s just something wrong with SSL on port 993. To make a long story short, I had to use the Enable-ExchangeCertificate to apply the SSL certificate to port 993. First, run “Get-ExchangeCertificate” to list the available certificates and retrieve the Thumbprint.

[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  .P....     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  I..W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

Copy & paste the thumbprint for whichever cert you want to use into Enable-ExchangeCertificate:

[PS] C:\Windows\system32>Enable-ExchangeCertificate -ThumbPrint xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -Services IIS,P
OP,IMAP -DoNotRequireSSL
[PS] C:\Windows\system32>Get-ExchangeCertificate

Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  ......     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  IP.W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

After running that, imaps on port 993 worked perfectly. I can connect to it with both SquirrelMail and Thunderbird.

The SquirrelMail config looks like this:

IMAP Settings
--------------
4.  IMAP Server            : webmail.example.com
5.  IMAP Port              : 993
6.  Authentication type    : login
7.  Secure IMAP (TLS)      : true
8.  Server software        : exchange
9.  Delimiter              : detect

Edit Feb 15, 2011: I just renewed the SSL cert and ran into a problem with a Ruby script that was suddenly unable to check a mailbox over IMAPS. The error received was:

/usr/lib/ruby/1.8/net/imap.rb:898:in `connect': unknown protocol (OpenSSL::SSL::SSLError)
        from /usr/lib/ruby/1.8/net/imap.rb:898:in `initialize'

After a few minutes, I remembered this blog post and ran Enable-ExchangeCertificate and it worked again. Glad I wrote it down.

CONNECTED(00000003) 26831:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

Posted on

Using Zabbix for SNMP monitoring disk usage percent for Windows hosts

A few years ago we moved from Nagios to Zabbix for our server monitoring needs. I wasn’t a big fan of Nagios, finding it a pain to manage with its myriad configuration files. It’s probably gotten better since I last toyed with it but since we moved to Zabbix I haven’t had much reason to look at Nagios again.
Continue reading “Using Zabbix for SNMP monitoring disk usage percent for Windows hosts”

Posted on

Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

Continue reading “Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group”

Posted on

Exchange 2010 Post-Upgrade weirdness: can’t edit Mail Non-Universal Group or Security Group

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

Continue reading “Exchange 2010 Post-Upgrade weirdness: can’t edit Mail Non-Universal Group or Security Group”

Posted on

Finally, all users moved from Exchange 2003 to Exchange 2010.

I’ve been working on migrating our Exchange environment from 2003 to 2010 for several months. My first post about this is from April 14th, when I was just trying to virtualize our existing Exchange 2003 system. Once that was complete, I started playing around with Exchange 2010 around June or July, and had most of the users moved over to 2010 by the end of August. The last holdouts were Blackberry users. I couldn’t move their mailboxes because our BES was hosted on our original Exchange 2003 server.

BES is another product that I inherited that I had no experience with. It’s BES 4.1.x and while I wasn’t a fan of the UI it seemed to do its job. However, when I started moving people to Exchange 2010 I learned that BES 4.1 doesn’t support Exchange 2010. So, to cut the (absurdly long) story short, I setup BES Express on a new VM, pointed it at our Exchange 2010 server, tested it out (and it worked), and just last week was able (finally) to move the last few users over to Exchange 2010. BES users had to have their phones wiped to join them to the BES Express server, which was the major sticking point.

I can’t believe it actually took that long to complete, but we managed to move all user mailboxes twice (Ex2003 physical -> Ex2003 VM, then Ex2003 VM -> Ex2010 VM) with no noticeable interruption to users (we did the moves at night). OWA 2010 alone would make it worth the upgrade, but I’m actually loving the Exchange Management Shell too.

Anyway… nice to have it completed.

Posted on

One reason I hate iTunes.

I’ve always hated iTunes. It’s a huge pile of bloatware and it’s slow as poo. It’s like 100 mb or more for an mp3 player. I remember winamp playing mp3s when it was a 500k download. Anyway.

I keep all my music on a Linux machine running samba. This way it’s available to every machine in the house. When I had Winamp on all my machines this was wonderful. But now that I’m forced into iTunes (thanks to having an iPhone), it turns out to be a major pain. In iTunes I unchecked the box for “let iTunes keep my libary organized” to prevent it from copying the entire library to each computer’s local disk. Initially adding my library of ~4000 tracks to iTunes takes over an hour (100 mbit wire) – it would take about 5 minutes in Winamp, even reading the ID3 tags for each track as it was added (rather than lazily as the song was played).

But the thing that iTunes does that is so annoying it prompted me to write this whiny rant is:

iTunes "Song Not Found"
iTunes 'Song Not Found'

If, for some reason, my M: drive (where the Samba share is mapped) is not connected when iTunes starts, every song in the library gets this “!” exclamation point of doom. If I attempt to play any of these tracks, I am given the option to locate the file. Nice in theory, but locating all 4000 tracks isn’t realistic. If I quit iTunes, reconnect the M: drive, and reopen iTunes, the ! persists. The only solution I’ve found to this is deleting the entire library from iTunes and re-adding it, which as I said, takes an extremely long time.

I have other reasons for hating iTunes, this is a blog, not a book.

Posted on

Converting Exchange 2003 conference rooms to Exchange 2010

I’m wrapping up moving mailboxes to Exchange 2010. The last ones to be moved (except for BlackBerry users… thanks BES) are the conference rooms. So the first step was to move them using the Local Move tool, which was pretty simple. But I don’t want them in 2010 as user mailboxes if they can be designated as “rooms,” which they can. So here’s how I’m doing it:

Identify the mailboxes to be moved

Once you figure out the syntax for the “-Filter” flag to get-mailbox, this is easy

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*conference*") }

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Conference Room2          ConferenceRoom2      exch2010be1      unlimited
Production Conference ... productionconf       exch2010be1      unlimited
Conference Room 1         conference1          exch2010be1      unlimited
L&D Conference Room       ldconference         exch2010be1      unlimited
Tech Conference Room      techconference       exch2010be1      unlimited
Client Services Confer... csconference         exch2010be1      unlimited
Suite 202 Conference Room 202conf              exch2010be1      unlimited

Convert them to rooms

As Microsoft says in this story about converting mailboxes to rooms, this can only be done via Exchange Management Shell (not EMC), so just pipe the output from the previous command to Set-Mailbox -Type Room:

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*confe
rence*") } | set-mailbox -type room
[PS] C:\Windows\system32>

Done! Now when you create an appointment in Outlook 2007, in Scheduling Assistant, you can click the “Add Room” button to add a room. Hooray.