The Barracuda Spam Firewall VMware Appliance (Vx) finally exists!

When I started at my current company, spam was handled with a separate server running SpamAssassin and a few other services. This sort of got the job done but required babysitting. I wasn’t part of the Sysadmin team at that point but I know they had to restart SpamAssassin relatively frequently, manually clear out the email queue when people noticed they weren’t receiving email, etc.

Continue reading “The Barracuda Spam Firewall VMware Appliance (Vx) finally exists!”

The VMware datastore LUN 2TB (well, 1.99999 TB) size limit

I started migrating our physical machines to VMs using VMware a few years ago and the first problem I ran into is still the most annoying one: the size limit for LUNs is, per VMware’s docs, (2TB – 512B). That’s 512 bytes shy of 2TB, so basically 1.99999 TB, or 2047.999 GB. So when I create a new LUN for a datastore in the SAN the max size is 2047 GB. Now, as the VMware KB article states, this is a limitation of SCSI, not VMware per se, but that doesn’t make it any less annoying. When I first setup ESX, I created a 5 TB LUN for the datastore. It showed up in vCenter as 1 TB. After some Googling I learned of the 2 TB limit — the usable space is basically usable space = (size of lun) % 2TB, where % is the modulo operator — and found something suggesting using extents to expand the datastore across luns. I did that, but I later learned that there seems to be a consensus that extents should be avoided.

There are other things I learned along the way – that you want to limit the number of vmdks per datastore anyway, for example, due to the risk of SCSI reservation errors and IO contention, but these are all things that it feels like we shouldn’t have to worry about. I can see having separate LUNs/datastores for different logical groupings of disks, allowing you to have different snapshot schedules for each datastore, or allowing you to put an entire datastore in Tier 1 or Tier 3 (to use Compellent parlance) based on its value to you. But having to segregate stuff for technical reasons seems like a problem that should already be solved.

And maybe it is… I’ve never tried NFS datastores, but if I created an 8TB LUN, mapped it to a physical box (skirting the 2TB limit imposed by VMware), export the volume from the host over NFS and use that as the datastore, I guess I’d be able to do all the things I want. Hmm. I’ll have to think about that. I guess I’d still keep the ESX host local luns on iSCSI so they could boot from SAN, though I suppose when we move to ESXi that won’t be much of an issue anyway.

Hmm… Well, I started writing this as a rant but I think I just morphed it into a new research project.

The Joy of Migrating from Exchange 2003 to 2010

I’ve been working on migrating from Exchange 2003 to Exchange 2010 for several weeks. Actually, at this point it feels like several months. Now that I think about it, I guess that’s because it’s actually been several months.

Back in January or February, I got fed up with the Exchange setup I inherited: our Exchange 2003 server was running on a server in the basement of our office, on non-UPS power, with a power company that likes to pull shenanigans (like 3-4 hour outages every few months). In addition, the physical machine itself has some weird bug where it would hang at the POST screen complaining about some USB device, even though there are no USB devices plugged in, and USB is disabled in the BIOS. Meanwhile, in the datacenter, I had recently finished migrating most of our ancient physical servers to virtual machines on beautiful new hardware. It didn’t take long to see the solution that seemed to be obvious: move Exchange to the datacenter, in a VM.

There was a major wrinkle in this plan, however: there were no quota limits enforced in Exchange, and the average mailbox was 6-7 gigabytes, with 4 users over 10 gigs. At the time, we only had a 5 mbit upload connection to the datacenter, and the total size of the mailboxes was around 400 gigs. I didn’t want to spend weeks and weeks moving tons of mail over a slow pipe – and with mailboxes being so big, I wasn’t sure I could even complete some of them overnight.

At this point I brought up the idea of migrating the company to Google Apps. I’m a big fan of Gmail and moving off of Exchange would have certainly simplified some aspects of my job, and nobody would need Outlook (especially not me). I knew it would be a tough sell internally, but the pricing certainly didn’t help; it came out to $83/user/year for Google Apps + document retention. The price came out to about the same as upgrading to Exchange 2010. If it had been half or a third the cost I may have pushed harder, but to make the story (a little) shorter, we ended up sticking with Exchange, and instituting quotas.

We phased in the quotas over the course of a month to give users time to archive and clean up their mailboxes. Once that was done, I setup a new Exchange 2003 frontend server (in a VM) in the datacenter and pointed our webmail (OWA & ActiveSync) there. So we had the frontend in the datacenter and the backend “mailbox” server still in the office. I then setup another VM running Exchange 2003 in the datacenter. This enabled me to move mailboxes over one at a time with almost no interruption in service, except for the user whose mail was in transit. Since we instituted quotas, the mailboxes were all under 2 GB, and I was able to do 6-10 mailboxes each night.

I can’t tell you how happy I was when we lost power yet everyone retained full connectivity to email via their phones (except BlackBerry users, since BES was still in the basement — note to RIM: ActiveSync!).

So phase 1 & 2 (instituting quotas and moving email out of the basement) were complete. Phase 3 was the bigger unknown – moving to Exchange 2010. After lots of reading and planning, installing, configuring and testing, about two weeks ago I setup a Client Access Server to serve as the new webmail “frontend.” Microsoft has some pretty great instructions for setting up 2003 and 2010 in coexistence, but basically you point your “real” webmail URL to the 2010 CAS and move your “old” Exchange 2003 webmail to another url (they suggest legacy.company.com). Then people log in to the 2010 interface, and if their mailbox is housed on the 2003 server, it seamlessly redirects them to https://legacy.company.com/, and they don’t have to log in again. Pretty slick, and I didn’t believe it would work until I saw it for myself (which, btw, it does). So ActiveSync and Outlook Anywhere were working through the 2010 CAS even for the users housed on the 2003 server (which was all of them).

This week I started moving users over to Exchange 2010. So far it’s been mostly positive. We have several Mac users, so the ability for them to have native mail & calendaring is pretty epic. The Outlook Web App in Exchange 2010 is phenomenal. I mean, it almost brings a tear to my eye, it’s so beautiful – especially when compared with 2003. And being able to do server-side searching in OWA & on my iPhone is fabulous.

All is not perfect, though. I keep getting stupid certificate errors for Autodiscover when I open Outlook 2007. I guess I’ll need to buy another SSL certificate and dedicate another IP to this service… ugh. And now that I moved my mailbox to Exchange 2010, Outlook Anywhere appears not to work. Oh well, almost there…

vCenter: Error parsing the server “(server IP)” “clients.xml” file

I got the above error today after running Windows Update on my XP VM a few days ago. A quick search showed that the error is caused by a Microsoft update to the .NET framework. To resolve it, remove update KB980773 (Add/Remove programs, make sure “Show Updates” is checked; KB980773 is under “Microsoft .NET Framework 2.0 Service Pack 2”). I removed it and was able to log in without problems.

References:

Edit 10/22/2010: You can also resolve this by upgrading your vCenter client to 4.1, which I recently did. 4.1 is available on vmware.com.

vCenter: Error parsing the server "(server IP)" "clients.xml" file

I got the above error today after running Windows Update on my XP VM a few days ago. A quick search showed that the error is caused by a Microsoft update to the .NET framework. To resolve it, remove update KB980773 (Add/Remove programs, make sure “Show Updates” is checked; KB980773 is under “Microsoft .NET Framework 2.0 Service Pack 2”). I removed it and was able to log in without problems.

References:

Edit 10/22/2010: You can also resolve this by upgrading your vCenter client to 4.1, which I recently did. 4.1 is available on vmware.com.

Windows XP guests hang on shutdown in VMware Workstation on Linux

I had this problem with FC11, where I couldn’t properly shut down or suspend the Windows XP VM I run (mostly for Outlook). When I’d shutdown inside the guest, it would mostly do what you’d expect, but then the screen would go blue, then black, and then the only way I could get the thing to exit was to kill the pid of vmware-vmx (or a ‘killall vmware-vmx’). I solved it somehow by removing some RPM, but when I went to FC12 the other day it came back, and I couldn’t remember how I’d fixed it initially. I Googled for nearly a full day before I found the original blog post that told me which RPMs to remove – for some reason I thought it was fprintd, but it was something else completely. Here’s the original post, and when I just read it I was going to comment on it to say thanks for posting it, but apparently I did that already. Anyway, I removed pcsc-lite again and everything appears to be good now.

Moving an Exchange 2003 server to another location with minimal risk and disruption?

So our Exchange server is located in our office building. This made sense at the time because that’s where the users are. Over time though, this has proved problematic for a few reasons. Primarily, our office is certainly not a datacenter and doesn’t offer the amenities of one – clean, reliable power, and redundant cooling. In an average year we lose power probably 10-15 times, often for an hour or more. The rest of our production environment is hosted in a top-tier datacenter, so after a while I started to wonder why our Exchange server wasn’t there, and making plans to move it there. Oh, and did I mention I’m not an Exchange admin in any sense of the term? I just inherited the Exchange server about 2 months ago.

Continue reading “Moving an Exchange 2003 server to another location with minimal risk and disruption?”

More fun with ovftool

I tried restoring my .ova into vCenter and it failed. It said something like “Device ‘virtual disk’ uses a controller that is not supported. this is a general limitation of the virtual hardware version.” This kind of pissed me off, especially when I couldn’t really find anything in Google about what the error meant, but it turned out to be pretty easy to fix – I just clicked “upgrade virtual hardware” in VMWare Server to upgrade the hardware version from 4 to 7. I then created another .ova, uploaded it to the datacenter and successfully deployed it in vCenter in its own resource pool. Yay!

I think the real problem was that some of our VMs were created with the BusLogic driver rather than the LSI Logic driver (which is recommended for Linux guests).

Ovftool also turned out to be a pretty sweet way to backup VMs. Once in .ova format you can use ovftool to convert back to .vmx just by specifying the .ova file as the source and something .vmx as the target. Because it so nicely shrinks the .vmdk (my 20 gig VM became a 2.5 gig .ova) and is pretty quick to run (seemed like < 10 minutes per 20-gig VM) it seems like a decent choice for backing up VMs.

VMWare rules

I love VMWare. I’ve been using it for less than a year but it’s easily one of my favorite products. I upgraded my ESX 3.5 cluster to 4.0 a little over a week ago and it was a pretty simple procedure – the vCenter server pushes out the new ISO to the ESX server, installs it, and reboots it. I haven’t really played with most of the 4.0 features yet, but the new performance overview graphs are awesome.

I did have one problem where the graphs weren’t displaying – it was giving me some kind of IE error page. After a lot of messing around with it I realized that the Tomcat server that serves those graphs wasn’t running – netstat -a showed nothing listening on 8443. My first guess was a port conflict of some sort. Nothing was listening on port 8443, but lots of other 8000+ ports were in LISTEN, so I looked in Services and realized I had VMware Converter server running. Once I stopped that I was able to start the VMWare Web Services and the graphs showed properly – at least when viewed from localhost. When I tried to view from a remote computer there was still an error. This turned out to be a dumb DNS issue; the graphs referenced the SDK via the URL “https://winxp2:8443/sdk&#8221; (btw, this is defined in a few different files named extension.xml in C:\Program Files\VMware\Infrastructure\VirtualCenter Server\extensions\*, I’m not sure which subdirectory links to which webservice so I’d make sure to change it everywhere if you change it anywhere.) The problem was that over the VPN I didn’t have the proper search domain setup (winxp2.company.com) so I ended up resolving this problem by editing the hosts file on my desktop to point to the proper IP. Kind of ghetto but it works. VMWare is very finicky about DNS stuff; it’s kind of scary how dependent on DNS everything is. But oh well. For the benefits it provides it’s a small price to pay. I could go on about it but I’m sure a VMWare sales rep could do it better. 🙂