“Call HostServiceSystem.Start for object XX on vCenter Server YY failed” when starting snmpd on ESXi 5.1

SSH into the host and run these:

~ # esxcli system snmp set --communities 
~ # esxcli system snmp set --enable true

After that, make sure you have the port (161/udp) open in the firewall config and it should start up ok. (The root problem is the file /etc/vmware/snmp.xml – those commands fix it).

Advertisements

Use PowerShell to disconnect the CDROMs from all VMs in vCenter

Recently I was moving all VMs from one NFS datastore to another so we could destroy the old volume. Storage vMotion took care of this for the most part, but even after moving the files, vCenter still showed that the VMs were using the old datastore. It turned out this was due to the VMs having mounted ISOs on that datastore. The solution was to eject/unmount the CDrom, but I didn’t want to do “Edit Settings…” and manually remove the CDrom for 200+ VMs.

I found this page which shows how to do almost exactly what I wanted via PowerShell. We’re using the vCenter Server Appliance though, which runs Linux, so I wasn’t sure how to run PowerShell stuff. But that was solved easily enough by installing VMWare’s PowerCLI package on my Windows desktop and connecting to the vCenter remotely:

PS C:usersevandocuments> Connect-VIServer vcenter.example.com

PS C:usersevandocuments> Get-Datacenter -name "Primary Datacenter" | Get-VM | Get-CDDrive | Set-CDDrive -Connected $false -NoMedia


Confirm
Are you sure you want to perform this action?
Performing operation "Setting Connected: False, NoMedia: True." on Target "CD/DVD drive 1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

Connect-VIServer pops up a username/password box where you put in your creds. The next command disconnects the cdrom for every VM in the “Primary Datacenter” datacenter in vCenter. I haven’t tried it but I think you can use different containers to scope the command however you want (cluster, resource pool, folder, etc). See here for other “location” options.

VMWare Tools Compatibility Matrix

Question: If I install the latest version of VMWare Tools on a guest, what kind of compatibility do I get when I vMotion to an older version of ESX? In other words, is VMware Tools backwards compatible with older versions of ESX/ESXi?

Answer: Yes, it’s backwards compatible. From this tool:

vmware-tools-compat

Rescan SATA bus (aka hot-adding a SATA disk on a Linux guest in VMware without rebooting)

Linux supports hot-adding disks but whenever I add a new vdisk in VMware the new disk doesn’t show up unless I reboot, which defeats the purpose of hot-add. This command forces a rescan of the bus:

echo "- - -" > /sys/class/scsi_host/host0/scan

dmesg shows the new disk has been found:

  Vendor: VMware    Model: Virtual disk      Rev: 1.0 
  Type:   Direct-Access                      ANSI SCSI revision: 02
 target0:0:2: Beginning Domain Validation
 target0:0:2: Domain Validation skipping write tests
 target0:0:2: Ending Domain Validation
 target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
 sdd: unknown partition table
sd 0:0:2:0: Attached scsi disk sdd
sd 0:0:2:0: Attached scsi generic sg3 type 0

Now, why there’s no “rescan_sata” command is something I can’t fathom, but that’s Linux for you.

VMWare 5’s new licensing model.

After reading up on the new VMware licensing&pricing model I understand the uproar. Limiting vRAM is a reasonable constraint, but 32GB per socket for Enterprise? 48 GB for “Enterprise Plus”? If you have a dual CPU server with 144 GB (easily configurable last year), with 4.1 you’d only need 2 enterprise licenses to use all 144 GB, since in 4.1 an “Enterprise” license covered 1 CPU (up to 6 cores) and up to 256 GB memory on the host.

But with 5.0 you’ll either have to buy 5 Enterprise licenses (160 GB) or 3 Enterprise Plus licenses (144 GB) just to use the full 144 GB. I guess VMware has done away with memory overcommit as a selling point? They used to tell us it was recommended to go up to 2:1 so we could safely put ~140 GB of VMs on a 72 GB machine – with the new model that’s completely gone.

To put it in monetary terms, on the machine with 144 GB ram from above, the cost for 4.1 would be $2875 * 2 = $5750. To stick with Enterprise it would be $2875 * 5 = $14,375, or $3495 * 3 = $10,485. A gigantic price jump. I haven’t read up on any features of vSphere 5, but I don’t think any feature can make up for at minimum nearly doubling the cost, with loss of a major selling point (memory overcommit). I mean, you can still overcommit as long as you’re willing to pay for the overcommitted memory. Also, the above numbers are per-host, so if you’ve got a 5-host cluster you’re looking at a $25,000 price hike.

VMWare has long been one of my favorite products, but this is making me consider alternatives. Almost 100% of the feedback I’ve read about this change has been negative. Seems like a huge mistake on VMware’s part.

VMware Workstation 7 – virtual ethernet fails to start after changing vmnet8 subnet

After my recent wipe of my laptop, I reinstalled VMware Workstation and my Win XP VM was working fine. The one wrinkle I faced was that the subnet for the vmnet8 (NAT) vnic had changed from 192.168.250.0/24 to 173.16.132.0/24. The host machine had been 192.168.250.1 so rather than reconfiguring everything on the guest to point to a new IP for the host I figured it would be easier to change the subnet for vmnet8. I went into the Virtual Network Editor and just changed the subnet. Seemed to work correctly, but after doing a release/renew in Win XP I couldn’t get an IP.

I tried disconnecting the vnic and reconnecting it; the guest recognized that the “cable was unplugged,” but still couldn’t get an IP. I rebooted the guest – same thing. Restarted the vmware service and saw this:

[root@ehoffman ~]# /etc/init.d/vmware restart
Stopping VMware services:
   VMware USB Arbitrator                                   [  OK  ]
   VM communication interface socket family                [  OK  ]
   Virtual machine communication interface                 [  OK  ]
   Virtual machine monitor                                 [  OK  ]
   Blocking file system                                    [  OK  ]
Starting VMware services:
   VMware USB Arbitrator                                   [  OK  ]
   Virtual machine monitor                                 [  OK  ]
   Virtual machine communication interface                 [  OK  ]
   VM communication interface socket family                [  OK  ]
   Blocking file system                                    [  OK  ]
   Virtual ethernet                                        [FAILED]
[root@ehoffman ~]#

That’s weird. The vnic is up with the specified IP:

[root@ehoffman vmnet8]# ifconfig
vmnet1    Link encap:Ethernet  HWaddr 00:50:56:C0:00:01
          inet addr:172.16.3.1  Bcast:172.16.3.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vmnet8    Link encap:Ethernet  HWaddr 00:50:56:C0:00:08
          inet addr:192.168.250.1  Bcast:192.168.250.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Checked the /var/log/vnetlib logfile and it appears to be some problem starting DHCP on vmnet8:

Internet Software Consortium DHCP Server 2.0
Copyright 1995, 1996, 1997, 1998, 1999 The Internet Software Consortium.
All rights reserved.

Please contribute if you find this software useful.
For info, please visit http://www.isc.org/dhcp-contrib.html

Configured subnet: 172.16.3.0
Setting vmnet-dhcp IP address: 172.16.3.254
Opened: /dev/vmnet1
Recving on     VNet/vmnet1/172.16.3.0
Sending on     VNet/vmnet1/172.16.3.0
Internet Software Consortium DHCP Server 2.0
Copyright 1995, 1996, 1997, 1998, 1999 The Internet Software Consortium.
All rights reserved.

Please contribute if you find this software useful.
For info, please visit http://www.isc.org/dhcp-contrib.html

Address range 192.168.250.128 to 192.168.250.254 not on net 192.168.250.1/255.255.255.0!
exiting.
Failed to start DHCP service on vmnet8
Failed to start some/all services
Feb 23 09:28:46 VNL_Load - LOG_ERR logged
Feb 23 09:28:46 VNL_Load - LOG_WRN logged
Feb 23 09:28:46 VNL_Load - LOG_OK logged
Feb 23 09:28:46 VNL_Load - Successfully initialized Vnetlib
Feb 23 09:28:46 VNL_StartService - Started "Bridge" service for vnet: vmnet0
Feb 23 09:28:47 VNL_CheckSubnetAvailability - Subnet: 172.16.3.0 on vnet: vmnet1 is available
Feb 23 09:28:47 VNL_CheckSubnetAvailability - Subnet: 192.168.250.0 on vnet: vmnet8 is available
Feb 23 09:28:47 VNL_StartService - Started "DHCP" service for vnet: vmnet1
Feb 23 09:28:47 VNL_EnableNetworkAdapter - Successfully enabled hostonly adapter on vnet: vmnet1
Feb 23 09:28:47 VNLServiceStart - Daemon process did not report status, returning failure
Feb 23 09:28:47 VNL_StartService - Failure in starting "DHCP" service for vnet: vmnet8
Feb 23 09:28:47 VNL_StartService - Started "NAT" service for vnet: vmnet8
Feb 23 09:28:47 VNL_EnableNetworkAdapter - Successfully enabled hostonly adapter on vnet: vmnet8
Feb 23 09:28:47 VNLServiceStatus - pid: 13512 for Netdetect service daemon on vnet: 0 is stale
Feb 23 09:28:47 VNL_StartService - Started "Netdetect" service for vnet: vmnet0
Feb 23 09:28:47 VNL_Unload - Vnetlib unloaded.
Started Bridge networking on vmnet0
Started DHCP service on vmnet1
Enabled hostonly virtual adapter on vmnet1
Started NAT service on vmnet8
Enabled hostonly virtual adapter on vmnet8
Started Network detection service

The important line there is Address range 192.168.250.128 to 192.168.250.254 not on net 192.168.250.1/255.255.255.0! Apparently DHCP is misconfigured. The config file for dhcpd for vmnet8 is /etc/vmware/vmnet8/dhcpd/dhcpd.conf . Here’s what it looked like:

subnet 192.168.250.1 netmask 255.255.255.0 {
        range 192.168.250.128 192.168.250.250;
        option broadcast-address 192.168.250.255;
        option domain-name-servers 192.168.250.1;
        option domain-name localdomain;
        default-lease-time 1800;                # default is 30 minutes
        max-lease-time 7200;                    # default is 2 hours
        option routers 192.168.250.2;
}

I changed the range to 192.168.250.2 to 192.168.250.127, thinking that was the problem, but it turned out to be the “subnet” line – the subnet should be “192.168.250.0 netmask 255.255.255.0” rather than “192.168.250.1 …” After changing that, everything Worked As Intended.

Upgrading ESX 4.0 to ESX 4.1 with Update Manager

This turned out to be so ridiculously easy I probably shouldn’t even mention it, but only because I remembered that Update Manager exists. I thought I was going to have to burn an ISO, go to the datacenter, pop the disc in, etc. But with Update Manager this is all handled remotely. Just download the files from vmware.com and upload them to your vCenter server through the client. This YouTube video by VMware walks you through the procedure:

Continue reading “Upgrading ESX 4.0 to ESX 4.1 with Update Manager”