Troubleshooting bogus vCenter icons in VMware vSphere 5

VMware vSphere 5 introduced a set of virtual machine (VM) icons to help virtualisation admins identify the different types of objects in their vCenter inventory. Although that’s good news, a bug in vSphere 5 appears to be causing these icons to issue bogus alarms.

One could dismiss the vCenter icon problem as a mere annoyance. After all, what really matters to administrators is that the platform is stable and works. But for many vSphere 5 admins, these false alarms are a painful distraction from real problems, and many are eagerly anticipating a new release of vSphere that fixes the issue.

Varieties of bogus vCenter icons
Here are two main situations when the vCenter icon problem manifests itself.

Scenario 1: Bogus vCenter icons most commonly appear after a catastrophic error such as total storage or power failure that has brought down an entire cluster of ESX hosts. I’ve witnessed it in my own lab environment, as have others.

Figure 1: Many VMs on the left have a red exclamation mark suggesting serious performance issues.

As you can see in Figure 1, many of the VMs have red exclamation marks next to them, which is usually indicative of some serious performance problem. In fact, these alarms are bogus.

Upon further investigation, administrators will not see any alarms listed under the “Alarms” tab. The red exclamation mark persists even if you shut down the VM and power it on again.

Scenario 2: VMware Icons and Site Recovery Manager

The bogus vCenter icons also appear in the context of VMware Site Recovery Manager (SRM). SRM uses special icons to indicate that a VM is protected or replicated using vSphere replication. Protected VMs are represented by an icon consisting of a box within a box and a lightning strike, as shown in Figure 2 (below).

Figure 2: the “lightning” and the “boxes-in-boxes” symbol denotes protected VMs.

VCenter represents VMs protected by vSphere Replication by a box and arrow icon as in Figure 3 (below).

Figure 3: The vSphere Replication servers appear with the “box with an arrow” symbol.

Occasionally, the VR management service icon can be lost altogether. The VRMS icon loses the arrow icon, and just gets the standard boxes-in-boxes icon. It’s not catastrophic -- the service still works.

What’s a bit more worrying is the “lightning strike” icon, which represents a placeholder object used by SRM when building a recovery plan. It’s not a real VM. The problem was first brought to my attention by Rob Plankers, another virtualisation administrator. He carried out a failover and failback with SRM, but the recovered VM still had the “lightning flash” icon, when it should have had the ordinary “boxes-in-boxes” icon.

Working together we discovered that this bogus placeholder would power on and boot an OS (something that should never happen). A placeholder VM isn’t supposed to power on. But this one did indicating it was a real VM with the wrong icon. Plankers unregistered and re-registered the VM, and the correct icon then appeared.

Occasionally, I’ve seen instances where an entirely different icon appears instead of the “lightning flash” icon typically found on a placeholder VM in SRM 5.0. These incorrect icons take the form of the classic “boxes-in-boxes” icon with the “Solutions Manager” icon from the home page of vCenter (as seen in Figure 4).


Figure 4: The “boxes-in-boxes” icon with the “Solutions Manager” icon

These icons should actually display with the "lightening strike".

Despite the funky icon, these are genuine placeholders and they work as expected whenever you run a recovery plan. This is a client error. If you crank up the vSphere Client from another management station, then the icons can be fine. That sounds like there is some sort of client cache of icons that is becoming broken or corrupted.

Removing false vCenter icons
The bogus vCenter icons appear to be a known issue internally at VMware, even though there’s no public VMware Knowledge Base article about it at the moment. This issue seems to be caused by a problem in the register and unregister process, when a VM with one type of icon is removed from vCenter (unregistered) and then added back in with its new icon (registered).

I’ve heard stories of customers escalating this into VMware support – and a variety of solutions to resolve it have been circulated. Here’s a brief list of what other users have done to resolve the issue:

1. Reload the vSphere Client
2. vMotion the affected VMs
3. Storage vMotion the VMs
4. License and then relicense the vSphere 5 product
5. Place the ESX hosts into maintenance mode, and then exit maintenance mode – this triggers a bulk unregister and re-register process
6. Restart the vCenter service
7. Restart the ESX host management services
8. Register and unregister afflicted VMs

I’ve had the most success resolving this issue by restarting the ESX host management services (Step number 7 from the above list). But sometimes that hasn’t proved sufficient, and I’ve been forced to manually unregister and register the afflicted VMs. As virtualisation admins know, this process can only be done when the VM is powered off, which can be troublesome if the affected VMs are running production applications such as Microsoft SQL Server or a virtualised vCenter instance.

The latter example presents a catch-22: In order to do the re-register, you must power off the very systems that allow you to carry out the process. In one case I was forced to access the ESX hosts directly (bypassing vCenter altogether) to perform the registration process.

VMware has issues to resolve here, and users hope to see this addressed in the VMware vSphere 5.1 release. The message at the moment is not to panic or over-react. If something seems odd, then it probably is just a graphical issue rather than a genuine error.
 

Mike Laverick is a former VMware instructor with 17 years of experience in technologies such as Novell, Windows, Citrix and VMware. Since 2003, he has been involved with the VMware community. Laverick is a VMware forum moderator and member of the London VMware User Group. He is also the man behind the virtualisation website and blog RTFM Education, where he publishes free guides and utilities for VMware customers. Laverick received the VMware vExpert award in 2009, 2010 and 2011. He also finds the time to run a weekly podcast called the Chinwag and the Vendorwag. He helped found the Irish and Scottish VMware user groups and now speaks regularly at larger regional events organised by the global VMware user group in North America, EMEA and APAC. Laverick published books on VMware Virtual Infrastructure 3, vSphere4, Site Recovery Manager and View.