The issues below will prevent VMs from being migrated to other available Hosts. PowerChute does not log migration errors for individual VMs in the Event Log but the error.log file contains additional information.
When performing manual VM Migration PowerChute uses the MigrateVM_Task API function. When there are errors migrating VMs the following appears in the error.log file:
VM Migration Error Message -> [Error message]
For detailed information:
https://www.vmware.com/support/developer/vc-sdk/ - Click on vSphere API Reference, Expand vSphere API Reference and go to All Methods->MigrateVM_Task. The faults section provides a table outlining the various errors that can occur when attempting to migrate VMs.
To troubleshoot individual VMs the logging level can be set to “warn” instead of “error”, see section on debug logging.
The events below are logged to the Event Log if some or all VMs do not successfully migrate when there are target hosts available:
“Unable to migrate all VMs from Host [hostname]”
“Unable to migrate any VMs from Host [hostname]”
The events below are logged to the error.log file:
“[VM] failed to migrate to Host [Host]”
“VMware Error message [Fault message]”
Examples of errors that can occur include:
InsufficientResourcesFault: Thrown if this operation would violate a resource usage policy.
MigrationFault: Thrown if it is not possible to migrate the virtual machine to the destination host. This is typically due to hosts being incompatible, such as mismatch in network polices or access to networks and datastores.
VmConfigFault: Thrown if the virtual machine is not compatible with the destination host. Typically, a specific subclass of this exception is thrown, such as IDEDiskNotSupported.
Note: The errors above and descriptions are taken from the vSphere API Reference Guide :
https://www.vmware.com/support/developer/vc-sdk/
This will occur if the other Hosts are in Maintenance mode, incompatible or have a UPS critical event active.
The event below is written to the PowerChute Event Log:
“Unable to find a suitable host to migrate VMs from Host: [Host]"
The event below is written to the PowerChute Event Log:
“Cannot connect to vCenter Server. PowerChute will not be able to perform VM migration."
The error below is written to the PowerChute Event Log:
"Insufficient time to migrate all VMs using the VM Migration Duration configured."
The error below is written to the PowerChute error.log:
"Insufficient time to migrate all VMs using VM Migration Duration"
if vCLS shutdown is skipped, the error below is written to the PowerChute event log. This error can occur if the user account used to connect to vCenter Server is not a member of the administrators group. In this case, PowerChute will be unable to find the vCLS VMs.
"Skipping vCLS shutdown. vCLS may already be disabled or it was not possible to find any running vCLS VMs. Please ensure that the user account used to connect to vCenter Server is a member of the administrators group, otherwise it may not be possible for PowerChute to find the vCLS VMs."
The errors below are written to the PowerChute Event Log:
“Cannot connect to vCenter Server. PowerChute may not be able to issue commands to Virtual Machines or Hosts.”
“vCenter Server authentication error. PowerChute may not be able to issue commands to Virtual Machines or Hosts.”
When vCenter Server Connection is not available PowerChute will attempt to connect directly to each ESXi host to perform VM shutdown. This requires a shared user account (see section on creating shared user accounts) to be present on each of the ESXi hosts. If PowerChute cannot connect to the ESXi hosts the following exceptions may appear in the error.log:
com.vmware.vim25.NoPermission
com.vmware.vim25.InvalidLogin
Solution:
Verify that the vCenter Server User Account configured in PowerChute can access each of the ESXi hosts by connecting directly to each host using the vSphere client. If the connection is lost, add the user account to each ESXi host. If the connection is successful verify that the user account has “Administrator” permissions.
When PowerChute attempts to connect directly to the ESXi hosts it must be able to connect to them using their Fully Qualified Domain Name or IP Address when vCenter Server is no longer available. If there are DNS issues or Hostname resolution issues the following errors appear in the error.log file:
VI SDK invoke exception:java.net.ConnectException: Connection timed out: connect
Solution:
Verify that DNS lookups of the ESXi Host FQDN are working from the PowerChute machine using the nslookup command.
If nslookup is not successful, the ESXi hosts can be added to the /etc/hosts file on Linux (PowerChute Appliance/vMA) or C:\Windows\system32\drivers\etc\hosts file on Windows machines. If the only DNS server available is running as a VM on the ESXi hosts being protected then it is necessary to use the hosts file. Alternatively ESXi hosts can be added to vCenter Server using their IP address instead of FQDN.
The event below is written to the PowerChute Event Log:
"Insufficient time to shut down all VMs using the VM Shutdown Duration configured."
The event below is written to the PowerChute Error.log:
“Insufficient time to migrate all VMs using VM Migration Duration”
VMware tools must be installed to perform graceful Guest OS shutdown. VMware Tools status is shown in the summary tab for VMs.
If DRS is enabled and set to fully automated for the cluster, VM Migration is enabled by default. If you disable VM Migration while DRS remains fully automated and enabled, when a maintenance mode task begins on the host, DRS will start migrating VMs to other available hosts. If PowerChute begins VM shutdown on the host at the same time as the DRS migration occurs, VMs that are in migration will not successfully shut down.
If DRS is enabled and set to fully automated, VM Migration must be enabled in PowerChute with a VM migration duration set, in order to allow Virtual Machines to migrate successfully.
The event below is written to the PowerChute Event Log:
“Attempting to power on VMs on Host [hostname] that did not start.”
The event below is written to the PowerChute Error.log:
“VM [VM] Power on failed.”
When entering and exiting maintenance mode the Host’s HA configuration might experience an error and end up in an invalid state. Invalid state will be shown on the Summary tab for the Host in the vSphere client. This will prevent PowerChute from powering on VMs and there will be repeated event log entries indicating that PowerChute is attempting to re-start VMs that did not start.
Solution:
Right click on the Host and select “Reconfigure for vSphere HA”
OR
Right click on the Host, select “Disconnect” and then “Reconnect”
The following event appears in the PowerChute Event Log:
“Cannot connect to vCenter Server. PowerChute will not be able to perform vApp Shutdown.”
vApp shutdown is not supported if the vCenter Server is unavailable during the shutdown sequence.
This is required to perform Guest OS shutdown. If Guest OS shutdown is enabled for a VM in the vApp and VMware Tools are installed this will cause the vApp shutdown task to be unsuccessful.
If the shutdown action is set to Power Off instead of Guest OS Shutdown, VMs in the vApp will not be shut down gracefully.
The event below is written to the PowerChute Event Log:
"Insufficient time to shut down vApp using the vApp Shutdown Duration configured."
The event below is written to the PowerChute Error Log:
"Insufficient time to shut down vApp using the vApp Shutdown Duration"
vApp shutdown is not supported if either the PowerChute VM or vCenter Server VM are part of a vApp.
The events below are written to the PowerChute Event Log:
“vApp [vApp] will not be shut down as it contains the Virtual Machine running PowerChute. Please remove PowerChute from the vApp.”
“vApp [vApp] will not be shut down as it contains the vCenter Server VM. Please remove vCenter Server VM from the vApp."
The event below is written to the PowerChute error.log:
[vApp] vApp could not be Powered Off
This will prevent the VM from being shut down gracefully and it will also prevent PowerChute from identifying the VM as the one running vCenter Server.
This can occur if there is a DNS configuration issue i.e. the IP address/Hostname cannot be resolved.
The event below is written to the PowerChute Event Log:
“PowerChute cannot locate the vCenter Server VM in the Inventory.”
Solution:
Check that VMware tools are installed and running on the vCenter Server VM
Check that the IP address or Hostname/FQDN for vCenter Server Connection in PowerChute matches the IP/Hostname shown on the Summary page for the VM.
On the Host Protection page ensure that the Host running the vCenter Server VM is marked with the vSphere icon. Using vSphere client vMotion the vCenter Server VM to another host and verify that the Host protection page is updated to reflect this change.
The event below is written to the PowerChute Event Log:
“The vCenter Server VM found in the Inventory is powered off.”
Solution:
Delete any old copies of vCenter Server VM from the Inventory.
The event below is written to the PowerChute Event Log:
vCenter Server VM [VM] cannot be gracefully shut down. Please check vCenter Server VM Shutdown duration.
The event below is written to the PowerChute Event Log:
PowerChute cannot locate the vCenter Server VM in the Inventory. See the troubleshooting section in the Online Help.
The event below is written to the PowerChute Event Log:
The vCenter Server VM found in the Inventory is powered off. See the troubleshooting section in the online help.
e.g. Domain is set to apcc.com in vCenter, but apcc on the ESXi host. This can be seen using the vSphere Client for each Host under Configuration > DNS and Routing > Host Identification > Domain.
To avoid this ensure that the same domain name is set in vCenter Server and on the ESXi host.
The following message appears in the error.log:
"checkForVCSAVMAndHostInCriticalHosts - cannot obtain HostSystem using findByIP and findByDnsName for critical host [hostname]"
The following message appears in the Event Log:
“Shutdown Host unsuccessful for Host [Host].”
The following message appears in the Event Log if the host shutdown is unsuccessful due to "UnknownHost", "InvalidLogin" or "NoPermission" Exception:
"Shutdown Host unsuccessful for Host [host]. [Reason]. [Link to troubleshooting guide]".
The following exception appears in the error.log when attempting to perform VM actions or Host actions using the vSphere API using the unlicensed version of vSphere ESXi:
com.vmware.vim25.NoPermission
The following is logged in the Error Log:
com.vmware.vim25.NoPermission
The following exceptions appear in the Error Log:
com.vmware.vim25.InvalidLogin
com.vmware.vim25.AuthenticationError
The following exception appears in the Error Log:
VI SDK invoke exception:java.net.UnkownHostException
The following exception appears in the Error Log:
com.vmware.vim25.InvalidLogin
The following event appears in the Event Log when this occurs:
“Cannot locate the PowerChute VM in the Inventory.”
Solution:
Check that VMware tools are installed and running on the PowerChute VM.
“The PowerChute VM found in the Inventory is powered off.”
Solution:
Delete any old copies of PowerChute VM from the Inventory.
The following message appears in the EventLog if it takes too long for PowerChute to put a Host into Maintenance Mode:
“Maintenance mode task cancelled on Host [Host] as there are still powered on VMs. Please verify that sufficient time has been configured for VM/vApp/VCSA VM shutdown duration.”
The following event appears in the error.log when this occurs:
"Maintenance Mode not entered for Host [Host]"
If PowerChute is configured to start a Maintenance Mode on hosts at the start of the shutdown sequence and the connection between PowerChute and vCenter Server is lost during the VM/vApp Shutdown step, this will result in the hosts being placed into Maintenance Mode later in the shutdown sequence during the Host Shutdown step via a direct host connection.
The following messages appear in the EventLog when this occurs:
"Starting Maintenance Mode Task on Host [Host]"
"Cannot connect to vCenter Server. PowerChute may not be able to issue commands to Virtual Machines or Hosts."
The following event appears in the error.log when this occurs:
"Error checking maintenance mode for Host [Host]"
The following event appears in the error.log when this occurs:
“Host is powered off. No need for Virtualization Shutdown"
The following events may appear in the EventLog when this occurs:
"Cannot connect to vCenter Server. PowerChute may not be able to issue commands to Virtual Machines or Hosts."
"Cannot connect to vCenter Server. PowerChute will not be able to perform VM Migration."
"Cannot connect to vCenter Server. PowerChute may not be able to perform vApp Shutdown."
The following exceptions may appear in the error.log when this occurs:
VI SDK invoke exception:java.net.ConnectException: Connection timed out: connect
VI SDK invoke exception:java.net.ConnectException: Connection refused: connect
java.rmi.RemoteException: VI SDK invoke exception:java.net.SocketTimeoutException: Read timed out
java.rmi.RemoteException: VI SDK invoke exception:java.net.SocketTimeoutException: connect timed out
These issues can occur due to network latency issues.
Solution:
Increase the values for the following configurable timeout settings in the Configuration INI file:
[HostConfigSettings]
VMware_connect_timeout = 10
VMware_read_timeout = 15
For example, to increase the timeout settings to 30 seconds each, change the values to the following:
VMware_connect_timeout = 30
VMware_read_timeout = 30
The following event will appear in the EventLog when this occurs:
"Attempting to power on VMs on Host that did not start"
Solution:
Increase the values for the following configurable settings in the Configuration INI file:
delay_after_exit_maintenance_mode = 30
delay_after_vcsa_powered_on_and_connected = 30
For example, to increase the delay to 60 seconds each, change the values to the following:
delay_after_exit_maintenance_mode = 60
delay_after_vcsa_powered_on_and_connected = 60
On the Host Protection page, ensure that the VCenter root certificate is accepted. If the certificate is not accepted, no error message will be displayed and the shutdown will not be completed as it will be unable to communicate with individual ESXi host.
Solution:
Ensure you always accept VCenter Root Certificate.