Article Content

Issue


VxRail final pre-check failed due to internal errors

During a full VxRail appliance shutdown, all of the vcenter actions were timing out. Disabling HA, migrating the service vms, etc. There was an option to retry and it would move on to the next action, which would time out.

Marvin.log
2018-09-13T22:21:52:702Z [ERROR] [MessageBroker-2] o.s.s.s.TaskUtils$LoggingErrorHandler:95 - Unexpected error occurred in scheduled task.
2018-09-13T22:34:22:028Z [ERROR] [localhost-startStop-1] c.v.m.c.s.m.PlatformInformationService:252 - Unable to determine platform type: Failed to find primary host
2018-09-13T22:34:23:254Z [ERROR] [localhost-startStop-1] c.v.m.c.s.m.PlatformInformationService:252 - Unable to determine platform type: Failed to find primary host
2018-09-13T22:34:23:492Z [ERROR] [localhost-startStop-1] c.v.m.c.s.m.PlatformInformationService:252 - Unable to determine platform type: Failed to find primary host

Connectors-cluster.log
Sep 13 22:31:19,167 GMT 2018 ERROR [AMQP Connection 127.0.0.1:5672] CachingConnectionFactory$DefaultChannelCloseLogger.log:1089 - Channel shutdown: connection error
Sep 13 22:31:19,167 GMT 2018 ERROR [AMQP Connection 127.0.0.1:5672] CachingConnectionFactory$DefaultChannelCloseLogger.log:1089 - Channel shutdown: connection error
Sep 13 22:31:19,167 GMT 2018 ERROR [AMQP Connection 127.0.0.1:5672] CachingConnectionFactory$DefaultChannelCloseLogger.log:1089 - Channel shutdown: connection error

Sep 13 23:13:00,858 GMT 2018  ERROR [SimpleAsyncTaskExecutor-1] PrecheckMQListener.check:66 - com.emc.mystic.manager.commons.emc.precheck.exception.PrecheckException

Sep 14 00:17:12,499 GMT 2018  ERROR [pool-51-thread-1] ServiceUtil.waitForTask:466 - Message The operation is not allowed in the current state.
Sep 14 00:17:12,499 GMT 2018 ERROR [pool-51-thread-1] VimOperator.migrateVMs:2016 - Migrating VM fails: vm-14
Sep 14 00:17:12,499 GMT 2018 ERROR [pool-51-thread-1] VimOperator.migrateVMs:2020 - Migrate VM failed:com.vmware.vim25.ManagedObjectReference@ 36891641
java.lang.RuntimeException: Migrating VM fails: vm-14
at com.emc.mystic.manager.commons.vmware.VimOperator.migrateVMs(VimOperator.java:2017) [mystic.manager.commons.vmware-4.0.401.jar:?]
at com.emc.mystic.manager.connectors.cluster.ShutdownStartupServiceImpl$1.run(ShutdownStartupServiceImpl.java:162) [mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Sep 14 00:17:12,500 GMT 2018 ERROR [pool-51-thread-1] ShutdownStartupServiceImpl$1.run:197 - Shutdown error
com.emc.mystic.manager.connectors.cluster.ClusterShutdownException: java.lang.RuntimeException: Migrating VM fails: vm-14
at com.emc.mystic.manager.connectors.cluster.ClusterShutdownException.getClusterShutDownException(ClusterShutdownException.java:56) ~[mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
at com.emc.mystic.manager.connectors.cluster.ShutdownStartupServiceImpl$1.run(ShutdownStartupServiceImpl.java:164) [mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.lang.RuntimeException: Migrating VM fails: vm-14
at com.emc.mystic.manager.commons.vmware.VimOperator.migrateVMs(VimOperator.java:2017) ~[mystic.manager.commons.vmware-4.0.401.jar:?]
at com.emc.mystic.manager.connectors.cluster.ShutdownStartupServiceImpl$1.run(ShutdownStartupServiceImpl.java:162) ~[mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
 ... 5 more

Cause

vCenter is unable to function due to a lack of DNS. In this customer's environment the DNS VMs were on the VxRail appliance as normal VMs.
 
Resolution
Workaround: Power one of the DNS server VMs on, and retry you will then need to shut it down when all other VMs are shutdown.  

Alternatively, point the vCenter appliance to include a DNS server not on the VxRail.
 
Notes

Recommendations: VxRail vCenter appliance should have it's DNS pointing to a server not hosted on the vxrail cluster, otherwise the DNS VM should be one of the last VMs to be shutdown and should be placed on the last ESXi server to be shutdown.

Customer also came across this issue with the Log Insight VM still running.
https://support.emc.com/kb/503715
Sep 13 23:46:57,625 GMT 2018 ERROR [pool-49-thread-1] ServiceUtil.waitForTask:466 - Message A general system error occurred: PBM error occurred during PreMigrateCheckCallback: Operation timed out
Sep 13 23:46:57,625 GMT 2018 ERROR [pool-49-thread-1] VimOperator.migrateVMs:2016 - Migrating VM fails: vm-17
Sep 13 23:46:57,625 GMT 2018 ERROR [pool-49-thread-1] VimOperator.migrateVMs:2020 - Migrate VM failed : com.vmware.vim25.ManagedObjectReference6f7a5a92
java.lang.RuntimeException: Migrating VM fails: vm-17
at com.emc.mystic.manager.commons.vmware.VimOperator.migrateVMs(VimOperator.java:2017) [mystic.manager.commons.vmware-4.0.401.jar:?]
at com.emc.mystic.manager.connectors.cluster.ShutdownStartupServiceImpl$1.run(ShutdownStartupServiceImpl.java:162) [mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Sep 13 23:46:57,626 GMT 2018 ERROR [pool-49-thread-1] ShutdownStartupServiceImpl$1.run:197 - Shutdown error
com.emc.mystic.manager.connectors.cluster.ClusterShutdownException: java.lang.RuntimeException: Migrating VM fails: vm-17
at com.emc.mystic.manager.connectors.cluster.ClusterShutdownException.getClusterShutDownException(ClusterShutdownException.java:56) ~[mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
at com.emc.mystic.manager.connectors.cluster.ShutdownStartupServiceImpl$1.run(ShutdownStartupServiceImpl.java:164) [mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.lang.RuntimeException: Migrating VM fails: vm-17
at com.emc.mystic.manager.commons.vmware.VimOperator.migrateVMs(VimOperator.java:2017) ~[mystic.manager.commons.vmware-4.0.401.jar:?]
at com.emc.mystic.manager.connectors.cluster.ShutdownStartupServiceImpl$1.run(ShutdownStartupServiceImpl.java:162) ~[mystic.manager.connectors.cluster-4.0.401-lib.jar:?]
 ... 5 more