vCenter Upgrade Troubleshooting

This article will guide you through various vcenter upgrade troubleshooting steps for windows based vcenter.

  1. You may see below error while upgrading virtual center

2.  To solve this,

a. Ensure tasks mentioned in point 5  Pre-check  are taken care.    Anti-virus / backup software may block VMware DLL / process and                  upgrade will fail.

b. Try the upgrade one more time, if it fails once again then download the “Process explorer” from Microsoft web site

Process Explorer

      c. Check if DLLs mentioned in below KBs are blocked by any process using Process explorer


If you find any process using DLL mentioned in above  KB,  kill those process and retry the upgrade.

 3. If it upgrade fails then follow steps 5 to 8 as mentioned in above VMware KB.   (Please ensure snapshot is taken before trying this               step   and all vcenter related services are stopped to delete the files mentioned in steps 5 to 8)

4. You may get different errors, like

Installation of component VCSServiceManager failed with error code ‘1603’. Check the logs for more details

Follow KB-    or

vCenter upgrade step by step

This article provides vCenter upgrade detailed procedure for windows based virtual center.

Please refer Pre-Check section for validation which needs to be done before upgrade

  1. Upgrade of vcenter / SSO Server (if external)

Login to SSO with service account if applicable (As mentioned in Pre-  check section)

Note – Reboot the server if its not rebooted since long time.

Go to My Computer –> CD Drive –> on Auto run –> Run as administrator

Accept the license agreement

2. Follow the same procedure for virtual center.

Login to virtual center with service account (if applicable) –> go to cd-rom and run as administrator


Here you might face some issue, please check Issue

3.   Login to Update manager server via RDP with service account

Select “Server” under vSphere update manager server and follow the required steps

Enter the SSO user password

This does not require reboot, but it would be great if you reboot the server.

Hope this helps, happy reading.  Feel free to leave your comment or suggestions.

vcenter upgrade pre-check

As a professional service team member in DellEMC, i work on upgrade / designing of virtual environment on daily basis for different customers. I am sharing the steps  which i follow while upgrading virtual environment and issues which i faced. This is first post of this series which will guide you to do vcenter upgrade pre-check before upgrade

Part 1 –  Pre-Check or data collection before starting vSphere upgrade

  1. Ensure, we have all the credentials in place like

            a. SSO User password

    b.Service account password (If Virtual Center or VUM Service is running with service account

    c.Data base user password

    d. AD Domain admin credentials (to remove / re-join machine in domain)

      2.  Check the target virtual center hardware and software requirement.

vSphere 6  # for virtual center 6

vSphere 6.5

      3. Check the SQL Server requirement for target virtual center.

DB Compatibility

     4.     Check the compatibility of additional plugin / software integrated with source virtual center to target virtual center

          e.g. To run NSX with vSphere 6.5, it should be above 6.3 or above version.


5.  Take backup of SSL certificate. We may need to restore if custom certificates are used

     6. For windows based virtual center, please ensure

   a. Remove anti-virus from server (You can re-install it after upgrade) and if removal is not allowed or possible then please ensure it is          disabled till upgrade is complete.

   b. Remove any backup agent software from Server

   c. If Virtual Center is part of OU in AD with security policy applied, then remove it from OU and move it other OU in AD where security policy are not applied. You can move it back original OU once upgrade is complete.

  d. Reboot the server once above software /settings are removed.

           Note – I would highly  recommend to work on Point 4 because (point a-c) may block VMware service related DLL files and our upgrade                        may fail. This is also recommended by VMware.


7.   Login to virtual center via RDP

a. Check the host name and free space on server


 Important – Please ensure we have sufficient space for upgrade.  For major upgrade like 5.5 to 6.0 I suggest min 30 GB free should be                               available  and for minor  upgrade 10 GB is fine.

b.  Open service.msc and check the virtual center and its related service status

Note – If virtual center service is configured to run with service account then note down account name. We have to login to virtual                                   center via RDP with this account to start upgrade.

c. Go to Administrative Tool and open ODBC (64 bit)


d. Check the advance setting of virtual center to note down SSO server if its embedded or external  (If its external, upgrade the SSO Server first)


e. Take backup of Inventory service database # this might come handy if you are upgrading from 5.1 to 5.5 or 5.5 to 6.0

                  On the source virtual center machine, open the command prompt in the vCenter Server and change the directory to                                  vCenter_Server_installation_directory\Infrastructure\Inventory Service\scripts

                  Run the following command at the prompt to back up the Inventory Service database.

                  backup.bat -file backup_file_name

                  When the backup operation finishes, the message Backup completed successfully appears.


8.   Login to VUM / Update manager via RDP

   a. Check the free space on server and check host name to ensure we logged in correct server / correct host name is set.

       Note – 10 GB free space is fine for major / minor upgrade

b. Open Service.msc and check VUM service status

Note – If VUM Service is running with service account, note down the name and login with service account to start the upgrade.


c. Check ODBC Connection (32 bit)


9.   Login to Database server via RDP

     a. Check the disk space on server and host name to ensure we logged in to correct server

      Note – Please ensure we have sufficient free space on server for smooth upgrade process.

    b. Check the MS SQL service status in service.msc

    c. Take backup of virtual center and VUM data base backup before Upgrade

    d. Open SQL Management studio with either

            e. on vcenter datbase –> Task –> backup

           f. Select Disk under destination

            g. Select the drive where you want to back up the data base. Give the file name “vcenter.bak”  as shown below

e.g. Here I have selected “I” drive and created “RCM Upgrade” folder

         h.  Repeat the same procedure for VUM data base backup


Related Article –     Part 2- vCenter upgrade steps

DellEMC Vision

DellEMC Vision (Formally known as VCE Vision)

What is DellEMC Vision Tool

DellEMC Vision software provides health and lifecycle management capabilities embedded in VCE SystemsVision software provides standardized, repeatable processes that enable a system administrator to ensure an optimized, secure, and functional data center or hybrid-cloud environment.

Vision software enables you to know about one or more VCE Systems in a data center, where they are located, and what components they contain. It reports on the health or operating status of all VCE Systems. Use Vision software to rapidly verify that VCE Systems are running the currently supported software or firmware for a particular Release Certification Matrix (RCM). Additionally, you can use Vision software to maintain security hardened infrastructure in accordance with baseline security settings.

Vision software effectively acts as a mediation layer between your VCE System and its existing management tools. The software allows for intelligent discovery by providing a continuous, near real-time perspective of the compute, network, storage, and virtualization resources as a single object, ensuring that the management tools reflect the most current state of the VCE System.

Product Architecture

Each VCE System has a Vision CoreMSM, and an MSP VM deployed and configured as part of the installation process

Data Center Architecture-

Vision software allows you to set up a multisystem configuration within your data center environment.

Vision software provides support for multiple VCE Systems. In a data center environment, an MSM VM can be associated with up to two Vision Core VMs:

Important:Only one MSP VM must be deployed and configured within a cluster. The MSP VM must be associated with the MSM VMs in its local data center for that cluster. If other MSP VMs have been deployed and configured, you must power down these VMs.

Vision software provides support for a clustered environment that includes multiple MSM VMs. These MSM VMs can be configured to run in a single physical data center, or in multiple physical data centers that are geographically separate.

The following illustration shows a single-site environment consisting of three MSM VMs, each of which is associated with a single Vision Core VM. The MSM VMs are configured to form a cluster. The MSM capabilities and functionality are exposed after the deployment and configuration of the MSM VMs. Use the Vision dashboard, the Vision shell, or REST APIs to interact and manage VCE Systems:

In a single-site configuration with one data center, Vision software supports up to three MSM VMs running within the data center. Each MSM VM can be associated with up to two Vision Core VMs.

Vision software also provides support for a multi-site clustering configuration that includes a maximum of three data centers. In a multi-site configuration with three data centers, each data center can have no more than two MSM VMs running, where each MSM VM is associated with up to two Vision Core VMs.

For Complete Instruction and guidelines for clustering, refer to DellEMC Vision Administrator Guide

DellEMC vblock discovery not working in Vision?

    DellEMC Vision, vBlock Discovery failed….??

What is DellEMC Vision,  Well it is a monitoring tool with a capability to check vBlock release compliance matrix (RCM) and Security hardening status for each vblock with baseline security settings. DellEMC Vision periodically discovers Vblock components and one of the ways to check if it is running OK is to check when was the last successful discovery, sometimes vblock discovery does not work or fails.

For Detailed introduction about DellEMC Vision – Please visit  DellEMC Vision

Most common issue we face in or after deploying Vision is, vBlock discovery  fails,   please follow below steps to troubleshoot 

Possible Reason – 1

  1.  Restart of VCE Vision Core didn’t worked.  Login to VCE vision Core VM using putty

          All services are up and running:








[root@vision ~]# cd /etc/init.d/

[root@vision init.d]# ./vce-fm-all-services status

vce-fm-master is running (pid=11093 )

vce-fm-adapter is running (pid=11599 )

vce-fm-agent is running (pid=11854 )

vce-fm-net-snmpd (pid=11349 ) is running

vce-fm-naaagent is running (pid=12101 )

    2. Go to logs folder to see if discovery is happening or not




[root@vision init.d]# cd /opt/vce/fm/logs/

[root@vision logs]# grep Performance FMAgent.log |grep "discoverAll and all"

[root@vision logs]#

Nothing in recent logs…

   3. Let’s check previous logs to find the last successful discovery:

[root@vision logs]# grep Performance FMAgent.log* |grep "discoverAll and all"

FMAgent.log.2018-03-16.4: INFO  2018-03-16 09:54:52,574 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(1592ms) totalMem:1053 freeMem:667

FMAgent.log.2018-03-16.4: INFO  2018-03-16 10:04:46,407 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(1216ms) totalMem:1053 freeMem:667

FMAgent.log.2018-03-16.4: INFO  2018-03-16 10:09:49,039 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(1439ms) totalMem:1053 freeMem:648

FMAgent.log.2018-03-16.4: INFO  2018-03-16 10:19:47,603 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(1473ms) totalMem:1053 freeMem:667

FMAgent.log.2018-03-16.4: INFO  2018-03-16 10:24:49,372 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(1245ms) totalMem:1053 freeMem:647

FMAgent.log.2018-03-16.4: INFO  2018-03-16 10:34:39,326 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(1458ms) totalMem:1053 freeMem:666

FMAgent.log.2018-03-16.4: INFO  2018-03-16 10:39:36,794 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(1598ms) totalMem:1053 freeMem:682

[root@vision logs]#


Okk.. so last successful discovery was on March 16 2018.


Lets dig in dipper for more information

[root@vision logs]# tail -f FMAgent.log

ERROR 2018-03-16 13:43:14,721 [main] [startVblocksDiscoveryService] : scheduler.VblocksDiscoveryService - EULA has not been accepted. discovery/health poll is disabled

ERROR 2018-03-16 13:43:14,721 [main] [startVblocksDiscoveryService] : scheduler.VblocksDiscoveryService - Issue startEulaAcceptance to begin EULA acceptance

INFO  2018-03-16 13:43:14,722 [main] [logEvent] : event.FMEventServiceFactory - publish event: id=EulaEULA has not been accepted. You must accept End User License Agreement to enable discovery/health poll.


So, the issue was Eula wasn't accepted. Last time we added new Chassis, then Eula wasn't accepted.


Note- Whenever any component added / removed or re-installed from/ in vBlock, then Eula needs to be accepted in VCE Vision Core VM to start discovery



4. We can follow below steps to accept Eula


[root@vision logs]startEulaAcceptance

Starting EULA acceptance: Wed Mar 16 13:50:18 UTC 2018

Enter Name> Mark Strong

Enter Title> Master of the Universe

Enter Licensee Company Name – vipal

Enter Accepting Company Name – vipal

Do you accept the End User License Agreement:[yes/no]- yes

Softcopy of End User License Agreement File created in: /opt/vce/fm/eula/VCE_EULA_V2_2.txt

Wed Mar 16 13:50:58 UTC 2015 EULA has been accepted.

5. VCE Vision Can now discover vBlock

[root@vision logs]# grep Performance FMAgent.log |grep "discoverAll and all"


INFO  2018-03-16 15:54:49,895 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(3992ms) totalMem:158 freeMem:95


INFO  2018-03-16 15:55:08,445 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(3512ms) totalMem:163 freeMem:115


INFO  2018-03-16 16:16:59,372 [main] [update2Listeners] : scheduler.VblocksDiscoveryService - Performance:update2Listeners finished with discoverAll and all(551ms) totalMem:234 freeMem:161

6. Is discovery fails again,

you can run "StopFMAgent" and "StartFMagent" command. Then the vBlock discovery will start as described in point 5.

[root@vision init.d]# stopFMagent


Disabling FMagent services from starting automatically...

Waiting for the FMagent services to stop.

vce-fm-master is not running

vce-fm-adapter is not running

vce-fm-agent is not running

vce-fm-net-snmpd is not running

vce-fm-naaagent is not running

[root@vision init.d]# startFMagent

Enabling FMagent services to start automatically…

Waiting for the FMagent services to start.......................................

vce-fm-master is running (pid=10205 )

vce-fm-adapter is running (pid=10724 )

vce-fm-agent is running (pid=10983 )

vce-fm-net-snmpd (pid=10466 ) is running

vce-fm-naaagent is running (pid=11284 )


Possible Reason 2

  1. Check NTP is configured on Vision Core server or not. If not, configure it
  2. Check NTP is configured on Vision MSM Server or not, if not please configure it. # You will get option to configure NTP while deploying MSM VM.
  3. Check if Core server is properly added in MSM. If not, run below command in MSM   /opt/vce/multivbmgmt/install/
  4. Make Sure, DNS entry is properly configured  for both MSM and Core VM (forward and reverse lookup)

Hope this helps…. happy reading 🙂