OVA deployment from ESXi local datastore

I wrote one basic script for  OVA deployment from ESXi hosts local data store. This idea came to my mind after working on one project where i had to deploy OVAs on around 250 ESXi clusters.

Here i used VMware OVFTool and Selenium webdriver. You can Download  both software from the links mentioned below. Import-Vapp command from powercli doesn’t support HTTPS source for OVA deployment from ESXi local data store. You can check its help section for more information.

Import-v App script can be found here OVA Deployment via PowerCLI

OVFTOOL Download

Selenium

Explanation Line by Line-

1 – Path to OVFtool exe.

3-5 –  I have added Selenium binary or DLL using Add-type method.

7 – Path to chrome driver to open ESXi local data store webpage and fetch the HTTPS link

8 -9  – We would need ESXi root credentials here

11- Enter the v Center details

13 – Location of the OVA which needs to be uploaded to local data store

15 – Folder which we will create inside local data store to store OVA

17-30 – Code to upload OVA on local data store of first ESXi inside Cluster

30 -36 – v Switch and port group information to create VMKernel on ESXi hosts.

42-52 – Collecting IP setting for the OVAs

54 – 60 – Collect Names for the OVAs

63 – 68 – Port group and local data store information.

72-83 –  Use of Selenium and Chrome driver to access ESXi local datastore and fetch HTTPS source path

86 – 115 –  Commands to deploy OVA on ESXi hosts with OVFTOOL and VMKernel creation commands.

Hope this helps..!!  Please change the script based on your requirements. Please do share your thoughts.

How to use plink to run command from windows

Recently i had to run some commands on 250 linux servers in one of my project. So i decided to use plink.exe from windows machine. This section will help you how to use plink to run linux command from windows on multiple server

plink.exe without sudo privilege 

plink.exe -ssh -t -pw <password> -l <user>@<host>  -m <script> <IP Address of host>

syntax –
-m – create filename called “script.txt” or the name you want and place it under folder where plink.exe is located
-pw – password of remote host
-l – user id to login to remote host
ip address of host
-t – it will open pseudo shell

lets say script.txt include,
hostname
pwd
ls /tmp

Above command will show output on your local windows machines cmd.exe or powershell

if the host key was not cached in the registry yet, you would get an interactive prompt saying “Store key in cache? (y/n)” which would not be conducive to a batch script.
And once again, the way around this is to pipe a “y” to stdin, which makes the command:
cmd.exe /c echo y | plink.exe -ssh -t -pw myP4ss -l myuser@trusty1 -m script.txt <IP Address of host?


plink and sudo with password prompt

In the above example we dealt with simple commands that required no input and no privilege escalation with sudo (hostname, pwd).  But many times the commands we want to run in batch processes require sudo, which throws up an interactive prompt for a password.

Sudo password echoed and embedded inside plink script

Make sure you call plink with “-t” so that a tty is setup.  Then sudo to a shell and echo the password into stdin before running the command you want with sudo.

Or you can echo the password directly to the command you want to run as sudo

Run the command on number of hosts using powershell

First you need to create one text file with list of remote server IPS or FQDN (if DNS is configured properly) in the text file and place it under same folder where plink.exe is located

get-content .\linux_Server.txt | foreach {invoke-command -command {& echo y | .\plink.exe $_ -l <username> -pw <password> -m <file with list of command>; write-host -fore “$_ host finished”}}

This command will ssh in to each host under .\linux_Server.txt and execute set of commands defined in <file with list of command>

Hope this helps… feel free to comment with  suggestion,

PowerCLI Script to assign IP on each host in ESX Cluster

Recently i came across a situation where i had to create and assign IP to vmkernel portgroup (for vmotion) for 3300 ESX host. So i wrote one basic powercli script to assign IP on each host in ESX Cluster.

============================================================

PowerCLI Script to create VMKernel portgroup and assign IP on each host in cluster 

#  Note – Replace respective name wherever applicable.  Below sample is for one cluster, you can customize for all cluster in vcenter.

$vmhost = get-cluster 0000 | get-vmhost | sort Name

$pg = $vmhost[0] | get-virtualportgroup | where {$_.name -like ‘vSphere*’}

$subnetprefic=’xxxx’     # e.g. ‘192.168.10’
$ipperserver = 1
$startip = 4

foreach ($esx in $vmhost){

0..$($ipperserver-1)|%{
$ipaddress = “$subnetprefix.$($startip + $_)”
$startip += $ipperserver

new-vmhostnetworkadapter -vmhost $esx -virtualswitch $vdswitch -ip $ipaddress -subnet $subnet -portgroup $pg
}
}

Hope this short script helps.. Happy reading!!

OVA Deployment via PowerCLI

Recently i was working on project where i had to deploy RP4VM OVA on more than 400 ESXi hosts. So i decided to write article for  OVA deployment via PowerCLI

Initially i faced a lot of issue like  Deployment was stuck during ‘validation’, i couldn’t run simultaneous deployment in vcenter because of existing vcenter issue etc.

We used naming convention like VxRAIL(DC name)_vRPA1. You can change it if required.

We had to create VMKernel port on esxi hosts for other use, this is also included in last section of the script.

Here is the powercli code which i wrote, explained line by line

1 –  It will load all PowerCLI module if you want to run script from PowerShell

3- Enter the vCenter name and username, password

5 – Please mention path to ova

 7 – Please mention cluster name where you want to deploy OVA

11 – 19 – We used naming convention e.g . VxRail(Clustername)_vRPA1.

You can change as per your need.

23-26 – IP Settings for VMKernel port on ESXi hosts

29-51 – IP setting for OVA

54-81 –  Loop for OVA deployment on ESXi host in cluster

85-99 – Port group and switch info  for VMkernel assignment on ESXi.

Newer version of script available here

Hope this helps… feel free to comment !!

vCenter 6.5 upgrade issue and troubleshooting

This post will give you an overview of vcenter 6.5 upgrade issue and its troubleshooting steps.

Issues Encountered during 6.0 to 6.5 Upgrade

Issue1:- Invalid Appliance (OS) root password

Description: – While Upgrading PSC appliance we need to provide OS *root* password and it was failing with invalid password.

Resolution1: – Reset PSC OS *root* password through web-console and try again.

  • Login to PSC URL <https://PSC_FQDN or IP Address:5480>
  • Go to Administration tab
  • Change Password

Most Important – I have noticed this issue while upgrading or migrating from vCenter 6.5 U3D to 6.5.  For other release i didn’t faced this issue.

Issue2:- VUM Extension blocking Migration

Description: – If VUM Extension is not in use un-register VUM from the vCenter server.

Resolution:- Follow this KB Article https://kb.vmware.com/s/article/1025360

  • Power- Off VUM server 

Issue3:- Unable to detect Network at Step 8

Description: – If vCenter server Management N/W is configured in Distributed Switch and if we provide ESXi host details at step 3 & 4 we are unable to select N/W option at Step8.

Because, “When ESXi is used as the deployment target, non-ephemeral distributed virtual port groups are not supported and hence, not shown drop-down list.

Resolution:- At Step 3 provide – Target Host details & Select Host, At Step4 provide vCenter server details for Appliance deployment.

Note:- If we provide At setep3 & step4 both vCenter details it will fails to migrate data.

Issue4:- AD users unable to access after PSC upgrade, In disjoint domain namespace the domain users might fail to authenticate after you update to vSphere 6.5

Description:- After PSC 6.0 to 6.5 Upgrade AD users unable to access vCenter server.

Resolution: – Delete AD with integrated windows authentication and Add again

Note:- To perform below activity requires AD rights.

  • Remove PSC’s from AD Object
  • Login to PSC <https://PSC_FQDN or IP Address/PSC>
  • Login with “administrator@vsphere.local”
  • Go to Appliance Settings –> Manage and Rejoin Server to AD Domain.
  • Enterprise AD Domain admin credentials requires from customer.
  • Reboot PSC’s.
  • Validate vCenter authentication with AD credentials.

Issue 5:- Re-Register VUM with vCenter server

Description:- If VUM extension error triggers then follow this procedure to Re-Register VUM with vCenter server.

Resolution: – Follow below steps

  • Login to VUM VM
  • Navigate to C:\Program Files (x86)\VMware\Infrastructure\Update Manager
  • Find ‘ VMwareUpdateManagerUtility.exe’, double click on it
  • Login to the update manager Utility using previously used SSO credentials.
  • Navigate to Re-register to vCenter Server –> enter the SSO credentials and new vCenter IP/FQDN and click Apply. This will re-register VUM to newly deployed VC
  • Once done, close the wizard, and restart the VMware vSphere update Manager Service
  • Login to vCenter and click on plug-ins and enable the VUM plug-in
  • Once you enable ‘update manager’ will appear in vCenter

Issue 6:- Error could not reach the given source vCenter server on “Port 22”

Description: – If SSH is disabled on PSC’s/Appliance vCenter server then this error triggers.

Resolution: – Enable SSH on PSC/Appliance vCenter prior to upgrade.

  • Login to PSC/Appliance vCenter through Web with port 5480
  • Go to Access
  • Click on Edit Option
  • Enable SSH

Issue 7 – Windows Migration to 6.5 appliance fails 

Description – If source database size is large, then migration pre-check will fail with below error

Resolution – Please check the source database, which table is utilizing more space. Once you find it, either shrink it or compress.  

Note – Please refer VMware KB 51589 here.

Reconfigure PSC from embedded to External

Re-configure PSC from Embedded to External

This article will guide you steps to, Reconfigure PSC from embedded to External. To achieve this, first of all you need to install new PSC (it could be windows based or appliance based) in same SSO domain.

Scenario – VMware vCenter 6.0 U3D is setup with embedded PSC.

Go to Cd-rom –> open vcsa-setup.html and choose install

Provide the required information. You can give vcenter name or ESXi name on which this PSC needs to be deployed.

Select the cluster where PSC needs to be deployed

Give the name of PSC and setup the password

 

Select the existing  SSO Domain name.

Most important – Similarly you can verify SSO site name and domain by below method , if you have multiple SSO site and domains then this steps will help you to cross check correct SSO site.

For VCSA appliance

/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name –server-name localhost

/usr/lib/vmware-vmafd/bin/vmafd-cli get-site-name –server-name localhost

Please refer David blog’s for more info SSO Site Name and Domain check

For windows embedded PSC

C:\ProgramData\VMware\vCenterServer\cfg\install-defaults\vmdir.domain-name

“C:\Program Files\VMware\vCenter Server\vmafdd\vmafd-cli.exe” get-ls-location –server-name localhost

Note – Installation path might very based on your environment setup

Please refer William Lam Blogs for More info SSO Site Name and Domain check

Select existing SSO Site name

 

Select the datastore where PSC  needs to be deployed

Enter the appropriate network settings and click finish to start installation.

Please go to the below path

“vcenter installation directory\bin” folder and run below command to see PSC replication partner

vdcrepadmin.exe -f showservers -h localhost -u administrator -w “password”

Note – User should be SSO user.

Here you can see, new PSC is configured as partner with embedded PSC now.

Lets cross check if replication is happening between Embedded PSC and new PSC or not.

 

Step-3 Now we come to the most important steps, to reconfigure PSC from embedded to external

Go to below path

“vcenter installation directory\bin” and run below command

cmsso-util reconfigure –repoint-psc “New PSC FQDN or IP” –username administrator -passwd “password” -domain-name “sso domain name”

Lets verify if new vcenter is pointing to new PSC or not.

Go to advanced setting of vcenter server

Under config.vpxd.sso.sts.uri  — you will see new PSC name

Or go to below path to verify

        VCSA:

         /usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location –server-name localhost

        Windows VC:

        C:\Program Files\VMware\vCenter Server\vmafdd\vmafd-cli get-ls-location –server-name localhost

       Hope this helps, feel free to comment  🙂

 

vCenter Upgrade Troubleshooting

This article will guide you through various vcenter upgrade troubleshooting steps for windows based vcenter.

  1. You may see below error while upgrading virtual center

2.  To solve this,

a. Ensure tasks mentioned in point 5  Pre-check  are taken care.    Anti-virus / backup software may block VMware DLL / process and                  upgrade will fail.

b. Try the upgrade one more time, if it fails once again then download the “Process explorer” from Microsoft web site

Process Explorer

      c. Check if DLLs mentioned in below KBs are blocked by any process using Process explorer

https://kb.vmware.com/s/article/2149266

                                                                                       

If you find any process using DLL mentioned in above  KB,  kill those process and retry the upgrade.

 3. If it upgrade fails then follow steps 5 to 8 as mentioned in above VMware KB.   (Please ensure snapshot is taken before trying this               step   and all vcenter related services are stopped to delete the files mentioned in steps 5 to 8)

4. You may get different errors, like

Installation of component VCSServiceManager failed with error code ‘1603’. Check the logs for more details

Follow KB- https://kb.vmware.com/s/article/2119768    or  https://kb.vmware.com/s/article/2127284

vCenter upgrade step by step

This article provides vCenter upgrade detailed procedure for windows based virtual center.

Please refer Pre-Check section for validation which needs to be done before upgrade

  1. Upgrade of vcenter / SSO Server (if external)

Login to SSO with service account if applicable (As mentioned in Pre-  check section)

Note – Reboot the server if its not rebooted since long time.

Go to My Computer –> CD Drive –> R.click on Auto run –> Run as administrator

Accept the license agreement

2. Follow the same procedure for virtual center.

Login to virtual center with service account (if applicable) –> go to cd-rom and run as administrator

  

Here you might face some issue, please check Issue

3.   Login to Update manager server via RDP with service account

Select “Server” under vSphere update manager server and follow the required steps

Enter the SSO user password

This does not require reboot, but it would be great if you reboot the server.

Hope this helps, happy reading.  Feel free to leave your comment or suggestions.

vcenter upgrade pre-check

As a professional service team member in DellEMC, i work on upgrade / designing of virtual environment on daily basis for different customers. I am sharing the steps  which i follow while upgrading virtual environment and issues which i faced. This is first post of this series which will guide you to do vcenter upgrade pre-check before upgrade

Part 1 –  Pre-Check or data collection before starting vSphere upgrade

  1. Ensure, we have all the credentials in place like

            a. SSO User password

    b.Service account password (If Virtual Center or VUM Service is running with service account

    c.Data base user password

    d. AD Domain admin credentials (to remove / re-join machine in domain)

      2.  Check the target virtual center hardware and software requirement.

vSphere 6  # for virtual center 6

vSphere 6.5

      3. Check the SQL Server requirement for target virtual center.

DB Compatibility

     4.     Check the compatibility of additional plugin / software integrated with source virtual center to target virtual center

          e.g. To run NSX with vSphere 6.5, it should be above 6.3 or above version.

 

5.  Take backup of SSL certificate. We may need to restore if custom certificates are used

     6. For windows based virtual center, please ensure

   a. Remove anti-virus from server (You can re-install it after upgrade) and if removal is not allowed or possible then please ensure it is          disabled till upgrade is complete.

   b. Remove any backup agent software from Server

   c. If Virtual Center is part of OU in AD with security policy applied, then remove it from OU and move it other OU in AD where security policy are not applied. You can move it back original OU once upgrade is complete.

  d. Reboot the server once above software /settings are removed.

           Note – I would highly  recommend to work on Point 4 because (point a-c) may block VMware service related DLL files and our upgrade                        may fail. This is also recommended by VMware.

 

7.   Login to virtual center via RDP

a. Check the host name and free space on server

 

 Important – Please ensure we have sufficient space for upgrade.  For major upgrade like 5.5 to 6.0 I suggest min 30 GB free should be                               available  and for minor  upgrade 10 GB is fine.

b.  Open service.msc and check the virtual center and its related service status

Note – If virtual center service is configured to run with service account then note down account name. We have to login to virtual                                   center via RDP with this account to start upgrade.

c. Go to Administrative Tool and open ODBC (64 bit)

 

d. Check the advance setting of virtual center to note down SSO server if its embedded or external  (If its external, upgrade the SSO Server first)

 

e. Take backup of Inventory service database # this might come handy if you are upgrading from 5.1 to 5.5 or 5.5 to 6.0

                  On the source virtual center machine, open the command prompt in the vCenter Server and change the directory to                                  vCenter_Server_installation_directory\Infrastructure\Inventory Service\scripts

                  Run the following command at the prompt to back up the Inventory Service database.

                  backup.bat -file backup_file_name

                  When the backup operation finishes, the message Backup completed successfully appears.

 

8.   Login to VUM / Update manager via RDP

   a. Check the free space on server and check host name to ensure we logged in correct server / correct host name is set.

       Note – 10 GB free space is fine for major / minor upgrade

b. Open Service.msc and check VUM service status

Note – If VUM Service is running with service account, note down the name and login with service account to start the upgrade.

 

c. Check ODBC Connection (32 bit)

 

9.   Login to Database server via RDP

     a. Check the disk space on server and host name to ensure we logged in to correct server

      Note – Please ensure we have sufficient free space on server for smooth upgrade process.

    b. Check the MS SQL service status in service.msc

    c. Take backup of virtual center and VUM data base backup before Upgrade

    d. Open SQL Management studio with either

            e.   R.click on vcenter datbase –> Task –> backup

           f. Select Disk under destination

            g. Select the drive where you want to back up the data base. Give the file name “vcenter.bak”  as shown below

e.g. Here I have selected “I” drive and created “RCM Upgrade” folder

         h.  Repeat the same procedure for VUM data base backup

 

Related Article –     Part 2- vCenter upgrade steps