Blog post born from a VMUG Presentation
Mid Feb, one of the London VMUG leaders posted on twitter, looking for someone to present on the subject of “upgrading from vSphere 5.5 to vSphere 6.5”.
— Simon Gallagher (@vinf_net) February 15, 2018
So I jumped at the chance, kind of, and offered to present. This blog post covers the content from that presentation.
- vSphere 5.5 – End of Support
- vSphere 6.5 – New features
- OK, so let’s just upgrade then?
- The plugin’s
- SSO is gone!
- Understand your topologies
- Pre-Upgrade Tasks
- The Upgrade, the big event
- VSAN Considerations
- vShield Manager is no more! Upgrade to NSX Manager
The presentation is available to download here – http://vexpert.me/London-vmug-dean (case sensitive link)
Or I’ve figured out how to embed it from Slideshare.net below (But animations don’t seem to work);
vSphere 5.5 – End of Support
- End of General Support for vSphere 5.5 is September 19, 2018
- Includes vCenter 5.5, ESXi 5.5, VSAN 5.5
- KB 51491
- In the event you are unable to upgrade before the End of General Support (EOGS) and are active on Support and Subscription, you have the option to purchase extended support in one year increments for up to two years beyond the EOGS date.
- Expect this to be more costly than general support.
- SLA’s are more akin to that of basic support rather than production support
- Annual security patch. Includes catastrophic/critical security fixes only
- Ability to create hot patches for Severity 1 issues only
- Technical Guidance for vSphere 5.5 is available until September 19, 2020 primarily through the self-help portal.
- During the Technical Guidance phase, VMware does not offer new hardware support, server/client/guest OS updates, new security patches or bug fixes unless otherwise noted.
- For example, there was no SPECTRE/Meltdown security patches released for vSphere 5.1
It’s not only the core vSphere 5.5 products that are affected, as we can see from the End-of-Support tracking page provided by virten.net. There are other VMware solutions that you have deployed that may also need upgrading.
vSphere 6.5 – New Features
Ok, so its presumed you are going to migrate straight to the latest version, so let’s do a re-cap of the latest features. Or you can find the official VMware 6.5 Whats new PDF here.
We still have two offerings in which to install vCenter.
- vCenter 6.5 for Windows
- Windows 2008 R2 or above
- Embedded PostgreSQL – 20 Hosts/200 VMs
- External DB for larger (SQL/Oracle)
- Penultimate release – vSphere.Next will be the last version to feature vCenter for Windows
- vCenter 6.5 Appliance (VCSA)
- PhotonOS – developed/Maintained by VMware. Open source, optimized for VMware infrastructure components, VMware control the full lifecycle
- PostgreSQL database that has the scalability of up to 2,000 hosts and 35,000 virtual machines
- vCenter High Availability – When vCenter HA is enabled, a three-node vCenter Server cluster (Active, Passive, and Witness nodes) is deployed.
- vCenter Server uses the VMware vSphere Update Manager Extension service
Below is the architecture diagram of vCenter High Availability (source)
- VMFS 6
- 512e and 4K SSDs/NVMe drives support
- Automatic unmap – no more scripts to reclaim space off your arrays.
- Requires creation from new (KB2147824)
- There is a PowerCLI command – Update-VmfsDatastore
- Description: This operation deletes the existing VMFS5 datastore to create a VMFS6 datastore. You should back up any files from the VMFS5 datastore to prevent any data loss.
- VM level encryption
- Requires a 3rd party external key manager solution
- Common REST API Framework
- will feature between all VMware solutions going forward to make scripting/automation easier and more consistant
- Predictive DRS
- Ties into your vRealize Ops Mgr deployment
- Using trend data, will ensure your cluster resources are allocated to best suit the trends
- Example; Think of your payroll server, it idles for most of the month. Then last day of the month it runs at full resource utilization whilst it churns through the payment runs for employees.
- Pro-Active HA
- Detects hardware conditions of the ESXi host , allowing ability evacuate the Virtual machines before the hardware issues cause an outage
- HTML5 web client built in – updates released into vCenter patches
Finally, the big one, no more Windows vSphere client – its gone.
OK, so lets just upgrade then?
From the official document, here is the overview of the upgrade process.
Upgrading vSphere includes the following tasks:
- Read the vSphere release notes.
- Verify that you have backed up your configuration.
- If your vSphere system includes VMware solutions or plug-ins, verify that they are compatible with the vCenter Server or vCenter Server Appliance version to which you are upgrading. See VMware Product Interoperability Matrix
- Upgrade vCenter Server. See Overview of the vCenter Server Upgrade Process.
- If you are using vSphere Update Manager, upgrade it. Refer to the VMware vSphere Update Manager documentation.
- Upgrade your ESXi hosts. See Overview of the ESXi Host Upgrade Process.
- To ensure sufficient disk storage for log files, consider setting up a syslog server for remote logging. Setting up logging on a remote host is especially important for hosts with limited local storage. See Required Free Space for System Logging and Configure Syslog on ESXi Hosts.
- Upgrade your VMs and virtual appliances, manually or by using vSphere Update Manager, to perform an orchestrated upgrade. See Upgrading Virtual Machines and VMware Tools.
All pretty simple and straight forward, abiet a lot of links? But this particular document only hints at some of the biggest challenges to come! The plugins to vCenter, or connecting VMware Solutions, which will need upgrading before your core vSphere environment, or in parallel.
It’s also interesting that at this point, VMware also specifically call out setting up additional logging in the environment as part of the upgrade process. Maybe the reports from GSS are they still don’t see a lot of customers with appropiate logging setup.
- If your vSphere system includes VMware solutions or plug-ins, verify that they are compatible with the vCenter Server or vCenter Server Appliance version to which you are upgrading
- You need to ensure that any VMware based products or 3rd party applications that tie into vCenter, support the latest versions
- If not, upgrade them first!
- Check VMware KB2147289
On this KB page, there is a list of the VMware Solutions which need to be considered during a vSphere 6.5 upgrade.
And on the same KB is the upgrade sequence, with a possible 8 products that need upgrading before vCenter!
SSO is gone
- Single Sign On has now been migrated into the Platform Services Controller (PSC)
- PSC deals with identity management for administrators and applications that interact with the vSphere platform.
- Post from VMware with various resources around changes from SSO to PSC (KB 2109560)
- No longer the option to distribute vSphere components such as Web Client, Inventory Service, vCenter, to different servers.
- Some of the services which are held in the PSC
- VMware License Service VMware Component Manager
- VMware Security Token Service
- VMware Common Logging Service
- VMware Syslog Health Service
- VMware Authentication Framework
- VMware Certificate Service
- VMware Directory Service
- PSC deals with identity management for administrators and applications that interact with the vSphere platform.
- If you need to consolidate your SSO Domains, do this in 5.5 before the upgrade (KB 2033620)
The below diagram shows the architectural changes when consolidating your SSO domains. (Source)
- Enhanced Linked Mode connects multiple vCenter Server systems together by using one or more Platform Services Controllers.
- Enhanced Linked Mode lets you view and search across all linked vCenter Server systems and replicate roles, permissions, licenses, policies, and tags.
- Replaces vCenter Linked Mode (which needs to be uninstalled before the migration)
Understand your Topologies
- With the replacement of SSO for PSC, the topologies also change
- During the migration or upgrade process a mixed environment is supported.
- There is no time frame in which to complete your upgrade when in mixed mode
- Advice from VMware is to do this as soon as possible
- One single SSO domain
- Multiple SSO sites if needed
The first topology is a simple embedded install, with vCenter + PSC installed on the same Server.
Second, is a external PSC deployment, allowing two vCenter servers to be connected. This will also enable the Enhanced Linked Mode. This diagram also demonstrates the mixed mode deployment, with both vCenter 5.5 and vCenter 6.5 connecting the PSC.
Below we have a highly available PSC deployment, which requires a load balancer to be deployed. This allows the vCenter servers to connect to the load balanced domain name for HA purposes.
The final diagram shows an example of the deprecated topologies, such as two connecting two embedded deployments together, or connecting a single vCenter server to an embedded deployment.
Rule of thumb, if you have multple vCenters you need to be part of the same deployment, you need an External PSC.
- Health check the environment
- Check all hardware status
- Check software services (vSphere Web Client, SSO)
- Backup status (Ensure good)
- VMware infrastructure (correct current patch levels, status of VMs, Logging enabled)
- Check your hardware – Compute, Storage, Networking, IO Cards against the VMware Compatibility List
- Be aware you can change the search options as per the below.
- Read the vSphere upgrade documentation
- Read the upgrade documentation for any VMware Solutions, plugins, 3rd party apps
- Think; Trend Micro Deep Security, Veeam, ArcServe, Turbonomic, OpsVizor, vRealize, NSX, VSAN.
- Ensure you have your infrastructure details and pre-req configurations in place
- DNS (including any records configured), NTP, Network settings (IP, Subnet, Gateway), Firewall Rules
- New VMs/Appliances may be deployed during the upgrade, hence the need to have your networking details on hand.
- Build a Checklist
- Build a table for your software and versions you will upgrade to, include notes/web links
- Detail the upgrade sequence for the various components that need upgrading (order your check list table in this seq.)
Below is an example of such table;
- Any support tickets (internal or with external support) that are open and cover hardware/services which interact with your VMware platform, ensure they are resolved first before the upgrade.
- Ensure that you have a backup prior to starting the upgrade process, along with a recovery plan in case you need to revert back.
- Open a support request with VMware Support prior to starting your upgrade process—it will expedite the process should any issues come up.
Below is a screenshot of VMware confirming the details they would need for a proactive ticket in regards to upgrading NSX.
- Plan your time accordingly.
- Estimating the time for migration of vCenter Server 5.5 or 6.0 to vCenter Server Appliance 6.5 (KB 2147711)
- Read the vSphere upgrade documentation (this isn’t a typo, I put it in twice for a reason!)
- Includes details to help you build your checklists
- Get familiar with the upgrade process – Check out independent blog posts to see screenshots of the process and such.
The Upgrade, the big event
- Migrate or Upgrade?
- Deploying PSC and vCenter as distributed components
- Straight upgrade of VCSA (actually deploys a new one) or Windows vCenter
- During upgrade, good idea to stop all vCenter based actions either manually, scripted or automated (including via 3rd party software interaction, i.e backups)
- During the upgrade whilst vCenter is unavailable
- vSphere Distributed Resource Scheduler (DRS) does not work while the upgrade is in progress.
- vSphere HA does work during the upgrade
- Before you run the vCenter 6.5 ISO – vCenter database tips
- Monitor scheduled database jobs to ensure they are running correctly – For more information, refer to KB article: Checking the status of vCenter Server performance rollup jobs (2012226)
- Collect Stats Rebuild Indexes – For more information, refer to KB article: Rebuilding indexes to improve the performance of SQL Server and Oracle vCenter Server databases (2009918)
- Delete old data – For more information, refer to KB article: Reducing the size of the vCenter Server database when the rollup scripts take a long time to run (1007453)
- On the vCenter 6.5 ISO – Locate the cleanup_orphaned_data_MSSQL.sql script in the ISO image and copy it to the Microsoft SQL server.
- Similar for PostgreSQL and Oracle
- If using custom SSL certificates, back them up before hand
- Check network connectivity on machine you run the upgrade/migration from
- DNS lookup and Reverse Lookup
- Basic ping connectivity
- Upgrade the components as per your checklist and sequencing
- Check each component after each upgrade, i.e log into web services, check health status
- Upgrade your 3rd party products first, likely they will support vSphere 5.5 and 6.5 at the same time.
- Bring your ESXi hosts up to 6.5, remember to use the vendor customized ISO’s
- Upgrade VMware Tools – remember this is now updated separately to ESXi releases
- Upgrade virtual hardware version
- Exposes new CPUID features to the VMs (think Spectre/Meltdown patches)
- Ability to use new features such as VM encryption
- Upgrade your licensing in https://my.vmware.com
- Install your new licenses – you will currently be on evaluation licenses (60 days)
- How to upgrade your licences in the VMware portal (KB 2006974)
- ESXi 5.1 and older cannot be managed by vCenter 6.5
- Devices deprecated and unsupported in ESXi 6.5 (KB 2145810)
- Number of qlogic and Emulex devices that are no longer supported
- End of Availability and End of Support for FCoE on Intel Network Controllers (KB 2147786)
- Microsoft SQL Server Express is not supported for vCenter Server 6.5. The vCenter Server 5.5 embedded Microsoft SQL Server Express database is replaced with an embedded PostgreSQL database during the upgrade to vCenter Server 6.5.
- Update manager is now built into the vCenter Appliance – if you have trouble migrating update manager, uninstall it from your windows machine and don’t migrate it.
- HP Server – ESXi upgrade to 6.5
- Run the validation first, HP customized installed may have conflicting Mellanox drivers
- Remove using esxcli
Below image shows the conflicting VIBs in the HPE Customized image.
- vSphere 6.5 no longer supports the following processors:
- Intel Xeon 51xx series
- Intel Xeon 30xx series
- Intel core 2 duo 6xxx series
- Intel Xeon 32xx series
- Intel core 2 quad 6xxx series
- Intel Xeon 53xx series
- Intel Xeon 72xx/73xx series
- VMware is announcing discontinuation of its third party virtual switch (vSwitch) program, and plans to deprecate the VMware vSphere APIs used by third party switches in the release following vSphere 6.5 Update 1 (KB 2149722)
- Upgrading the VCSA 6.5 to a later version
- Login to the vCenter Server appliance 6.5 VAMI page using the URL (https://vCenter-appliance-name:5480) and login with the root credentials
- vCenter Servers must be upgraded to vSphere 6.5 Update 1 before vSAN hosts are upgraded to vSAN 6.6 or vSAN 6.6.1
- Verify that you have backed up your virtual machines
- Verify that the software and hardware components, drivers, firmware, and storage I/O controllers that you plan on using are supported by VMware vSAN for 6.6 and later, and are listed on the VMware vSAN Compatibility Guide website
- Verify that you are using the latest patched version of VMware vSAN prior to upgrade.
Please review VMware KB 2146381
The above screenshot shows that to upgrade to VSAN 6.6.1, you need to be on ESXi 5.5 Express Patch 7, which sits between 5.5 u2 and 5.5 u3. This particular patch fixes a bug of vSAN data availability after an upgrade to vSphere 6.0 or higher.
vShield Manager is no more! Upgrade to NSX Manager
- vCloud Network security & vShield Manager are now end of life (19th September 2016), and is not supported by vCenter 6.5
- Migrate vShield Manager 5.5 to NSX Manager 6.2 (KB 2144620)
- 6.2 is End of General Support – 20th August 2018
- Upgrade NSX Manager to a supported version by vCenter 6.5
- Best practices for upgrading to vCenter Server 6.5 (KB 2147686)
- The presentation this blog post is based on – http://vexpert.me/London-vmug-dean
- VMware 6.5 upgrade eBook – http://emadyounis.com/upgrade/new-resource-vmware-vsphere-6-5-upgrade-ebook/
- VMware StorageHub – https://Storagehub.vmware.com
- vSphere Central – https://vspherecentral.vmware.com