VMware vSphere 8 Upgrade: 12 Lessons from Our Biggest Rollout
In March 2024, we started the largest vSphere upgrade in our company's history: 500 ESXi hosts across 8 data centers, from vSphere 7.0 to 8.0. The project took 14 weeks. It should have taken 8. Here are the 12 lessons that would have saved us six weeks of pain.
What is a vSphere 8 Upgrade?
A vSphere 8 upgrade involves updating the ESXi hypervisor, vCenter Server, and all related components (vSAN, NSX, Aria Operations) to version 8.0. It is not just a simple version bump. vSphere 8 introduces new APIs, deprecates old features, and changes how some components interact.
The upgrade process has two main paths: in-place upgrade (update existing hosts) and fresh install (install vSphere 8 on new hardware and migrate VMs). For most organizations, in-place upgrade is faster and less disruptive. But it carries more risk because you are modifying existing configurations.
vSphere 8 also introduces the vSphere Distributed Services Engine, which offloads certain operations to DPUs (Data Processing Units). This is a significant architectural change that affects how you plan hardware for future deployments.
Why This Upgrade Matters
vSphere 7 reached general availability in April 2020 and will reach end of general support in April 2025. After that, no more security patches or bug fixes. For organizations running production workloads on vSphere 7, the upgrade is not optional.
vSphere 8 brings performance improvements (up to 2x improvement in certain workloads), security enhancements (native TPM 2.0 support, encrypted vMotion), and operational improvements (improved DRS, simplified lifecycle management). The benefits are real, but so is the effort.
For our 500-host upgrade, the business driver was compliance. Our clients require supported infrastructure. Running unsupported vSphere versions is a compliance violation that could cost us contracts. The upgrade was a business necessity, not a technology choice.
Our 12 Lessons Learned
These lessons come from 14 weeks of real-world upgrade experience. They are ordered by the phase in which we learned them.
Lesson 1: Read the Release Notes Thoroughly
We skipped this step. Bad idea. vSphere 8 deprecates several features we relied on: VMware Flash Read Cache, certain vSAN disk management options, and some VDS (Virtual Distributed Switch) features. We discovered these deprecations during the upgrade, not before.
Spend a full day reading release notes, upgrade guides, and known issues. Create a spreadsheet of every feature you use and verify it exists in vSphere 8. We now do this for every major upgrade. It takes one day and saves weeks.
Lesson 2: Test in a Lab First
We tested in a small lab (3 hosts, 20 VMs) before upgrading production. The lab test revealed two critical issues. First, our backup software (Veeam) needed a patch for vSphere 8 compatibility. Second, one of our custom power scripts used deprecated APIs that failed silently.
Test every component in your environment: backup software, monitoring tools, custom scripts, third-party integrations. If it touches vSphere, test it. We now maintain a lab environment that mirrors production for exactly this purpose.
Lesson 3: Upgrade vCenter First
vCenter must be upgraded before ESXi hosts. This is not optional. vSphere 8 vCenter can manage vSphere 7 hosts, but vSphere 7 vCenter cannot manage vSphere 8 hosts. If you upgrade hosts first, they will disconnect from vCenter.
The vCenter upgrade itself takes 2-4 hours. Plan for vCenter downtime during the upgrade. All VMs continue running, but you cannot manage them through vCenter. DRS, HA, and other vCenter-dependent features pause temporarily.
We upgraded vCenter on a Friday evening. If something went wrong, we had the weekend to fix it. Plan your vCenter upgrade for a low-activity period.
Lesson 4: Check Hardware Compatibility
vSphere 8 has stricter hardware requirements than vSphere 7. Check the Hardware Compatibility List (HCL) for every component: servers, storage controllers, NICs, GPUs. We found that 15% of our hosts had storage controllers not on the vSphere 8 HCL.
For incompatible hardware, you have three options: replace the hardware, use a community driver (unsupported), or stay on vSphere 7 for those hosts. We replaced storage controllers on 75 hosts. It cost $50,000 but ensured full compatibility.
Lesson 5: Update Firmware Before OS
ESXi 8 requires specific firmware versions on server hardware. We upgraded ESXi hosts before updating firmware and hit boot failures on 20% of hosts. The fix: update BIOS, RAID controller firmware, and NIC firmware before upgrading ESXi.
We created a firmware update script using Dell iDRAC API. It updates all firmware components automatically. The script runs for 30 minutes per host. We batch updates in groups of 10 hosts.
Lesson 6: Plan for vSAN Compatibility
If you use vSAN, check vSAN compatibility with vSphere 8. vSAN 8 introduces a new storage architecture (Express Storage Architecture) that is incompatible with vSAN 7 disk groups. You can upgrade to vSAN 8 on the legacy architecture, but you cannot migrate to ESA without reformatting disks.
We chose to stay on vSAN 7 legacy architecture for the initial upgrade. We will migrate to ESA later when we replace storage hardware. Do not try to do everything at once.
Lesson 7: Handle Distributed Switch Upgrades Carefully
VDS (Virtual Distributed Switch) upgrades are tricky. vSphere 8 introduces VDS 8.0 with new features and changed behavior. Upgrading VDS is a one-way operation. You cannot downgrade.
We tested VDS upgrade in the lab and discovered that our NetFlow configuration was incompatible. We fixed the configuration before upgrading production VDS. The VDS upgrade itself takes 5-10 minutes but affects all connected hosts. Plan a brief network disruption.
Lesson 8: Update Lifecycle Manager Baselines
vSphere 8 uses vSphere Lifecycle Manager (vLCM) for host management. If you have existing vLCM baselines from vSphere 7, they need to be updated. We upgraded hosts using old baselines and got inconsistent configurations.
Create new vLCM baselines specifically for vSphere 8. Include all required firmware and driver versions. Test the baselines on a small group before applying to all hosts.
Lesson 9: Monitor Upgrade Progress
vSphere 8 upgrades take 30-60 minutes per host. For 500 hosts, that is 250-500 hours of upgrade time. We ran upgrades in parallel (10 hosts at a time) to reduce total time. But parallel upgrades require careful monitoring.
We created a monitoring dashboard that shows upgrade status for every host. If a host fails, we investigate immediately. Do not wait for all upgrades to finish to discover failures. Real-time monitoring is essential for large-scale upgrades.
Lesson 10: Test VM Functionality After Upgrade
Every upgraded host needs VM testing. We run a validation script that checks VM boot, network connectivity, and application functionality. The script runs automatically after each host upgrade.
One host passed the upgrade but had a storage driver issue. VMs booted but had extremely slow disk performance. The validation script caught this. Without it, the issue would have gone undetected until users complained.
Lesson 11: Keep Rollback Plans Ready
Every upgrade phase needs a rollback plan. For vCenter, we took a backup before upgrade and tested restoration. For ESXi hosts, we kept boot media with vSphere 7 available. If an upgrade failed, we could reinstall vSphere 7 in 30 minutes.
We used rollback plans twice during the project. Once for a vCenter plugin that was incompatible. Once for a host that had a hardware issue during upgrade. Having rollback plans ready saved hours of troubleshooting.
Lesson 12: Communicate Throughout
We underestimated the importance of communication. The upgrade affected 500 hosts across 8 data centers. Multiple teams were involved: server, network, storage, applications. Without clear communication, chaos ensues.
We created a daily status email that showed: hosts upgraded, hosts pending, issues encountered, and next steps. Every stakeholder received the email. It took 15 minutes to write and prevented dozens of "what is happening?" conversations.
Best Practices
Beyond the 12 lessons, these practices improve upgrade success.
**Use automated upgrade scripts.** We wrote PowerCLI scripts that automate the entire upgrade process: pre-checks, firmware updates, ESXi upgrade, validation, and reporting. Automation reduces human error and speeds up the process.
**Schedule upgrades during low-activity periods.** vSphere 8 upgrades cause brief host reboots. Schedule these during maintenance windows when few users are affected. We upgraded production hosts on weekend nights.
**Maintain a staging environment.** Keep a small vSphere 8 environment running for testing. When new patches or updates are released, test them in staging before applying to production.
**Document everything.** Create an upgrade runbook that documents every step, every issue, and every resolution. Future upgrades will be faster because you can reference the runbook.
Common Mistakes
These are the errors we see most often in vSphere 8 upgrades.
**Mistake 1: Skipping pre-upgrade checks.** VMware provides pre-upgrade checks that verify compatibility. Skipping them is asking for trouble. Run every check and resolve every issue before upgrading.
**Mistake 2: Upgrading everything at once.** Upgrade vCenter first, then hosts, then vSAN, then NSX. Each component depends on the others. Sequential upgrades reduce risk.
**Mistake 3: Ignoring third-party compatibility.** Your backup software, monitoring tools, and custom scripts need vSphere 8 compatibility. Test them before upgrading. We found three incompatible tools during our upgrade.
**Mistake 4: Not backing up vCenter.** vCenter contains all your configuration. If the upgrade fails and you have no backup, you rebuild from scratch. That takes days, not hours. Always backup vCenter before upgrading.
**Mistake 5: Rushing the upgrade.** vSphere 8 upgrades take time. Rushing leads to mistakes. We allocated 14 weeks for 500 hosts. It was tight but realistic. Plan conservatively.
Conclusion
The vSphere 8 upgrade is necessary but not trivial. Our 500-host, 14-week project taught us that success depends on preparation, testing, and communication. The 12 lessons in this article would have saved us six weeks.
If you are planning a vSphere 8 upgrade, start with the release notes. Test in a lab. Upgrade vCenter first. Check hardware compatibility. Plan for vSAN and VDS. Monitor everything. And communicate constantly.
The upgrade is a marathon, not a sprint. Pace yourself, test thoroughly, and do not skip steps. The result is a modern, secure, performant platform that will serve you for the next five years.
Want to go deeper? Explore [VMware alternatives](/en/vmware-alternative), [Run infrastructure services](/en/products/run), or [platform comparison](/en/compare).
FAQ
**Q: Can I upgrade directly from vSphere 7 to vSphere 8?**
A: Yes. vSphere supports direct upgrade from 7.0 Update 3 and later. Earlier versions of vSphere 7 require updating to 7.0 Update 3 first.
**Q: How long does a single host upgrade take?**
A: 30-60 minutes including reboot. The actual upgrade process is 10-15 minutes; the rest is reboot and validation time. Plan for 1 hour per host to be safe.
**Q: Do I need to upgrade all hosts simultaneously?**
A: No. Upgrade hosts one at a time or in small batches. vCenter 8 can manage both vSphere 7 and 8 hosts during the upgrade process. This allows gradual migration.
**Q: What happens to my VMs during the upgrade?**
A: VMs are migrated to other hosts using vMotion before the upgraded host reboots. If vMotion is not available, VMs are powered off and restarted on other hosts. Plan for brief downtime for VMs on upgraded hosts.
**Q: Can I downgrade from vSphere 8 to vSphere 7?**
A: Not officially supported. VMware recommends restoring from backup if you need to revert. This is why backups before upgrade are critical.