HCI Troubleshooting: Common Issues and Fixes
Last Friday at 4:45pm, a client's Nutanix cluster started throwing storage performance alerts. Latency spiked from 0.3ms to 8ms. VMs were sluggish. The IT team panicked — they had a board meeting Monday and needed everything running smoothly.
After 30 minutes of diagnosis, the cause was simple: someone had added 5 new VMs without checking capacity, and the cluster was running at 92% storage utilization. Above 80%, Nutanix performance degrades. We offloaded some data to a secondary tier, brought utilization down to 65%, and latency dropped back to normal.
Most HCI issues have simple root causes. Here are the 12 most common problems we've encountered and how to fix them.
1. High Storage Latency
Symptom: VMs are slow. Storage latency is above 5ms (normal is under 1ms).
Root cause: Storage utilization above 80%. When HCI storage fills up, the software layer struggles to find free blocks for new writes.
Fix: Check storage utilization in your HCI console (Nutanix Prism, Sangfor iManager, vCenter). If above 80%, either delete unused data, migrate VMs to other clusters, or add storage nodes.
Prevention: Set alerts at 70% utilization. Address them before they become critical.
2. Network Partition (Split-Brain)
Symptom: Nodes can't see each other. The cluster shows "degraded" status. Some VMs may be unreachable.
Root cause: Network failure between nodes. Could be a switch failure, cable issue, or misconfigured VLAN.
Fix: Check inter-node connectivity. Ping between all nodes. Check switch port status. Look for CRC errors on switch interfaces. Replace faulty cables or reboot the problematic switch.
Prevention: Use redundant 10GbE links. Enable network monitoring on your HCI platform.
3. VM Won't Start After Node Failure
Symptom: A node failed, but some VMs aren't restarting on surviving nodes.
Root cause: Insufficient resources on surviving nodes. If the failed node was carrying most of the workload, other nodes may not have enough CPU/RAM.
Fix: Check resource utilization on surviving nodes. If they're maxed out, shut down non-critical VMs to free resources. The cluster's HA mechanism should then restart the critical VMs.
Prevention: Never run more than 70% cluster utilization. Leave 30% headroom for failover.
4. Slow VM Migration
Symptom: Live migration takes 30+ minutes instead of the expected 5-10 minutes.
Root cause: Slow inter-node network. Migration requires copying memory pages between nodes. If the network is congested or slow, migration crawls.
Fix: Check network utilization during migration. If the migration network is shared with production traffic, consider dedicating a separate VLAN for migration. Also check for background jobs (backup, replication) competing for bandwidth.
Prevention: Use dedicated migration networks. Schedule migrations during off-peak hours.
5. Deduplication/Compression Not Working
Symptom: Storage savings are lower than expected. Vendor promised 50-70% savings, you're seeing 10%.
Root cause: Data that doesn't compress well. Encrypted data, already-compressed data (ZIP, MP4), and random data (databases) don't deduplicate well.
Fix: Check which data types are consuming storage. Databases and media files won't compress much. Focus deduplication on OS volumes, file servers, and VDI desktops where compression ratios are highest.
Prevention: Set expectations correctly. Deduplication ratios depend entirely on data type.
6. Cluster Won't Form After Hardware Change
Symptom: After replacing a failed drive or adding a node, the cluster won't re-form.
Root cause: Configuration mismatch. The new hardware may have different firmware, different AOS version, or incorrect network configuration.
Fix: Check firmware and AOS versions match across all nodes. Verify network configuration (IP addresses, VLANs, DNS). Reboot the new node and check if it joins the cluster automatically.
Prevention: Always verify firmware compatibility before installing new hardware.
7. Replication Failing
Symptom: DR replication shows "failed" or "lagging." Your RPO target isn't being met.
Root cause: Network connectivity between sites, version mismatch, or storage full at the DR site.
Fix: Check network connectivity between primary and DR sites. Verify AOS versions match. Check DR site storage utilization. If the DR site is full, add storage or reduce replicated data.
Prevention: Monitor replication health daily. Set alerts for replication lag above threshold.
8. High CPU Steal Time
Symptom: VMs report high "steal time" (CPU time stolen by the hypervisor). Applications feel sluggish.
Root cause: CPU overcommitment. Too many vCPUs allocated relative to physical cores.
Fix: Check CPU overcommit ratio. If above 2:1, reduce vCPU allocation on non-critical VMs. Consider adding nodes if CPU is consistently maxed.
Prevention: Keep CPU overcommit below 2:1 for production. 3:1 is acceptable for dev/test only.
9. Snapshot Performance Impact
Symptom: VM performance drops significantly during snapshot operations.
Root cause: Snapshots create a "delta" disk that grows over time. Large or old snapshots consume I/O and storage.
Fix: Delete snapshots older than 24-48 hours. Check for "orphaned" snapshots (snapshots that the hypervisor lost track of). Limit concurrent snapshot operations.
Prevention: Schedule snapshots during off-peak hours. Set automatic snapshot expiry policies.
10. Management Console Unresponsive
Symptom: Nutanix Prism, Sangfor iManager, or vCenter is slow or unresponsive.
Root cause: Management VM resource contention. The management service shares resources with production VMs.
Fix: Check the management VM's resource utilization. If CPU or memory is maxed, increase its allocation. If the management service is crashed, restart it via the CLI.
Prevention: Reserve dedicated resources for management VMs. Don't let them compete with production.
11. Storage Tier Imbalance
Symptom: Some nodes are running hot (high storage utilization) while others have plenty of free space.
Root cause: VM placement not balanced. New VMs were placed on the same node without rebalancing.
Fix: Trigger a storage rebalance operation (available in most HCI consoles). This redistributes data across nodes evenly.
Prevention: Enable automatic rebalancing. Most HCI platforms have this feature — make sure it's on.
12. Certificate Expiry Warnings
Symptom: Management console shows certificate warnings. Some API integrations fail.
Root cause: Self-signed certificates expired. HCI platforms use certificates for inter-node communication and management access.
Fix: Regenerate certificates through the management console or CLI. For production, consider deploying certificates from your internal CA.
Prevention: Set calendar reminders for certificate renewal 30 days before expiry.
General Troubleshooting Approach
When you hit an HCI issue, follow this sequence:
Check the dashboard. Most HCI platforms show cluster health, storage utilization, and alerts at a glance.
Identify the symptom. Is it performance (slow), availability (unreachable), or capacity (full)?
Check the basics. Network connectivity, disk health, resource utilization. 80% of issues are in these three areas.
Check the logs. HCI platforms have detailed logs. Nutanix: "ncli" commands. Sangfor: web console logs. VMware: vCenter events.
Open a support ticket. If you can't identify the root cause in 30 minutes, call your vendor. That's what support is for.
Conclusion
Most HCI issues are predictable and preventable. Keep utilization below 80%, monitor your network, test failover regularly, and keep your software updated. The issues that do arise are usually fixable in under an hour if you follow a systematic approach.
Bookmark this guide. The next time your HCI cluster throws an alert at 4:45pm on a Friday, you'll know exactly where to start.
Want to go deeper? Explore [VMware alternatives](/en/vmware-alternative), [Run infrastructure services](/en/products/run), or [platform comparison](/en/compare).
FAQ
Q: How often should I check HCI health?
A: Daily for production environments. Review the dashboard every morning. Set up automated alerts for critical thresholds (storage >70%, CPU >80%, replication lag >15 minutes).
Q: When should I call vendor support?
A: If you can't identify the root cause in 30 minutes, or if the issue involves hardware failure, data corruption, or cluster-wide problems. Don't waste time guessing.
Q: Can I fix HCI issues without vendor support?
A: Most software issues (configuration, performance) can be fixed by your IT team. Hardware issues (failed drives, memory errors) require vendor support for replacement parts.
Q: How do I prevent recurring HCI issues?
A: Keep utilization below 80%, update software regularly, test failover quarterly, and document every issue and fix. A troubleshooting runbook saves hours during the next incident.
Sizing and Capacity Planning
Proper sizing is critical for HCI deployments. Start by inventorying your current workloads: CPU cores, memory per VM, storage per VM, and IOPS requirements. A general rule of thumb: each HCI node should run at 60-70% capacity to allow for growth and failover.
For a typical deployment of 50-100 VMs, we recommend starting with 4 nodes, each with: 2x 16-core CPUs, 256GB RAM, 4x 1.92TB NVMe SSDs, and 2x 25GbE NICs. This provides enough resources for most small-to-medium workloads with room to grow.
Migration Strategy from Traditional Infrastructure
Migrating from traditional SAN/NAS-based infrastructure to HCI requires careful planning. We recommend the following approach: First, identify non-critical workloads for initial migration (development, testing, staging environments). Second, use live migration tools (HCX for VMware, Xi Frame for Nutanix) to move VMs with zero downtime.
Third, validate performance on HCI before migrating production workloads. Monitor for 2-4 weeks to ensure IOPS, latency, and throughput meet requirements. Fourth, migrate production workloads in phases, starting with the least critical and progressing to mission-critical systems.
Disaster Recovery with HCI
HCI provides built-in high availability within a cluster, but you still need a disaster recovery plan for site-level failures. Options include: HCI-to-HCI replication between data centers (RPO as low as 5 minutes), cloud-based DR using HCI vendor cloud services, and hybrid DR with cloud object storage for backup.
We typically recommend a 3-2-1 backup strategy: 3 copies of data, on 2 different media types, with 1 copy offsite. With HCI, this translates to: local vSAN replication (copy 1), backup to secondary storage (copy 2), and cloud backup (copy 3).