Nested virtualization

On this page 7

Nested virtualization is the ability to run a hypervisor inside a virtual machine, so that the guest can create and run its own virtual machines. In virtualization and cloud infrastructure, it produces a stack of layers: the L0 hypervisor on physical hardware, an L1 guest hypervisor inside a virtual machine, and L2 nested guests running inside that.

Nested virtualization is used for training labs, testing hypervisor software, running Android emulators or VM-based sandboxes inside cloud instances, and building CI pipelines that need full virtual machines. It works on modern x86 and Arm hardware, but it adds measurable overhead and is usually not recommended for latency-sensitive production workloads.

How nested virtualization works

Hardware virtualization extensions such as Intel VT-x and AMD-V were designed for one hypervisor controlling the CPU. When a guest hypervisor at L1 executes a virtualization instruction, the processor traps to the L0 hypervisor, which then emulates the virtualization extensions for L1. In effect, L0 presents a virtual VT-x or AMD-V to its guest.

Each time an L2 guest performs a privileged operation, control may pass from L2 to L0, then to L1 for handling, and back down again. These multiplied VM exits are the main source of overhead. Modern processors reduce the cost with features such as VMCS shadowing on Intel and virtualized VMLOAD and VMSAVE on AMD, which let L1 work with virtualization state without trapping on every access.

LayerWhat runs thereExample
L0Host hypervisor on physical hardwareCloud provider hypervisor or KVM on bare metal
L1Guest VM running a hypervisorLinux VM with KVM, or Windows VM with Hyper-V
L2Nested guest VMTest VM, emulator or sandbox
L3 and deeperFurther nestingPossible in research, rarely practical
Layers in a nested virtualization stack

Memory virtualization in nested environments

Memory translation becomes more complex with each layer. A single hypervisor uses two-dimensional paging, Intel EPT or AMD NPT, to map guest physical memory to host physical memory. With nesting, L0 must combine the page tables of L1 and L2 into shadow extended page tables, because the hardware supports only one extra translation level directly.

Building and maintaining those combined tables costs CPU time whenever L1 changes its mappings for L2. Workloads that create many processes or allocate memory frequently inside L2 therefore suffer more overhead than steady compute workloads, while I/O-heavy workloads depend mostly on how devices are presented to the nested guest.

Device I/O adds a further layer. When L2 uses paravirtualized devices provided by L1, each request passes through two software device stacks. Direct device assignment to L2 is possible on some platforms with nested IOMMU support, but it is complex and rarely available in public cloud offerings.

Performance and platform support

The overhead of nested virtualization varies widely with the workload and with hardware assistance. CPU-bound code inside L2 can run close to single-level virtual machine speed, while workloads dominated by system calls, interrupts or page table changes can slow down substantially. Any figure without workload context is unreliable, so benchmarking the actual workload is essential.

Workload typeTypical nested overheadMain reason
CPU-bound computationLowFew VM exits during pure computation
Memory allocation heavyModerate to highShadow page table maintenance
Network and disk I/OModerate to highTwo device emulation layers
Frequent interrupts and timersHighExits multiplied across layers
Boot and provisioningNoticeably slowerMany privileged operations at startup
Typical impact by workload type

Platform support is no longer exotic. KVM enables nesting through module parameters such as nested=1 for kvm_intel or kvm_amd, Hyper-V supports it per VM, and VMware products expose hardware virtualization to guests as an option. Major public clouds offer nested virtualization on selected instance families, while others require bare-metal instances instead.

Nested virtualization compared with alternatives

Nested virtualization is one of several ways to get isolated environments inside a cloud instance. The comparison below helps decide whether nesting is needed at all.

OptionIsolation levelWhen it fits
Nested virtualizationFull VM with its own kernelHypervisor testing, full OS labs, VM-based sandboxes
ContainersShared kernel, namespaces and cgroupsApplication packaging without separate kernels
Software emulationFull machine, no hardware assistanceDifferent CPU architectures, slow but portable
Bare-metal instanceDirect hardware hypervisorProduction virtualization at full speed
Separate cloud VMsProvider-managed VMsMost cases where nesting is not essential
Options for running isolated workloads inside a cloud instance

Containers are often enough when the goal is only isolation of applications. Nested virtualization becomes necessary when the workload needs its own kernel, a different operating system, a microVM sandbox for untrusted code, or the virtualization software itself is under test.

Practical guidance and pitfalls

  • Check the instance type first: nesting is available only on specific cloud instance families and must sometimes be enabled explicitly.
  • Expose the right CPU features: the L1 guest needs the virtualization flag, vmx or svm, visible in its CPU model.
  • Expect limited live migration: some hypervisors restrict live migration or snapshots of VMs that run nested guests.
  • Size memory generously: L1 must hold its own operating system plus all L2 guests, and memory overcommit at two levels compounds risk.
  • Keep security boundaries in mind: nested guests add attack surface to L1, and vulnerabilities in nested virtualization code paths have historically been found in major hypervisors.

For production virtualization at scale, bare-metal instances remain the more predictable option. Nested virtualization is best treated as a convenience for development, testing, training and isolation scenarios where its overhead is acceptable.

A non-technical view

A virtual machine is a computer simulated inside a real one. Nested virtualization is a simulated computer that runs its own simulated computers, like a set of nesting dolls where each doll contains another.

It works, but every extra layer adds some extra effort and delay, because requests have to pass through more layers before reaching the real hardware. It is ideal for practice, testing and experiments, and less ideal for workloads that must be as fast as possible.

A concrete example

A training company teaches system administrators how to install and operate a hypervisor cluster. Buying physical lab servers for every student is too expensive, so each student receives a cloud VM with 16 vCPUs and 64 GB of RAM on an instance family that supports nested virtualization.

Inside that VM, students install KVM and build a three-node cluster of L2 virtual machines, practice migrations and simulate failures. The nested guests run slower than on dedicated hardware, but every student gets a complete, isolated lab that is created from a template in minutes and deleted at the end of the day.