Definitive Nested ESXi 6.5/NSX guide

Steps to create a nested ESX 6.5 / NSX lab.


This guide documents a working nested NSX lab after I had numerous problems setting up nested ESX 6.5 with NSX with existing documentation written for ESX 5.5 and earlier.

This guide covers items specific to nested ESXi & NSX ( that is installing NSX into virtual ESXi hosts ) not generic instructions for NSX as there are plenty of instructions on-line for that.  Since this is a lab, only a single NSX controller is used,  but if you have the capacity all 3 can be installed.

Thanks to William Lam’s ESX appliance and blog entries below I was able to get started with nested ESXi, but continued to have random issues with networking.   There are two reasons I’m switching from W. Lam’s appliance back to the ESXi install:

  • The vGhetto ESXi appliance has ESX Mac learn dvFilter installed, which may conflict with the newer method of using ESX-LearnSwitch on the physical host.
  • ESXi 6.0+ installer now has vmtools pre-installed , so that feature of the appliance isn’t needed anymore

There were a few other issues I ran into along the way

  • There is an issue (link #4) with nested ESXi on a host running also running NSX causing loss of connectivity over vxlan tunnels.  Easiest solution is to remove NSX from the physical host.
  • Some configurations of dvs portgroups ( with ESX-Learnswitch) caused the nested ESX nic to fail to load on boot.   Creating new dvs portgroups fixed this issue.
  • The cloning method (link #3) only appears to work for vmk0 (Mgmt) vmkernel, and duplicate Mac’s appear on other vmKernels causing intermittent network issues.


Nested ESXi 6.5 with NSX Lab

My existing physical Lab design uses multiple vLAN’s,  uplinks, etc. to try and simulate a real production environment.  Many of the existing NSX labs place all networks ( LAN, vMotion, SAN, etc.) on a flat single net which doesn’t fit into my lab environment.

My Existing ESX Lab Networks

Two distributed vSwitch’s
each with two physical 1G uplinks

  • dvsLAN  (vm’s and management)
    lan: / 16   vlan:24
  • dvsSAN
    storage: / 24      vlan:25
    vMotion: / 24     vlan:881
    vxlan:        vlan: 30



I’m not showing detailed setup images as those steps are covered in many places and in the links below.    This page documents what doesn’t work,  or specific requirements  on top of those in the linVM’s and managementks.
The steps to setup the lab are:

  1. Typical ESXi physical host running ESX 6.5
    128GB RAM, 2 sockets, 24 cores, NFS or iSCSI storage
    dvSwitch v6.5.0,  NSX vibs not installed on host
    NSX manager already installed and integrated with vCenter
    Virtual and Physical switches for LAN enabled for MTU: 1600
    Virtual and Physical switches for SAN, vMotion set for jumbo MTU:9000
  2. Install ESXi 6.5 to a VM template called ESXi-6.5-template, vHW:v13
    1. Set CPU=6,  expand CPU section and set:
      Hardware Virtualization (Expose HW assisted Virt to guest)
      (NSX controllers require 4 cpu, so I’ve added 6 to the host)
    2. Set RAM=8GB
    3. Install a 4GB disk for ESXi image, no other disks
    4. Setup a network interface for LAN uplink. ( vSwitch0 )
    5. Setup a network interface for SAN uplink. ( vSwitch1 )
    6. Under VM options, Force EFI setup on next boot.
  3. Prepare template VM for cloning by executing inside the VM template:
    Refer to vGhettto (link #3)

    1. esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
    2. sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf
  4. Convert this VM to a template.
  5. Install ESX-Learnswitch vib onto ESXi 6.5 physical host (link #2)
  6. Prepare dVS portgroups for nested ESXi according to (link #2):
    (Note these portgroups used ONLY for the ESXi nested VM’s, nothing else )

    1. Nested-LAN ( for Mgmt, VTEP ) properties
      1. Security – promiscuous mode, allow forged transmit,  allow mac changes
      2. VLAN – trunk mode (see Link #5)
        Physical uplink passes vLans: 24, 30
      3. Learnswitch enabled on port
      4. assocated vmKernel set to 1600 MTU,  and dvs set to 1600 MTU
    2.  Nested-SAN ( for vMotion, NFS )  properties
      1. Security – allow forged transmit, allow mac changes
      2. VLAN – trunk mode
        Physical uplink passes vLans: 25, 881
      3. associated vmKernel set to 9000 MTU
  7. Clone VM template to ESXi host
    1. Use Customization wizard for mgmt Kernel IP changes.
    2. Power on new ESXi host VM
  8. Join Host to new Cluster ‘NSX-compute’
  9. Configure new host
    1. Add additional vmkernels for vMotion, NFS, etc.
    2. Migrate vSwitch0 uplink and vmk0 kernel port to dVS management vLAN if needed with wizard
    3. Migrate vSwitch1 uplink and other vmkernels to dVS vLANs if needed.
    4. Setup NFS storage as needed
  10. Install NSX
    1. Create IP pools
      Networking & Security – Groups – IP Pools
      1. Controller-pool  ( in management IP LAN ) –    /16
      2. vTEP-pool ( new unused LAN subnet for vxlan ) –  /24  vlan: 30
    2. Setup new vLAN for vxlan segment on physical switch
      vlan: 30
    3. Install one Controller ( into NSX-compute cluster )
      For a lab, only a single controller needed – not three
    4. Install NSX to Comp1   (compute cluster)
    5. Prepare vxlan on hosts
    6. Setup Segment ID
      1. pool: 5000-5999      ( 999 segments )
      2. Multicast:  ( for optional multicast )
    7. Setup transport zone ‘Global’
      1. To include NSX-compute cluster,  Unicast mode.


  1. Nested ESXi 6.5 virtual appliance   (vGhetto)
  2. Nested ESXi LearnSwitch   (vGhetto)
  3. How to clone a nested ESXi VM  (vGhetto)
  4. NSX issue on dvs in nested ESXi  (telecomOccasionally)
  5. Nested Virtualization  (Limitless) installing ESX, trunked portgroup



Author: leemarzke

VMware Infrastructure consultant

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s