ESX/NSX nested lab – Part I – Nuance of iqlusion

This part includes setup of Nested ESX VM’s. The second part will include nested NSX.

Setup

This section covers what worked for me and the major steps required. You may need to reference the listed links for more info.

The steps to setup the lab are:

I have two ESXi physical host running ESX 6.5 (typical)
128GB RAM, 2 sockets, 24 cores, NFS or iSCSI storage
dvSwitch v6.5.0.Note: NSX vibs not installed on the 2 physical host(s) as it breaks things.NSX manager 6.4.4 already installed and integrated with vCenterVirtual and Physical switches for LAN enabled for MTU: 1600Virtual and Physical switches for SAN, vMotion set for jumbo MTU:9000
The distributed Switch, dvS, can be used for both Physical hosts, and the nested hosts, if the uplinks from the nested hosts are setup to mirror the configuration of the physical hosts. The nested host has 4 vnics that are used for uplinks – to mirror the 4 physical Nic’s on my physical hosts. Note that the Nested vNic’s are attached to the special “Nested…” port groups on the switch. This can get confusing pretty fast so see the diagram below on how the special nested porgroups are configured.
Install ESXi 6.5 to a VM for each host, named esxN1, esxN2, etc.
using vHW:v13
1. Set CPU=6, cores per socket=6 expand CPU section and set:
  Hardware Virtualization (Expose HW assisted Virt to guest)
  Note that NSX controllers require 4 cpu – so your virtual hosts should have at least that many cores. Also set cores/socket same as CPU, so that you only use one ESX license instead of 6, allowing more nested Hosts ( Assume you have vMUG Advantage licenses or equivalent )
2. Set RAM=8GB or more
3. Install a single 4GB disk for ESXi image, no other disks. If the disk is increased to 8GB, then a local scratch partition will be used for logs, else the logs will be on RAM disk.
4. Setup 4 network interfaces. Connect first of these to your management network.
5. Under VM options, Force EFI setup on next boot.
Install ESX-Learnswitch vib onto ESXi 6.5 physical host (link #2)
Note that ESXi 6.7 includes LearnSwitch, so no install is needed.
Summary:
scp VMware-ESX-6.5.0-xxxxx-learnswitch.zip root@esxhost:/tmp
ssh root@esxhost
esxcli software vib install -d /tmp/VMware-ESX-6.5.0-xxxxx-learnswitch.zip
Prepare dVS portgroups for nested ESXi according to (link #2):
I’m using Nest-LAN1, Nest-LAN2, Nest-SAN1, Nest-SAN2 so that each
nested ESXi host has 2 LAN uplinks and 2 SAN uplinks for symmetry to my physical hosts. Some migrations to/from vSwitch to dvS are difficult without having two uplinks for each PG.Note these 4 portgroups are used ONLY for the nested ESXi VM uplinks, not the nested hosts , or any other VM’s. Since link status isn’t likely correct for these PG’s we map them 1-to-1 to the vnics’s, so only the physical host’s PG’s make teaming decisions based on the more likely accurate link-status of the physical nics.
1. Nested-LAN ( for Mgmt, VTEP ) properties
  1. Security – allow promiscuous mode, allow forged transmit, allow mac changes
  2. VLAN set to trunk mode VLAN = 24,100 (see Link #5)
    For LAN and vxlan traffic
    Insure that Physical uplink passes vLans: 24, 100
  3. assocated vmKernel set to 1600 MTU, and dvs set to 1600 MTU
    This is for future use of NSX vxlan.
2. Nested-SAN ( for vMotion, NFS ) properties
  1. Security – allow promiscuous mode, allow forged transmit, allow mac changes
  2. VLAN – trunk mode vlan = 25,881
    Physical uplink passes vLans: 25, 881
  3. associated vmKernel set to 9000 MTU
3. Setup LearnSwitch for these 4 dv Portgroups
  Extract python script per link #2.
  Set vcenter admin/password and four Portgroup name(s) in cfg script
  Run:
  python learnswitch_cfg.py vcenter_IP dvsNAME esxHost1IP add
  python learnswitch_cfg.py vcenter_IP dvsNAME esxHost2IP add
Install new ESXi VM from ISO
1. Assign Mgmt IP
2. Setup time server
Join Host to new Cluster ‘compute’
Nested ESX VM’s esx-n1, etc and Compute cluster with Nested Hosts
Configure new host
Add additional vmkernels for vMotion, NFS, etc. Initially these will need to run from 2 local vSwitches , vSwitch0 for LAN, vSwitch1 for SAN/vMotion
Join host to vDS, and Migrate vmkernel ports and uplinks to vDS.
Test that NFS storage and vMotion are working correctly.
Test vmotion between Nested ESX running on different physical hosts.
Verify NFS storage latency is normal.