ESX/NSX nested lab – Part I

This part includes setup of Nested ESX VM’s.  The second part will include nested NSX.

Setup

This section covers what worked for me and the major steps required.   You may need to reference the listed links for more info.

The steps to setup the lab are:

  1. I have two ESXi physical host running ESX 6.5 (typical)
    128GB RAM, 2 sockets, 24 cores, NFS or iSCSI storage
    dvSwitch v6.5.0.Note: NSX  vibs not installed on the 2 physical host(s) as it breaks things.NSX manager 6.4.4 already installed and integrated with vCenterVirtual and Physical switches for LAN enabled for MTU: 1600Virtual and Physical switches for SAN, vMotion set for jumbo MTU:9000
  2. The distributed Switch, dvS, can be used for both Physical hosts, and the nested hosts, if the uplinks from the nested hosts are setup to mirror the configuration of the physical hosts.   The nested host has 4 vnics that are used for uplinks – to mirror the 4 physical Nic’s on my physical hosts.   Note that the Nested vNic’s are attached to the special “Nested…” port groups on the switch.   This can get confusing pretty fast so see the diagram below on how the special nested porgroups are configured.
  3. Install ESXi 6.5 to a VM for each host, named esxN1, esxN2, etc.
    using vHW:v13

    1. Set CPU=6, cores per socket=6  expand CPU section and set:
      Hardware Virtualization (Expose HW assisted Virt to guest)
      Note that NSX controllers require 4 cpu  – so your virtual hosts should have at least that many cores.  Also set cores/socket same as CPU, so that you only use one ESX license instead of 6, allowing more nested Hosts  ( Assume you have vMUG Advantage licenses or equivalent )
    2. Set RAM=8GB or more
    3. Install a single 4GB disk for ESXi image, no other disks.  If the disk is increased to 8GB,  then a local scratch partition will be used for logs, else the logs will be on RAM disk.
    4. Setup 4 network interfaces.  Connect first of these to your management network.
    5. Under VM options, Force EFI setup on next boot.
  4. Install ESX-Learnswitch vib onto ESXi 6.5 physical host (link #2)
    Note that ESXi 6.7 includes LearnSwitch, so no install is needed.
    Summary:
    scp VMware-ESX-6.5.0-xxxxx-learnswitch.zip root@esxhost:/tmp
    ssh root@esxhost
    esxcli software vib install -d /tmp/VMware-ESX-6.5.0-xxxxx-learnswitch.zip
  5. Prepare dVS portgroups for nested ESXi according to (link #2):nestpg
    I’m using Nest-LAN1,  Nest-LAN2, Nest-SAN1, Nest-SAN2 so that each
    nested ESXi host has 2 LAN uplinks and 2 SAN uplinks for symmetry to my physical hosts.  Some migrations to/from vSwitch to dvS are difficult without having two uplinks for each PG.Note these 4 portgroups are used ONLY for the nested ESXi VM uplinks,  not the nested hosts , or any other VM’s.     Since link status isn’t likely correct for these PG’s we map them 1-to-1 to the vnics’s,  so only the physical host’s PG’s make teaming decisions based on the more likely accurate link-status of the physical nics.

    1. Nested-LAN ( for Mgmt, VTEP ) properties
      1. Security – allow promiscuous mode, allow forged transmit,  allow mac changes
      2. VLAN set to trunk mode VLAN = 24,100   (see Link #5)
        For LAN and  vxlan traffic
        Insure that Physical uplink passes vLans: 24, 100
      3. assocated vmKernel set to 1600 MTU,  and dvs set to 1600 MTU
        This is for future use of NSX vxlan.
    2.  Nested-SAN ( for vMotion, NFS )  properties
      1. Security – allow promiscuous mode, allow forged transmit, allow mac changes
      2. VLAN – trunk mode vlan = 25,881
        Physical uplink passes vLans: 25, 881
      3. associated vmKernel set to 9000 MTU
    3. Setup LearnSwitch for these 4 dv Portgroups
      Extract python script per link #2.
      Set vcenter admin/password and four Portgroup name(s) in cfg script
      Run:
      python learnswitch_cfg.py  vcenter_IP  dvsNAME esxHost1IP add
      python learnswitch_cfg.py  vcenter_IP  dvsNAME esxHost2IP add
  6. Install new ESXi VM from ISO
    1. Assign Mgmt IP
    2. Setup time server
  7. Join Host to new Cluster ‘compute’

    nestedESX
    Nested ESX VM’s esx-n1, etc and Compute cluster with Nested Hosts
  8. Configure new host
    Add additional vmkernels for vMotion, NFS, etc.  Initially these will need to run from 2 local vSwitches , vSwitch0 for LAN,  vSwitch1 for SAN/vMotion
  9. Join host to vDS, and Migrate vmkernel ports and uplinks to vDS.
  10. Test that NFS storage and vMotion are working correctly.
    Test vmotion between Nested ESX running on different physical hosts.
    Verify NFS storage latency is normal.

Continue on with ESX/NSX nested lab – Part II setup.

Links

  1. Nested ESXi 6.5 virtual appliance   (vGhetto)
  2. Nested ESXi LearnSwitch   (vGhetto)
  3. How to clone a nested ESXi VM  (vGhetto)
  4. NSX issue on dvs in nested ESXi  (telecomOccasionally)
  5. Nested Virtualization  (Limitless) installing ESX, trunked portgroup

Changes

  1. Add nested ESXi PG graphic

Author: leemarzke

VMware Infrastructure consultant

2 thoughts on “ESX/NSX nested lab – Part I”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s