Introduction

This post documents my evolving home-lab setup as of December 2018

Since many of my clients are smaller SMB’s they often need hands-on help with physical storage, networking and security in addition to VMware.   Since my background was coming from a software ( not IT ) area I needed a lab that used physical hardware ( e.g production servers and network equipment )

Equipment

• My lab has gone through several iterations,  with the following equipment used as of Dec 2018.  This lab is for learning hardware and networking in addition to VMware so it is a bit more involved.  I try to segregate traffic on vLANs just like production.
• I have only 2 physical ESXi servers, but lots of Cores and RAM to run a Nested ESXi environment ( described later )

• Note that older equipment is often very reasonably priced on Ebay.  R630’s are now showing up for great prices.  1GB SFP optics and LC fibre cables are also very inexpensive.

• Servers
  • Dell R710,  Dual socket, 6 cores, 144GB RAM,  4 x 1GB Nic
  • Dell R630, Dual socket, 10 cores, 256GB RAM,  4 x 1GB Nic, 2 x 10GB SFP+
• Shared Storage  ( Running FreeNAS 11.2 )
  • • Dell 2950, 2 socket, 4 core, 32GB RAM, 2 x 1GB Nic,
      6 x 4TB LFF WD Red SATA for ZFS pool

  1 x Intel 320, 80GB SSD,  L2ARC ( with power fail caps )
  1 x Intel 3700, 100G SSD, ZFS SLOG (with power fail caps )
  See FreeNAS blog entry.

• Networking
  • 2 x Cisco 2970G-24TS rack mount switches
    24 x 1G ports on each switch
    4 x SFP uplink ports on each switch
  • Note The 2970G switch in my mobile rack uplinked to it’s sister switch in my Office over 2 x SFP fiber connections ( LACP on uplinks , passing 802.1Q vLAN trunks up to the office )   This was my first hands-on experience with LC fibre and switch-to-switch LAG.
• Power
  • APC SUA2200RM2U 2200VA UPS
    To reduce chance of disk corruption on FreeNAS due to power interruptions
    Available inexpensively from refurbups.com
• Backup
  • For static ISO files ,  templates,  etc.  I sync FreeNAS NFS shares to Amazon S3
  • For VM’s I take daily ZFS snapshots , and replicate them to cloud storage.
  • Both of these are done from FreeNAS GUI
    ( see FreeNAS backup and replication )
• Security
  • Perimeter FW –  pfSense VM
    vLAN trunks for WAN, LAN over LAG (fibre) from rack to office.
  • Backup FW –  Ubiquity Edgerouter X
    Since  main FW is a VM,  this small device is for emergency use if my lab cluster is down and I need to get to the Internet.  The EdgeRouter also supports
    ECMP and BGP for uplinks from an NSX edge.
• Remote access
  • Console
    Physical KVM for my older servers  and iDRAC Enterprise for newer ones.
    Note that modern browser security often prevents older iDRAC from working.
  • VPN remote access
    OpenVPN ( part of pfSense server )
• Rack –  This is an older screw-mount style 4-post mobile rack and KVM found for under $300 on Ebay for local pickup.   The rack has several 4-inch fans in the top for cooling ( I added more )    I also added egg-crate foam insulation to the inside walls to deaden the sound quite a bit.

Configuration

Network Design

My existing physical Lab design now uses a single distributed vSwitch (dvS) with (4) 1G uplinks from each physical hosts.

Normally you would use separate dvS for LAN and SAN segment to separate traffic.   However there are two reasons for using one dvS.

1. When migrating to 10G converged networking your hosts typically have only 2 uplinks ( and therefor only a single dvS is supported )
2. When setting up a nested ESXi environment it’s easier to have only a single dvS

Since I’m not yet ready to upgrade my lab to 10G switches,  the current design is a compromise.

dvS Uplinks 1 and 2 are dedicated for LAN, DMZ , WAN and VTEP traffic,  while uplinks 3 and 4 are for SAN and vMotion.   This keeps the traffic separated , and each pair of uplinks is redundant at the physical level.

Note that for this design each dv-portgroup must be set to use the correct set of uplinks.
For example a LAN portgroup teaming policy would use uplink 1 (active), uplink 2 (standby) and the other uplinks set to not used.

When migrating later to 10G networking,  there would be only 2 redundant uplinks wih all traffic separated by vLAN.   Separating the Storage and Network traffic is not really needed with 10G,  and NIOC can manage it.

Distributed Switch (dvS) Setup

The single dvS  and associated physical switch ( Cisco 2970G-24TS ) is setup as follows:

• Distributed vSwitch – dvS
  Settings:   switch MTU=9000Uplinks ( all traffic on vLAN’s )
  • Uplink 1 and Uplink 2
    LAN, DMZ, WAN traffic on vLAN 24,26 and 902
    typical physical switch settings for Cisco 2970 uplink 1

    interface GigabitEthernet0/11
    description ESX2 vmnic0 LAN
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 24,26,902
    switchport mode trunk
    spanning-tree portfast trunk

    repeat same settings for Gi0/12 for uplink 2

  • Uplink 3 and Uplink 4
    NFS, vMotion traffic on vLAN 25, 881
    typical physical switch settings for Cisco 2970 uplink3

    interface GigabitEthernet0/5
    description ESX2 vmnic3 SAN,vMot
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 25,881
    switchport mode trunk
    spanning-tree portfast trunk

    repeat settings for Gi0/6 for uplink 4

• Portgroups settings on dvS
  • dv-LAN
    Settings:   ( Uplinks 1 and 2 used )

    vlan 24
    Teaming:
    Active: Uplink 1
    Standby: Uplink 2
    Unused:   Uplink 3, Uplink 4

  • dv-NFS
    Settings:  ( Uplinks 3 and 4 used )

    vlan 25
    Teaming:
    Active: Uplink 3
    Standby: Uplink 4
    Not used:  Uplink 1, Uplink 2

  • dv-Motion
    Settings:  ( Uplinks 3 and 4 used )

    vlan 881
    Teaming:
    Active: Uplink 3
    Standby: Uplink 4
    Not used:  Uplink 1, Uplink 2

Verify shared Storage and vMotion working correctly.

Physical switch vLAN’s

Best practice is to avoid using the default vLAN (1) on the switch.

ESX management traffic and iDRAC etc. are running on a Management vLAN
configured on the switch.  Storage traffic is on a separate vLAN 25.  As I showed above under ‘dvs Setup’ the Management vlan 24 is passed over trunks into ESX for the management port group.

interface Vlan24
description Mgmt
no ip address
no ip route-cache
!
interface Vlan25
description Storage NFS iSCSI
no ip address
no ip route-cache

FYI – the ‘no ip route cache’ is the default – since this is only a layer-2 switch.

Storage notes

I’m using NFS storage in my setup,  so I have only a single “NFS” portgroup on the dvS that has and active and standby uplinks for redundancy.  For simplicity I’m not using LACP.

Note that my FreeNAS storage also supports iSCSI, but in that case proper redundancy requires multipath setup.    This is typically done by using two portgroups ( SCSI-1,  SCSI-2 ) with each one using a specific uplink ( and no standby ).  iSCSI is likely not compatible with LACP uplinks