Introduction
This post documents my evolving home-lab setup as of December 2018
Since many of my clients are smaller SMB’s they often need hands-on help with physical storage, networking and security in addition to VMware. Since my background was coming from a software ( not IT ) area I needed a lab that used physical hardware ( e.g production servers and network equipment )
Equipment
- My lab has gone through several iterations, with the following equipment used as of Dec 2018. This lab is for learning hardware and networking in addition to VMware so it is a bit more involved. I try to segregate traffic on vLANs just like production.
- I have only 2 physical ESXi servers, but lots of Cores and RAM to run a Nested ESXi environment ( described later )
- Note that older equipment is often very reasonably priced on Ebay. R630’s are now showing up for great prices. 1GB SFP optics and LC fibre cables are also very inexpensive.
- Servers
- Dell R710, Dual socket, 6 cores, 144GB RAM, 4 x 1GB Nic
- Dell R630, Dual socket, 10 cores, 256GB RAM, 4 x 1GB Nic, 2 x 10GB SFP+
- Shared Storage ( Running FreeNAS 11.2 )
-
- Dell 2950, 2 socket, 4 core, 32GB RAM, 2 x 1GB Nic,
6 x 4TB LFF WD Red SATA for ZFS pool
- Dell 2950, 2 socket, 4 core, 32GB RAM, 2 x 1GB Nic,
1 x Intel 320, 80GB SSD, L2ARC ( with power fail caps )
1 x Intel 3700, 100G SSD, ZFS SLOG (with power fail caps )
See FreeNAS blog entry. -
- Networking
- 2 x Cisco 2970G-24TS rack mount switches
24 x 1G ports on each switch
4 x SFP uplink ports on each switch - Note The 2970G switch in my mobile rack uplinked to it’s sister switch in my Office over 2 x SFP fiber connections ( LACP on uplinks , passing 802.1Q vLAN trunks up to the office ) This was my first hands-on experience with LC fibre and switch-to-switch LAG.
- 2 x Cisco 2970G-24TS rack mount switches
- Power
- APC SUA2200RM2U 2200VA UPS
To reduce chance of disk corruption on FreeNAS due to power interruptions
Available inexpensively from refurbups.com
- APC SUA2200RM2U 2200VA UPS
- Backup
- For static ISO files , templates, etc. I sync FreeNAS NFS shares to Amazon S3
- For VM’s I take daily ZFS snapshots , and replicate them to cloud storage.
- Both of these are done from FreeNAS GUI
( see FreeNAS backup and replication )
- Security
- Perimeter FW – pfSense VM
vLAN trunks for WAN, LAN over LAG (fibre) from rack to office. - Backup FW – Ubiquity Edgerouter X
Since main FW is a VM, this small device is for emergency use if my lab cluster is down and I need to get to the Internet. The EdgeRouter also supports
ECMP and BGP for uplinks from an NSX edge.
- Perimeter FW – pfSense VM
- Remote access
- Console
Physical KVM for my older servers and iDRAC Enterprise for newer ones.
Note that modern browser security often prevents older iDRAC from working. - VPN remote access
OpenVPN ( part of pfSense server )
- Console
- Rack – This is an older screw-mount style 4-post mobile rack and KVM found for under $300 on Ebay for local pickup. The rack has several 4-inch fans in the top for cooling ( I added more ) I also added egg-crate foam insulation to the inside walls to deaden the sound quite a bit.
Configuration
Network Design
My existing physical Lab design now uses a single distributed vSwitch (dvS) with (4) 1G uplinks from each physical hosts.
Normally you would use separate dvS for LAN and SAN segment to separate traffic. However there are two reasons for using one dvS.
- When migrating to 10G converged networking your hosts typically have only 2 uplinks ( and therefor only a single dvS is supported )
- When setting up a nested ESXi environment it’s easier to have only a single dvS
Since I’m not yet ready to upgrade my lab to 10G switches, the current design is a compromise.
dvS Uplinks 1 and 2 are dedicated for LAN, DMZ , WAN and VTEP traffic, while uplinks 3 and 4 are for SAN and vMotion. This keeps the traffic separated , and each pair of uplinks is redundant at the physical level.
Note that for this design each dv-portgroup must be set to use the correct set of uplinks.
For example a LAN portgroup teaming policy would use uplink 1 (active), uplink 2 (standby) and the other uplinks set to not used.
When migrating later to 10G networking, there would be only 2 redundant uplinks wih all traffic separated by vLAN. Separating the Storage and Network traffic is not really needed with 10G, and NIOC can manage it.
Distributed Switch (dvS) Setup
The single dvS and associated physical switch ( Cisco 2970G-24TS ) is setup as follows:
- Distributed vSwitch – dvS
Settings: switch MTU=9000Uplinks ( all traffic on vLAN’s )- Uplink 1 and Uplink 2
LAN, DMZ, WAN traffic on vLAN 24,26 and 902
typical physical switch settings for Cisco 2970 uplink 1interface GigabitEthernet0/11 description ESX2 vmnic0 LAN switchport trunk encapsulation dot1q switchport trunk allowed vlan 24,26,902 switchport mode trunk spanning-tree portfast trunk
repeat same settings for Gi0/12 for uplink 2
- Uplink 3 and Uplink 4
NFS, vMotion traffic on vLAN 25, 881
typical physical switch settings for Cisco 2970 uplink3interface GigabitEthernet0/5 description ESX2 vmnic3 SAN,vMot switchport trunk encapsulation dot1q switchport trunk allowed vlan 25,881 switchport mode trunk spanning-tree portfast trunk
repeat settings for Gi0/6 for uplink 4
- Uplink 1 and Uplink 2
- Portgroups settings on dvS
- dv-LAN
Settings: ( Uplinks 1 and 2 used )vlan 24 Teaming: Active: Uplink 1 Standby: Uplink 2 Unused: Uplink 3, Uplink 4
- dv-NFS
Settings: ( Uplinks 3 and 4 used )vlan 25 Teaming: Active: Uplink 3 Standby: Uplink 4 Not used: Uplink 1, Uplink 2
- dv-Motion
Settings: ( Uplinks 3 and 4 used )vlan 881 Teaming: Active: Uplink 3 Standby: Uplink 4 Not used: Uplink 1, Uplink 2
- dv-LAN
Verify shared Storage and vMotion working correctly.
Physical switch vLAN’s
Best practice is to avoid using the default vLAN (1) on the switch.
ESX management traffic and iDRAC etc. are running on a Management vLAN
configured on the switch. Storage traffic is on a separate vLAN 25. As I showed above under ‘dvs Setup’ the Management vlan 24 is passed over trunks into ESX for the management port group.
interface Vlan24 description Mgmt no ip address no ip route-cache ! interface Vlan25 description Storage NFS iSCSI no ip address no ip route-cache
FYI – the ‘no ip route cache’ is the default – since this is only a layer-2 switch.
Storage notes
I’m using NFS storage in my setup, so I have only a single “NFS” portgroup on the dvS that has and active and standby uplinks for redundancy. For simplicity I’m not using LACP.
Note that my FreeNAS storage also supports iSCSI, but in that case proper redundancy requires multipath setup. This is typically done by using two portgroups ( SCSI-1, SCSI-2 ) with each one using a specific uplink ( and no standby ). iSCSI is likely not compatible with LACP uplinks
4 thoughts on “vSphere / ESXi homelab setup”