Part 4: Deploy and Configure a 3 Node 2016 Hyper-V Cluster

Previous Post in Series:  Part 3:  Deploy a 2 Node Shared SAS 2016 Storage Spaces Cluster

Welcome to part 4 of the Server 2016 Features series.  In this section we’re going to deploy a 3 node Hyper-V cluster, below is a quick breakdown of the tasks that will be covered:

Prepare Hyper-V Hosts Using PowerShell

• Rename NICs for SMB Storage
• Set an IP for both SMB Storage NICs
• Set a VLAN on SMB storage NIC 2
• Enable jumbo packets on SMB Storage NICs
• Rename future team NICs
• Disable NICs that won’t be used
• Disable IPv6 on all connected NICs
• Disable “Register in DNS” relevant NICs
• Add required Windows Features
• Configure one of the future team NICs with management access (configure IP information)
• Set local administrator password to never expire – LAB BUILD ONLY
• Enable Remote Desktop
• Set Windows Time service Startup Type to Automatic
• Disable Windows Firewall – LAB BUILD ONLY
• Enable and configure VMQ on future team NICs
• Enable NetworkDirect connectivity between IP subnets
• Disable IE Enhanced Security Configuration – LAB BUILD ONLY
• Rename Server
• Rename Administrator Account – USER DISCRESION

Additional Tasks

• Apply all available Windows updates to the server
• Add Hyper-V hosts into SCVMM for management
• Create and Apply SCVMM Logical Networking to Hyper-V Hosts
• Create Hyper-V cluster
• Configure Cluster Networks
• Configure Live Migration Settings
• Add SMB file shares to new Hyper-V cluster

Prepare Hyper-V nodes using PowerShell

The following steps assume either remote access to the server via something like iDRAC or iLO or by having console access to the physical machine.

There are a few tasks that are required on every Hyper-V server that I deploy, as such I’ve put most of these tasks into a PowerShell script. At the point of writing this, the script has to be run locally on each node.

The script is written to work based on the hardware that I’ve been using recently to build my Hyper-V hosts and will, as a result need some tweaking before running it in your own environment. Mainly around NIC names and numbers etc. I’ve placed a comment at the end of each line that will likely need reviewed before running, although all code should be reviewed before running it on your own infrastructure 🙂

On my Hyper-V hosts I have a 2x10Gb port Mellanox connectX-3 card for my converged network traffic. This will include Management, tenant VM and SDN related traffic

I have a 2x10Gb port Chelsio T5 card for SMB storage

What this script does is detailed above and I’ve copied the code below:

##################################
#                                #
# Hyper-V - Post OS Installation #
#                                #
# Written by: David Fleming      #
# Date: 16th January 2017        #
# Version 4.0                    #
##################################

#region
Clear-Host

# Set Variables
$User = [adsi]"WinNT://$env:computername/administrator"
$Renamed_Admin = "NEWADMINUSERNAME"
$UnusedNICS = Get-NetAdapter NIC1, NIC2, NIC3, NIC4
$HostName = Read-Host "Please Enter a New Hostname for This Server"
$IP_Octet = Read-Host "Please enter the last Octet for this servers IP Addressing"
# This fucntion allows the users to chose between two sites and will pick the correct subnets and VLANs for the host.

Function Menu-Select
{
PARAM
(
[Parameter(Mandatory=$true)]
$Options,
$DisplayProperty
)
[int]$OptionPrefix = 1
# Create menu list
foreach ($Option in $Options)
{
if ($DisplayProperty -eq $null)
{
Write-Host ("{0,3}: {1}" -f $OptionPrefix,$Option)
}
else
{
Write-Host ("{0,3}: {1}" -f $OptionPrefix,$Option.$DisplayProperty)
}
$OptionPrefix++
}
Write-Host ("{0,3}: {1}" -f 0,"To Cancel")
[int]$Response = Read-Host "Please Select The Correct Infrastructure For The New Host"
$ChosenSite = $null
if ($Response -gt 0 -and $Response -le $Options.Count)
{
$ChosenSite = $Options[$Response-1]
$script:Site = $ChosenSite
}
return $ChosenSite
}

$Infrastructures = "SITE A","SITE B"
$ChosenSite = Menu-Select $Infrastructures

# This IF statement will make sure that the correct IPs and VLANs are being used based on the users site choice above
if ($Site -eq "SITE A")
{
# Chelsio SMB Variables
$SMB1_VLAN = "VLAN ID HERE"
$SMB2_VLAN = "VLAN ID HERE"
$SMB1iPAddress = "10.10.10." + $IP_Octet # Add the first 3 octets of your SMB 1 subnet here
$SMB2iPAddress = "10.10.11." + $IP_Octet # Add the first 3 octets of your SMB 2 subnet here
# Management Variables
$ManagementIP = "10.10.12." + $IP_Octet
$ManagementDefaultGW = "10.10.12.1"
$DNS1 = "10.10.12.11"
$DNS2 = "10.10.12.12"
}
Else
{
# Chelsio SMB Variables
$SMB1_VLAN = "VLAN ID HERE"
$SMB2_VLAN = "VLAN ID HERE"
$SMB1iPAddress = "10.11.10." + $IP_Octet # Add the first 3 octets of your SMB 1 subnet here
$SMB2iPAddress = "10.11.11." + $IP_Octet # Add the first 3 octets of your SMB 2 subnet here

# Management Variables
$ManagementIP = "10.11.12." + $IP_Octet
$ManagementDefaultGW = "10.11.12.1"
$DNS1 = "10.11.12.11"
$DNS2 = "10.11.12.12"
}
#endregion

#region
# Chelsio NICs
# Rename Network Adapters
Write-Host "Renaming Chelio Adapters" -ForegroundColor Yellow
Get-NetAdapter | where InterfaceDescription -Like "Chelsio*" | where Name -Like "SLOT * *" | Rename-NetAdapter -NewName "SMB 2 (VLAN $SMB2_VLAN)"
Get-NetAdapter | where InterfaceDescription -Like "Chelsio*" | where Name -Like "SLOT *" | Rename-NetAdapter -NewName "SMB 1 (VLAN $SMB1_VLAN)"

# Set IPs on SMB NICs
Write-Host "Configuring IPs on SMB NICs" -ForegroundColor Yellow
New-NetIPAddress -InterfaceAlias "SMB 1 (VLAN $SMB1_VLAN)" -IPAddress $SMB1iPAddress -PrefixLength 24
New-NetIPAddress -InterfaceAlias "SMB 2 (VLAN $SMB2_VLAN)" -IPAddress $SMB2iPAddress -PrefixLength 24

# Enable VLAN ID on SMB 2 NIC
Write-Host "Enabling VLAN ID of $SMB2_VLAN on SMB 2 NIC" -ForegroundColor Yellow
$SMB2NIC = Get-NetAdapter "SMB 2 (VLAN $SMB2_VLAN)"
Set-NetAdapter -Name $SMB2NIC.Name -VlanID $SMB2_VLAN -Confirm:$false

# Enable Jumbo Packets on SMB NICs
Write-Host "Enabling Jumbo Packets on SMB NICs" -ForegroundColor Yellow
$SMB2_NICs = Get-NetAdapter | where InterfaceDescription -Like "Chelsio*"
Get-NetAdapterAdvancedProperty -Name $SMB2_NICs.InterfaceAlias -RegistryKeyword "*jumbopacket" | Set-NetAdapterAdvancedProperty -RegistryValue 9000
#endregion

# Mellanox NICs
# Rename Network Adapters
Write-Host "Renaming Mellanox Adapters" -ForegroundColor Yellow
Get-NetAdapter | where InterfaceDescription -Like "Mellanox*" | where Name -Like "SLOT * *" | Rename-NetAdapter -NewName "SDN 2"
Get-NetAdapter | where InterfaceDescription -Like "Mellanox*" | where Name -Like "SLOT *" | Rename-NetAdapter -NewName "SDN 1"

# Disable Unused 1Gb NICs
Write-Host "Disabling Unused 1Gb NICs" -ForegroundColor Yellow
Get-NetAdapter -Name $UnusedNICS.Name | Disable-NetAdapter -Confirm:$false

# Disable IPv6 on all NICs
Write-Host "Disabling IPv6 Binding on all NICs" -ForegroundColor Yellow
$NICIPv6 = Get-NetAdapter | where ConnectorPresent -eq $true
Disable-NetAdapterBinding -Name $NICIPv6.Name -ComponentID ms_tcpip6

# Disable "Register in DNS" Setting on Relevant NICs
Write-Host "Disabling DNS Registration on all relevant NICs" -ForegroundColor Yellow
$RegisterDNS_NICs = Get-NetAdapter -Name "SMB 1 (VLAN $SMB1_VLAN)", "SMB 2 (VLAN $SMB2_VLAN)", "SDN 2"
Set-DnsClient -InterfaceAlias $RegisterDNS_NICs.Name -RegisterThisConnectionsAddress $False

# Add Hyper-V and Failover Clustering Features
Write-Host "Adding Hyper-V and Failover Clustering Features" -ForegroundColor Yellow
Add-WindowsFeature -Name Hyper-V -IncludeManagementTools
Add-WindowsFeature -Name Failover-Clustering -IncludeManagementTools
Add-WindowsFeature -Name RSAT-Clustering -IncludeAllSubFeature

# Management Networking
#region **MANGEMENT NETWORKING TO ALLOW ADDITION OF HOST TO VMM**
# Configure IP Address and DNS Settings on SDN 1 NIC
Write-Host "Configuring IP and DNS on SDN 1 NIC" -ForegroundColor Yellow
New-NetIPAddress -InterfaceAlias "SDN 1" -IPAddress $ManagementIP -PrefixLength 24 -DefaultGateway $ManagementDefaultGW
Set-DnsClientServerAddress -InterfaceAlias "SDN 1" -ServerAddresses $DNS1, $DNS2
#endregion

# Set Local Administrator Password Never Expires Flag to True
Write-Host "Setting Local Adinistrator Password to Never Expire" -ForegroundColor Yellow
$User.UserFlags.value = $User.UserFlags.value -bor 0x10000
$User.CommitChanges()

# Enable Remote Desktop
Write-Host "Enabling Remote Desktop" -ForegroundColor Yellow
set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server'-name "fDenyTSConnections" -Value 0
set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp' -name "UserAuthentication" -Value 1

# Setting Windows Time Service Startup to Automatic
Write-Host "Setting Windows Time Service Startup to Automatic" -ForegroundColor Yellow
Set-Service W32Time -StartupType Automatic
Start-Service W32Time

# Disable Windows Firewall
Write-Host "Disabling Windows Firewall" -ForegroundColor Yellow
Set-NetFirewallProfile -All -Enabled False

#Enable VMQs MEL NICs
Write-Host "Enabling and Configuring VMQ on Mellanox NICs" -ForegroundColor Yellow
Set-netadaptervmq -name "SDN 1" -ENABLED $TRUE
Set-netadaptervmq -name "SDN 2" -ENABLED $TRUE

#SUM of Queues Mel NICS Sockets 2 - 20 cores (if NVGRE used)
Set-NetAdapterVMQ –Name "SDN 1" –BaseProcessorNumber 12 –MaxProcessors 4
Set-NetAdapterVMQ –Name "SDN 2" –BaseProcessorNumber 16 –MaxProcessors 4
#endregion

#Chelsio Offload Settings (Compute& SOFS)
Write-Host "Modifying Global TCP/IP Offload Settings" -ForegroundColor Yellow
Set-NetOffloadGlobalSetting -NetworkDirectAcrossIPSubnets Allowed

# Disable IE ESC
Write-Host "Disabling IE ESC" -ForegroundColor Yellow
$AdminKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}"
$UserKey = "HKLM:\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}"
Set-ItemProperty -Path $AdminKey -Name "IsInstalled" -Value 0
Set-ItemProperty -Path $UserKey -Name "IsInstalled" -Value 0
Stop-Process -Name Explorer

# Rename Administrator Account and Reboot
Write-Host "Renaming Local Administrator Account to $Renamed_Admin" -ForegroundColor Green
$admin=[adsi]"WinNT://./Administrator,user"
$admin.psbase.rename("$Renamed_Admin")
Shutdown /r /t 15

When you run the script, you will be asked to provide answers to the following questions:

• Please enter a new hostname for this server
• Please enter the last octet for this servers IP addressing

Apply all available Windows Update to both nodes – this may take several passes

Set Required Local Administrator Permissions

Create an account in Active Directory to be used as the Hyper-V Run As account e.g. domain\HYPV_RAA. This is the account that VMM will use when interacting with the Hyper-V hosts and cluster.

Add this account to the local administrators group on all Hyper-V nodes

Add Hyper-V hosts into SCVMM for management

Launch your SCVMM console and connect to the cluster you created earlier in this guide.

Our Hyper-V hosts should now be in a good place to add to SCVMM. Before we do that, we’ll want to create a new host group. Host groups can either inherit the settings of their parent group or can be customised as required. By right-clicking a host group and selecting “Properties”, you have control over the groups, Placement Rules, Host Reserves, Dynamic Optimisation, Network and Storage settings.

Create a Hyper-V Host Group

Browse to “Fabric”, “Servers”, right-click “All Hosts” and select “Create Host Group”

You should now have a host group called “New host group”, you can rename this group by right-clicking it and selecting “Rename”

Now depending on your environment, you may want to jump into the properties and modify the default settings mentioned above, for this guide though, we’ll stick with the default. But hey, now at least you’ve got the option without having to modify the parent group 🙂

Add Hyper-V Hosts to new Host Group

There are loads of different ways and points in the deployment timeline we can do this step, I have always preferred to have my final management networking in place before adding my hosts to a cluster…it’s just a preference of mine.

Right-click your new host group and select “Add Hyper-V Hosts and Clusters”

Select “Windows Server computers in a trusted Active Directory domain” and select “Next”

On the credentials screen you’ll be asked for the user account to be used when SCVMM is working with the Hyper-V hosts. We created this account earlier in the guide but will now want to create a SCVMM Run As account for it.

Click “Browse” and when prompted for a Run As Account, select “Create Run As Account”

Enter a “Name” and optionally a “Description” for the account

Under “Username”, enter the account you created earlier in the following format domain\username. Now enter and confirm the password for the account

Keep “Validate domain credentials” checked as this will confirm you’ve entered things correctly and save you a potential headache later. Click “Finish” and then “Next”

On the “Discovery Scope” page, either type the FQDN of your hosts into the box provided or generate an AD query to search for them e.g. HYPV- Once you have entered your hosts, click “Next”

Select the hosts you want to add to SCVMM and click “Next”

Make sure the correct host group is selected on the “Host Settings” page and click “Next” and then “Finish”

A SCVMM job should kick off that installs an agent on each host to allows it to be managed. Providing you’re following this guide to the letter :), you’ll receive a warning about the fact you don’t have MPIO enabled, that’s OK here as we’re using SMB Multichannel to Scale-Out file server shares. With that in mind, the error can be ignored.

Create and Apply SCVMM Logical Networking to Hyper-V Hosts

For this deployment we’re using two 10gb NICs for our management, SDN (upcoming part of the guide) and tenant VM traffic. We’re using a separate 2 x 10gb NICs for our SMB storage and Live migration traffic. The storage NICs have already been configured and given relevant IPs by our PowerShell script but now we want to configure our Logical Switch and host OS management NIC. With that in mind, let’s get cracking.

Create Logical Network

The first thing we want to do is create a logical network for our host management network but before we go into that, what IS a logical network? Let Microsoft explain 🙂

“There are different types of networks in most organizations, including corporate networks, management networks, and others. These networks might be isolated physically or virtually using network virtualization and virtual LANs (VLANs). In VMM each of these networks is defined as a logical network. Logical networks are logical objects that mirror your physicals networks and are used to model your VMM network fabric.

When you create logical networks to model your environment you assign them properties that match your physical environment. You specify the type of network, the network sites associated with the logical network, and static address pools if you’re not using DHCP to assign IP addresses to VMs you create in the network sites.

When you provision virtualization hosts in the VMM fabric, you associate physical adapters on those hosts with logical networks.

From <https://technet.microsoft.com/en-us/system-center-docs/vmm/manage/manage-network-logical-networks> “

Now that you’re an expert on the concepts of logical networking, let’s crack on 🙂

Navigate to “Fabric”, “Networking”, right-click on “Logical Networks” and select “Create Logical Network”

Enter a name for the logical network and click “Next”

Select “One connected network” and “Create a VM network with the same name…”, now click “Next”

Click “Add” under network sites
Place a tick in the host group that holds your Hyper-V hosts
Enter the VLAN ID and IP subnet for your management network (SEE NOTE BELOW), now click “Next” and “Finish”

NOTE:  I strongly suggest using the native VLAN set on the TOR switch port so can put a VLAN ID of 0 in VMM for the management network. This makes for the most seamless logical network deployment to your hosts.

Create Management Network Static IP Pool

Although this step isn’t required for your hosts, it will be required if we want to put any VMs on our management network, for example, our SDN VMs (upcoming section in this series). When creating a Management network IP Pool, I tend to favour using a range that sits above our host IPs as this ensures if the day comes where I’m expanding the cluster, VMM hasn’t given out IP .108 when I want to use it for HYPV8, I’m picky like that 🙂

Right-click the Logical Network you just created and select “Create IP Pool”

Enter a name for the IP Pool. I tend to go with the Logical Network Name – IP Pool
Select the logical network you just created from the drop-down and click “Next”

Select “Use an existing network site” and select the network site and IP subnet you created earlier from their drop-downs. Now click “Next”

Change your starting and ending IP addresses so suit your environment and click “Next”.

NOTE:  You can also reserve IPs for other purposes here if you have a need to do so

Under “Default gateways”, click “Insert” and type in the gateway IP for your management network, generally .1. Now click “Next”

Under DNS servers, click “Insert” and type in the IP address of the DNS server on your management network. You can add multiple DNS servers here by repeating the process

Under “Connection specific DNS suffix”, type the FQDN of your management domain (the domain your hosts are members of). Now click “Next”, “Next” and “Finish”

Create Uplink Port Profile

Before we can create the logical switch we’ll be applying to our Hyper-V hosts, we have to create an Uplink Port Profile. It’s this profile that allows us to tell the Hyper-V switch that we’re using teaming on our physical NICs and what logical networks we want to bind to this physical team.

Navigate to “Fabric”, “Networking”, right-click “Port Profiles” and click “Create Hyper-V Port Profile”

Enter a name for your Uplink Port Profile, I tend to use Environment Name, Production Uplink. So in this case it would be “Demo Lab Production Uplink”

Click on “Uplink port profile” and leave the defaults for “Load Balancing Algorithm” and “Teaming mode”, now click “Next”

NOTE: You can read more on the different teaming configurations HERE (SOOOO much detail)

Select the Management Network you created earlier and click “Next” and “Finish”

Create Logical Switch

Now that we have our Logical Network, IP Pool and Uplink Port Profile, we can create our Logical Switch. This has been improved in Server 2016 and even lets us configure our host OS Network Adapters within the switch. In 2012 R2 this had to be done on a per host basis. Don’t worry if I’ve lost you, it’s about to make much more sense 🙂

Think of a logical switch like a container, a container that holds the information on:

• The Uplink Mode – Options are No team, LBFO NIC Teaming and SET (Switch Embedded Teaming). For this demo we’ll be using SET
• Minimum Bandwidth Mode – Options are Default, Weight, Absolute and none. For this demo we’ll be using Weight
• Enable SR-IOV – This can only be done at the point of logical switch creation, not after the fact.
• Tell the logical switch it’s managed by a SDN Network Controller service. We’ll be setting this up later in this series
• Add/remove extensions from the logical switch
• Specified port classifications and virtual ports
• Configured uplinks and host OS network adapters

Navigate to “Fabric”, “Networking”, right-click “Logical Switches” and select “Create Logical Switch”

Enter a name for your logical switch, choose “Embedded Team” as the Uplink Mode and click “Next”

Choose the default “Weight” for the “Minimum bandwidth mode” and click “Next”

IMPORTNANT: Untick all extensions as they’ll block our SDN deployment later in the series. Now click “Next”

For this demo, we’re going to add the Port Classification and Virtual Port for “Host management”. These can be added or removed at any time so don’t worry too much about planning too far ahead.

Click “Add” under “Virtual Ports” and click “Browse” under “Port Classification”

Select “Host management” and click “OK”

Tick “Include a Hyper-V virtual network adapter port profile…”, click browse, select “Host management” and click “OK”, then “OK”

Review your choices and click “Next”

On the “Uplinks” screen, click “Add” and choose “Existing Uplink Port Profile”

Choose the Uplink Port Profile you created earlier and click “OK”

With the Uplink highlighted, click “New virtual network adapter”

Type a name for this adapter, we’re using it for management, so lets’ call it eh…..”Management” 🙂

Click “Browse”, select the Logical Network you created earlier and click “OK”

Tick “This virtual network adapter will be used for host management” and “Inherit connection settings from…”

Select “Host management” from the “Port Profile” drop-down and click “Next” and “Finish”

Apply Logical Networking to Hyper-V Hosts

Now that we’ve created our logical networking, we can apply it to the Hyper-V hosts.

Navigate to “Fabric”, “Servers” and click on the host group that holds your Hyper-V hosts.

Right click on your first Hyper-V host and select “Properties”, then select “Virtual Switches”

Click “New Virtual Switch” and select “New Logical Switch”

Select the Logical Switch you created earlier from the “Logical Switch” drop-down

Under “Physical Adapters” select the two NICs you want as part of your NIC team and make sure the Uplink against them is the one you created earlier. Click “Add” to configure the second NIC

Click “OK”. A SCVMM job will now be kicked off that will deploy a logical switch and a virtual host OS NIC to the host. Part of this job will pass the IP details from your configured physical management NIC to new virtual host OS adapter.

NOTE: There is no additional configuration required here as it’s all contained within the Logical Switch

Repeat the above steps for all Hyper-V hosts that will be members of your cluster

Create Hyper-V Cluster

Before creating the cluster, do the following:

Validate the cluster configuration by running the following command:

Test-Cluster –Node SERVER1, SERVER2

The cluster validation report is saved to the node you run the PowerShell from and in the following location: C:\Windows\Cluster\Reports\Validation Report DATE & TIME.htm

Assuming the validation has succeeded, move on and create the cluster below or attend to any issues if not.

Create a cluster using the PowerShell below:

New-Cluster –Name MyCluster –Node Server1, Server2 –StaticAddress 192.168.1.12 -NoStorage

Once the cluster has been successfully created, go back to your SCVMM console and refresh one of the Hyper-V hosts, you’ll see the cluster object appear in your host group.

Add AD Permissions for Cluster CNO

I covered this earlier as part of the SQL cluster deployment guide, you’ll find it HERE

Create a Cluster Quorum

I covered this earlier as part of the SQL cluster deployment guide, you’ll find it HERE

Configure Cluster Networks

Launch “Failover Cluster Manager” and browse to “Networks” on the left navigation pane. Both your SMB networks usage should be listed as “Cluster and Client”

Configure Live Migration Settings

Click on “Live Migration Settings” on “Actions” pane on the right side of the window. Make sure only the SMB networks are ticked and both are at the top, now click “OK”

Run the following PowerShell from your SCVMM server. Update the Hyper-V hostnames as required and run once for each host

Invoke-Command -ComputerName (Get-SCVMHost).Name {Set-VMHost -VirtualMachineMigrationPerformanceOption SMB}

Add SMB File Shares to Hyper-V Cluster

As part of our SOFS cluster deployment in Part 3 of the series, we created a SMB file share. We’re now in a position to add that file share to our shiny new Hyper-V cluster for storing our running VM files.

In SCVMM, navigate to “Fabric”, “Servers”, the “host group” that holds your cluster, right-click your cluster and select “Properties”

Click on “File Share Storage”, “Add…” and select the file share from the drop-down. Now click “OK” twice.

You can now deploy Virtual Machines to your Scale-Out file share over SMB 3

So all going well, you now have a working Hyper-V and Storage Spaces environment. For most people that would be enough but not for us, we’ve still got the Host Guardian Service and SDN to configure yet 🙂

Join me in part 5 of the series where we’ll go through the Host Guardian Service deployment