Balancing VMs Across Hosts in VMware Clusters: A Custom Solution

In virtualized environments, evenly distributing virtual machines (VMs) across hosts in a cluster is crucial for optimal performance and resource utilization. VMware’s Distributed Resource Scheduler (DRS) is designed to automate this process. However, some users have encountered issues where DRS does not balance VMs as expected.

One user on Reddit highlighted this problem:

This is a known problem – if the DRS score of VMs is good, DRS will not migrate them to the empty host. I even opened a support ticket regarding this and was told that it is expected behaviour and that the toggle ‘Try to balance number of VMs per host’ does not actually do anything in the current DRS scheduler. It sucks…

Another user confirmed:

That is correct in newer versions of DRS. The only thing that matters is the VM’s happiness score and not the host. Host can be pegged but DRS wouldn’t do anything.

To address this issue, I developed a PowerShell script using VMware PowerCLI that manually balances VMs across hosts within a cluster. This script ensures that no host exceeds a specified maximum number of VMs, providing a more balanced and efficient environment.

The Solution

The solution consists of three main scripts:

• Balance-VMs.ps1: Balances the VMs across hosts in the cluster based on the specified criteria.
• Generate-EncryptionKey.ps1: Generates a secure encryption key for encrypting and decrypting vCenter credentials.
• Create-VCenterCredentials.ps1: Encrypts and stores vCenter credentials using the generated encryption key.

Part 1: Generating the Encryption Key

To securely store your vCenter credentials, you first need to generate an encryption key.

Generate-EncryptionKey.ps1

<#
.SYNOPSIS
    Generates a secure encryption key for encrypting and decrypting vCenter credentials.

.DESCRIPTION
    This script creates a 256-bit (32-byte) encryption key using a cryptographically secure random number generator.
    The key is saved to a specified file path and should be securely stored with restricted access permissions.

.PARAMETER EncryptionKeyPath
    The file path where the encryption key will be stored. Default is "C:\Secure\Credentials\encryptionKey.key".

.EXAMPLE
    .\Generate-EncryptionKey.ps1
    Generates an encryption key and saves it to the default path.

.EXAMPLE
    .\Generate-EncryptionKey.ps1 -EncryptionKeyPath "D:\Keys\MyEncryptionKey.key"
    Generates an encryption key and saves it to the specified path.

.AUTHOR
    virtualox

.GITHUB_REPOSITORY
    https://github.com/virtualox/VM-Balancer

.LICENSE
    This script is licensed under the GPL-3.0 License. See the LICENSE file for more information.

.NOTES
    - Ensure the encryption key file is stored in a secure location with restricted access.
    - This key is required for both encrypting and decrypting the vCenter credentials.
    - Do not share the encryption key file publicly or store it in insecure locations.
#>

[CmdletBinding()]
param (
    [string]$EncryptionKeyPath = "C:\Secure\Credentials\encryptionKey.key"
)

# Function to check if the encryption key already exists
function Test-EncryptionKeyExists {
    param (
        [string]$Path
    )
    return (Test-Path -Path $Path)
}

# Function to generate a secure encryption key
function Generate-EncryptionKey {
    param (
        [string]$Path
    )
    try {
        # Create a 32-byte (256-bit) key
        $key = New-Object byte[] 32

        # Use the appropriate RNG method based on .NET version
        if ([System.Security.Cryptography.RandomNumberGenerator].GetMethod('Fill', [Type[]]@([Byte[]]))) {
            # For .NET Core and .NET 5+
            [System.Security.Cryptography.RandomNumberGenerator]::Fill($key)
        }
        else {
            # For .NET Framework
            [System.Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($key)
        }

        # Save the key to the specified path
        Set-Content -Path $Path -Value $key -Encoding Byte -Force

        Write-Output "Encryption key successfully generated and saved to '$Path'."
    }
    catch {
        Write-Error "Failed to generate encryption key: $_"
        exit 1
    }
}

# Function to check if running as administrator
function Test-IsAdministrator {
    $currentUser = [Security.Principal.WindowsIdentity]::GetCurrent()
    $principal = New-Object Security.Principal.WindowsPrincipal($currentUser)
    return $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
}

# Main Execution

# Check if running as administrator
if (-not (Test-IsAdministrator)) {
    Write-Warning "You need to run this script as an Administrator to set file permissions."
    exit 1
}

if (Test-EncryptionKeyExists -Path $EncryptionKeyPath) {
    Write-Warning "Encryption key already exists at '$EncryptionKeyPath'."

    do {
        $userInput = Read-Host "Do you want to overwrite the existing key? (Y/N)"
    } until ($userInput -match '^[YyNn]$')

    if ($userInput -ne 'Y' -and $userInput -ne 'y') {
        Write-Output "Operation cancelled by the user."
        exit
    }
}

# Ensure the directory exists
$directory = Split-Path -Path $EncryptionKeyPath -Parent
if (-not (Test-Path -Path $directory)) {
    try {
        New-Item -Path $directory -ItemType Directory -Force | Out-Null
        Write-Output "Created directory '$directory'."
    }
    catch {
        Write-Error "Failed to create directory '$directory': $_"
        exit 1
    }
}

# Generate the encryption key
Generate-EncryptionKey -Path $EncryptionKeyPath

# Secure the encryption key file by setting appropriate permissions
try {
    $currentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name

    # Secure the encryption key file
    $aclFile = Get-Acl -Path $EncryptionKeyPath

    # Remove all existing permissions except for the current user
    $accessRules = $aclFile.Access | Where-Object { $_.IdentityReference -ne $currentUser }
    foreach ($rule in $accessRules) {
        $aclFile.RemoveAccessRule($rule)
    }

    # Define the access rule: Only the current user has full control
    $accessRuleFile = New-Object System.Security.AccessControl.FileSystemAccessRule(
        $currentUser,
        [System.Security.AccessControl.FileSystemRights]::FullControl,
        [System.Security.AccessControl.InheritanceFlags]::None,
        [System.Security.AccessControl.PropagationFlags]::None,
        [System.Security.AccessControl.AccessControlType]::Allow
    )
    $aclFile.SetAccessRuleProtection($true, $false)
    $aclFile.SetAccessRule($accessRuleFile)
    Set-Acl -Path $EncryptionKeyPath -AclObject $aclFile

    Write-Output "Set restricted permissions on '$EncryptionKeyPath'."
}
catch {
    Write-Warning "Failed to set permissions on '$EncryptionKeyPath'. Please ensure it is secured properly."
}

Notes:

• Ensure the encryption key file is stored in a secure location with restricted access permissions.
• This key is required for both encrypting and decrypting the vCenter credentials.
• Do not share the encryption key file publicly or store it in insecure locations.

Part 2: Encrypting and Storing vCenter Credentials

Next, you’ll encrypt and store your vCenter credentials using the encryption key generated in Part 1.

Create-VCenterCredentials.ps1

<#
.SYNOPSIS
    Encrypts and stores vCenter credentials using a shared encryption key.

.DESCRIPTION
    This script prompts the user to enter vCenter credentials, encrypts the password using a predefined encryption key,
    and saves the encrypted password along with the username to a specified file. This setup allows multiple administrators
    or automated tasks to access shared credentials securely.

.AUTHOR
    virtualox

.GITHUB_REPOSITORY
    https://github.com/virtualox/VM-Balancer

.LICENSE
    This script is licensed under the GPL-3.0 License. See the LICENSE file for more information.

.USAGE
    .\Create-VCenterCredentials.ps1

.NOTES
    - Ensure that the encryption key has been generated using Generate-EncryptionKey.ps1 before running this script.
    - The encryption key path and credential storage path must match those used in Balance-VMs.ps1.
    - Store the encrypted credentials file in a secure location with restricted access.
#>

# === Configuration Variables ===

# Path to the encryption key file
$encryptionKeyPath = "C:\Secure\Credentials\encryptionKey.key" # <-- Must match the key generated by Generate-EncryptionKey.ps1

# Path where the encrypted credentials will be stored
$credentialPath = "C:\Secure\Credentials\vcCredentials.xml"    # <-- Update this path as needed

# === End of Configuration Variables ===

# Function to check if the encryption key exists
function Test-EncryptionKeyExists {
    param (
        [string]$Path
    )
    return (Test-Path -Path $Path)
}

# Function to encrypt and store credentials
function Encrypt-And-Store-Credentials {
    param (
        [string]$KeyPath,
        [string]$CredPath
    )
    try {
        # Prompt user for vCenter credentials
        $credential = Get-Credential -Message "Enter your vCenter credentials"
        $username = $credential.Username
        $password = $credential.Password

        # Read the encryption key
        $key = Get-Content -Path $KeyPath -Encoding Byte

        # Encrypt the password
        $encryptedPassword = $password | ConvertFrom-SecureString -Key $key

        # Create a custom object to store username and encrypted password
        $credentialObject = [PSCustomObject]@{
            Username          = $username
            EncryptedPassword = $encryptedPassword
        }

        # Save the credential object to the specified path
        $credentialObject | ConvertTo-Json | Set-Content -Path $CredPath -Force

        Write-Output "vCenter credentials have been encrypted and stored successfully at '$CredPath'."
    }
    catch {
        Write-Error "Failed to encrypt and store credentials: $_"
        exit 1
    }
}

# Function to secure the credentials file by setting appropriate permissions
function Secure-CredentialsFile {
    param (
        [string]$Path
    )
    try {
        $currentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name

        # Secure the credentials file
        $aclFile = Get-Acl -Path $Path

        # Remove all existing permissions except for the current user
        $accessRules = $aclFile.Access | Where-Object { $_.IdentityReference -ne $currentUser }
        foreach ($rule in $accessRules) {
            $aclFile.RemoveAccessRule($rule)
        }

        # Define the access rule: Only the current user has full control
        $accessRuleFile = New-Object System.Security.AccessControl.FileSystemAccessRule(
            $currentUser,
            [System.Security.AccessControl.FileSystemRights]::FullControl,
            [System.Security.AccessControl.InheritanceFlags]::None,
            [System.Security.AccessControl.PropagationFlags]::None,
            [System.Security.AccessControl.AccessControlType]::Allow
        )
        $aclFile.SetAccessRuleProtection($true, $false)
        $aclFile.SetAccessRule($accessRuleFile)
        Set-Acl -Path $Path -AclObject $aclFile

        Write-Output "Set restricted permissions on '$Path'."
    }
    catch {
        Write-Warning "Failed to set permissions on '$Path'. Please ensure it is secured properly."
    }
}

# Main Execution

# Check if encryption key exists
if (-not (Test-EncryptionKeyExists -Path $encryptionKeyPath)) {
    Write-Error "Encryption key not found at '$encryptionKeyPath'. Please generate it using Generate-EncryptionKey.ps1 before storing credentials."
    exit 1
}

# Ensure the directory exists
$directory = Split-Path -Path $credentialPath -Parent
if (-not (Test-Path -Path $directory)) {
    try {
        New-Item -Path $directory -ItemType Directory -Force | Out-Null
        Write-Output "Created directory '$directory'."
    }
    catch {
        Write-Error "Failed to create directory '$directory': $_"
        exit 1
    }
}

# Encrypt and store the credentials
Encrypt-And-Store-Credentials -KeyPath $encryptionKeyPath -CredPath $credentialPath

# Secure the credentials file
Secure-CredentialsFile -Path $credentialPath

Notes:

• Ensure that the encryption key has been generated before running this script.
• The encryption key path and credential storage path must match those used in Balance-VMs.ps1.
• Store the encrypted credentials file in a secure location with restricted access.

Part 3: Balancing VMs Across Hosts

Finally, the main script performs the VM balancing operation.

Balance-VMs.ps1

<#
.SYNOPSIS
    Balances VMs across hosts in a vCenter cluster, ensuring no host exceeds the specified VM limit.

.DESCRIPTION
    This script connects to a vCenter server using encrypted credentials, identifies overloaded hosts within a specified cluster,
    and migrates VMs to underloaded hosts to maintain a balanced distribution. It excludes specified VMs from migration
    based on their names or assigned tags. Additionally, it includes an enhanced dry-run mode to simulate migrations safely.

.AUTHOR
    virtualox

.GITHUB_REPOSITORY
    https://github.com/virtualox/VM-Balancer

.LICENSE
    This script is licensed under the GPL-3.0 License. See the LICENSE file for more information.

.USAGE
    .\Balance-VMs.ps1 [-DryRun]

.PARAMETER DryRun
    Executes the script in simulation mode without performing actual migrations. Useful for testing and verifying actions.

.NOTES
    - Ensure that the encryption key and encrypted credentials have been set up using Generate-EncryptionKey.ps1 and Create-VCenterCredentials.ps1.
    - Adjust the configuration variables and exclusion criteria within the script as needed.
    - The script requires appropriate permissions to migrate VMs within the specified vCenter cluster.
#>

param (
    [switch]$DryRun
)

# === Configuration Variables ===

# Path to the encryption key file
$encryptionKeyPath = "C:\Secure\Credentials\encryptionKey.key"  # <-- Must match the key generated by Generate-EncryptionKey.ps1

# Path to the encrypted credentials file
$credentialPath = "C:\Secure\Credentials\vcCredentials.xml"     # <-- Must match the credential path used in Create-VCenterCredentials.ps1

# vCenter Server details
$vcServer = "your_vcenter_server"       # <-- Replace with your vCenter server name or IP
$clusterName = "YourClusterName"        # <-- Replace with your cluster name

# VM balancing settings
$maxVMsPerHost = 60                     # Maximum number of VMs per host

# Exclusion Settings

## Name-Based Exclusion
# Specify exact VM names or use wildcards for patterns
$excludeVMNames = @(
    "cp-replica-*",    # Exclude VMs with names starting with 'cp-replica-'
    "horizon-*",       # Exclude VMs managed by Horizon (assuming they start with 'horizon-')
    "*-replica*",      # Exclude any VM containing '-replica' in its name
    "Important-VM1",   # Exclude specific VM by exact name
    "Critical-VM2"     # Add more as needed
)

## Tag-Based Exclusion
# Specify the names of tags assigned to VMs that should be excluded
$excludeVMTags = @(
    "DoNotMigrate",
    "Infrastructure",
    "VDI"
)

# === End of Configuration Variables ===

# Import the PowerCLI module
Import-Module VMware.PowerCLI -ErrorAction Stop

# Suppress certificate warnings (optional, remove if not needed)
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

# Function to check if the encryption key exists
function Test-EncryptionKeyExists {
    param (
        [string]$Path
    )
    return (Test-Path -Path $Path)
}

# Function to check if the credentials file exists
function Test-CredentialsFileExists {
    param (
        [string]$Path
    )
    return (Test-Path -Path $Path)
}

# Function to retrieve credentials
function Get-Credentials {
    param (
        [string]$KeyPath,
        [string]$CredPath
    )
    try {
        # Read the encryption key
        $key = Get-Content -Path $KeyPath -Encoding Byte

        # Read the encrypted credentials JSON
        $credentialJson = Get-Content -Path $CredPath -Raw | ConvertFrom-Json

        $username = $credentialJson.Username
        $encryptedPassword = $credentialJson.EncryptedPassword

        # Decrypt the password
        $securePassword = $encryptedPassword | ConvertTo-SecureString -Key $key

        # Create PSCredential object
        $cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $securePassword

        return $cred
    }
    catch {
        Write-Error "Failed to retrieve and decrypt credentials: $_"
        exit 1
    }
}

# Function to check if the host is eligible (connected and not in maintenance mode)
function Is-HostEligible {
    param (
        $VMHost
    )
    return ($VMHost.ConnectionState -eq 'Connected' -and -not $VMHost.ExtensionData.Runtime.InMaintenanceMode)
}


# Function to apply exclusion filters to VMs
function Apply-ExclusionFilters {
    param (
        [VMware.VimAutomation.ViCore.Impl.V1.Inventory.VirtualMachineImpl]$VM
    )
    # Name-Based Exclusion
    foreach ($pattern in $excludeVMNames) {
        if ($VM.Name -like $pattern) {
            return $false
        }
    }

    # Tag-Based Exclusion
    foreach ($tag in $excludeVMTags) {
        if ($VM.Tags -contains $tag) {
            return $false
        }
    }

    return $true
}

# Function to secure the script execution
function Verify-Security {
    # Ensure encryption key exists
    if (-not (Test-EncryptionKeyExists -Path $encryptionKeyPath)) {
        Write-Error "Encryption key not found at '$encryptionKeyPath'. Please generate it using Generate-EncryptionKey.ps1 and store credentials using Create-VCenterCredentials.ps1."
        exit 1
    }

    # Ensure credentials file exists
    if (-not (Test-CredentialsFileExists -Path $credentialPath)) {
        Write-Error "Encrypted credentials file not found at '$credentialPath'. Please store credentials using Create-VCenterCredentials.ps1."
        exit 1
    }
}

# Function to get host object by name
function Get-HostByName {
    param (
        [string]$Name
    )
    return $hosts | Where-Object { $_.Name -eq $Name }
}

# Main Execution

# Verify security prerequisites
Verify-Security

# Retrieve credentials
$cred = Get-Credentials -KeyPath $encryptionKeyPath -CredPath $credentialPath

# Connect to vCenter
try {
    Connect-VIServer -Server $vcServer -Credential $cred -ErrorAction Stop
    Write-Output "Successfully connected to vCenter Server '$vcServer'."
}
catch {
    Write-Error "Failed to connect to vCenter Server '$vcServer'. $_"
    exit 1
}

# Get the cluster object
$cluster = Get-Cluster -Name $clusterName -ErrorAction SilentlyContinue

if (-not $cluster) {
    Write-Error "Cluster '$clusterName' not found."
    Disconnect-VIServer -Server $vcServer -Confirm:$false
    exit 1
}

# Get all eligible hosts in the cluster
$hosts = Get-VMHost -Location $cluster | Where-Object { Is-HostEligible -VMHost $_ }

if ($hosts.Count -eq 0) {
    Write-Error "No eligible hosts found in cluster '$clusterName'. Ensure that hosts are not in Maintenance or Disconnected states."
    Disconnect-VIServer -Server $vcServer -Confirm:$false
    exit 1
}

# Create a hashtable to store host VM counts
$hostVMCounts = @{}

foreach ($vmHost in $hosts) {
    # Retrieve all VMs on the host
    $vmsOnHost = Get-VM -Location $vmHost -ErrorAction SilentlyContinue

    # Apply Exclusion Filters
    $vmsToConsider = $vmsOnHost | Where-Object { Apply-ExclusionFilters -VM $_ }

    # Count the number of VMs to consider for migration
    $vmCount = $vmsToConsider.Count
    $hostVMCounts[$vmHost.Name] = $vmCount
}

# Identify overloaded and underloaded hosts
$overloadedHosts = $hostVMCounts.GetEnumerator() | Where-Object { $_.Value -gt $maxVMsPerHost } | Sort-Object -Property Value -Descending
$underloadedHosts = $hostVMCounts.GetEnumerator() | Where-Object { $_.Value -lt $maxVMsPerHost } | Sort-Object -Property Value

if ($overloadedHosts.Count -eq 0) {
    Write-Output "All hosts have $maxVMsPerHost VMs or fewer. No balancing needed."
    Disconnect-VIServer -Server $vcServer -Confirm:$false
    exit 0
}

if ($underloadedHosts.Count -eq 0) {
    Write-Output "No hosts available with less than $maxVMsPerHost VMs to migrate VMs to."
    Disconnect-VIServer -Server $vcServer -Confirm:$false
    exit 0
}

# Start balancing
foreach ($overloaded in $overloadedHosts) {
    $overHostName = $overloaded.Key
    $overHostVMCount = $overloaded.Value
    $overHost = Get-HostByName -Name $overHostName

    $vmsToMoveCount = $overHostVMCount - $maxVMsPerHost

    if ($vmsToMoveCount -le 0) {
        continue
    }

    Write-Output "Host '$overHostName' is overloaded with $overHostVMCount VMs. Need to move $vmsToMoveCount VMs."

    # Get VMs on the overloaded host, excluding replicas and tagged VMs, sorted by power state (optional: adjust sorting as needed)
    $vmsOnOverHost = Get-VM -Location $overHost | Where-Object { Apply-ExclusionFilters -VM $_ } | Sort-Object -Property PowerState

    foreach ($vm in $vmsOnOverHost) {
        # Check again if there are still VMs to move
        if ($vmsToMoveCount -le 0) {
            break
        }

        # Find a target host that is underloaded and can host the VM
        $targetHostEntry = $underloadedHosts | Where-Object { $_.Value -lt $maxVMsPerHost } | Sort-Object -Property Value | Select-Object -First 1

        if ($targetHostEntry) {
            $targetHostName = $targetHostEntry.Key
            $targetHostObj = Get-HostByName -Name $targetHostName

            # Additional check to ensure the target host is still eligible
            if ($targetHostObj.ConnectionState -ne 'Connected' -or $targetHostObj.ExtensionData.Runtime.InMaintenanceMode) {
                Write-Warning "Target host '$targetHostName' is no longer eligible. Skipping."
                # Remove the host from underloaded list
                $underloadedHosts = $underloadedHosts | Where-Object { $_.Key -ne $targetHostName }
                continue
            }

            if ($DryRun) {
                Write-Output "[Dry-Run] Would migrate VM '$($vm.Name)' from '$overHostName'."

                # Simulate the migration by updating the VM counts
                $hostVMCounts[$overHostName] -= 1
                $hostVMCounts[$targetHostName] += 1

                # Update the underloaded hosts list if the target host reaches the limit
                if ($hostVMCounts[$targetHostName] -ge $maxVMsPerHost) {
                    $underloadedHosts = $underloadedHosts | Where-Object { $_.Key -ne $targetHostName }
                }

                $vmsToMoveCount -= 1
            }
            else {
                Write-Output "Migrating VM '$($vm.Name)' from '$overHostName' to '$targetHostName'. (Hardware Version: $($vm.HardwareVersion))"

                try {
                    # Perform the migration
                    Move-VM -VM $vm -Destination $targetHostObj -ErrorAction Stop

                    # Update the VM counts
                    $hostVMCounts[$overHostName] -= 1
                    $hostVMCounts[$targetHostName] += 1

                    # Update the underloaded hosts list if the target host reaches the limit
                    if ($hostVMCounts[$targetHostName] -ge $maxVMsPerHost) {
                        $underloadedHosts = $underloadedHosts | Where-Object { $_.Key -ne $targetHostName }
                    }

                    $vmsToMoveCount -= 1
                }
                catch {
                    Write-Warning "Failed to migrate VM '$($vm.Name)': $_"
                }
            }
        }
        else {
            Write-Warning "No available target hosts with less than $maxVMsPerHost VMs to migrate VM '$vm.Name'."
            break
        }
    }
}

Write-Output "VM balancing complete."

# Disconnect from vCenter
Disconnect-VIServer -Server $vcServer -Confirm:$false

Key Features:

• Connection and Authentication: Uses encrypted credentials to connect to the specified vCenter server.
• Host and VM Retrieval: Gathers information about hosts and VMs in the specified cluster.
• Exclusion Criteria: Excludes VMs from migration based on name patterns and assigned tags.
• Balancing Logic: Identifies overloaded hosts (those exceeding maxVMsPerHost) and migrates VMs to underloaded hosts.
• Dry-Run Mode: The -DryRun parameter allows you to simulate the balancing process without making actual changes.

Usage Example:

To perform a dry run:

.\Balance-VMs.ps1 -DryRun

To execute the balancing operation:

.\Balance-VMs.ps1

Important Notes:

• Ensure that the encryption key and encrypted credentials are properly set up before running the script.
• Adjust the configuration variables and exclusion lists to fit your environment.
• The script requires appropriate permissions to perform VM migrations in vCenter.

Conclusion

Balancing VMs across hosts in a VMware cluster can improve performance and resource utilization. While DRS is designed to automate this process, it may not always behave as expected due to its focus on VM happiness scores. This custom script provides a way to manually enforce a more balanced distribution of VMs.

By tailoring the exclusion criteria and configuration settings, you can adapt the script to meet the specific needs of your environment. The inclusion of a dry-run mode also allows for safe testing before making actual changes.

Additional Resources

• GitHub Repository: virtualox/VM-Balancer
• License: This script is licensed under the GPL-3.0 License.