Use PowerCLI to manage users on an ESXi host

Starting with vSphere 6.0 VMware introduced a new set of ESXCLI commands to manage the life cycle of local accounts and permissions.

I have created ESXiAccountManagement.ps1 script that includes the following functions:

  • Get-ESXiAccount: List all local accounts and their role
  • Add-ESXiAccount: Create local accounts and assign one of the predefined roles
  • Set-ESXiAccount: Updates user’s Description, permission (one of the predefined roles) and most popular reset the user password.
  • Remove-ESXiAccount: Removes a local account
  • Get-ESXiAccountSecurity: Get ESXi host local account security settings and can also assist in checking events for account locked and bad logon events (past 1, 2 or 24 hrs)

Running the script

Make sure to run the Get-Help (Example: Get-Help Get-ESXiAccount –ShowWindow) command to understand all the parameters and switches. All these functions make use of the new ESXCLI commands so the script will only run against ESXi 6.x hosts. The script also assumes that you already have a connection to a vCenter, let’s walk through a few examples to show the script in action.

Get-ESXiAccount

A script function to list all local accounts, can be run against a host, cluster, or datacenter

Syntax

Get-ESXiAccount [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

EsxAcct0

Add-ESXiAccount

A script function to add/create a new local user. The script will first check if a User ID with the same name already exists before creating it. You are prompted for the user’s password during the script execution, this uses Get-Credential object which stores and relays the password as a secure string.

The –Permission parameter will show you a list of options, which are the predefined roles supported (Admin, ReadOnly, NoAccess)  to grant a user.

Syntax

Add-ESXiAccount [-Name] <String> [[-Description] <String>] [[-Permission] <Object>] [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

Creating a user account

Let’s create a monitoring service account needed on the ESXi host, user ID: sysmonitoring

EsxAcct1EsxAcct2

Note the following on this example:

  • I’m specifying the User ID, Description and Permission for the new user (sysmonitoring)
  • I was prompted for the password
  • User creation failed because the password I used did not meet the password quality control policy
  • The password policy in place is the default, and is displayed as part of the feedback on the error message

Let’s try that again using a stronger password

EsxAcct3

Set-ESXiAccount

A script function to reset a user’s password, update description, or update permission. Same as Add-ESXiAccount, you will be prompted for the new password when resetting a user’s password and the password strength must meet the policy requirements.

Syntax

Set-ESXiAccount [-Name] <String> [[-Description] <String>] [[-Permission] <Object>] [-ResetPassword ] [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

Recovering the root user on a ESXi host

The most popular use case for this is when you have forgotten the root user password on the ESXi host. You can stop using host profiles, SSH keys, etc, and just do it through PowerCLI!

EsxAcct4

EsxAcct5

Note the following on this example:

  • I was able to recover/reset the root password using PowerCLI
  • Password complexity applies, so keep that in mind or else you will get the error as the previous example when adding a new user

Changing user description and permissions

Now that I have created sysmonitoring user, I want to update the description and also found out that it needs admin privileges instead of ReadOnly (hypothetical example).

EsxAcct6

Remove-ESXiAccount

A script function to delete an ESXi local user, can be run against a host, cluster, or datacenter. This will remove user created accounts, not system accounts (root for example)

Syntax

Remove-ESXiAccount [-Name] <String> [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

Trying to remove a system account (root)

EsxAcct7

Note the following on this example:

  • System accounts are protected against deletion
  • Some of these scripts functions are powerful, use with caution

Removing sysmonitoring user ID (user created account)

Continuing with our example, I now don’t have the need to keep sysmonitoring any longer, so I shall remove him

EsxAcct8

Note the following on this example:

  • Sysmonitoring user was successfully removed
  • A listing of all local accounts on the ESXi host shows that sysmonitoring user is no longer present

Get-ESXiAccountSecurity

A script function to list account lockout policy, password quality control policy and to gather events for troubleshooting account lockouts and bad logons

Syntax

Get-ESXiAccountSecurity [[-EventsPastHrs] <Int32>] [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

-EventsPastHrs is the switch used to query for account lockout , and bad logon events. It has 3 predefined options (1, 2, 24) hours.

Troubleshooting account logon scenario

Let’s look at a hypothetical example:

  • Dellomsa was created and given an admin role
  • This account has stopped working, I cannot logon with it. Suspecting a bad password, I went ahead and did a password reset
  • After resetting the password, I’m still not able to login

To troubleshot further I then looked for account events

EsxAcct9

Note the following on this example:

  • System has the default settings for Lock Failures, Unlock Time, and complexity
  • I cannot login with dellomsa because its locked out, so I need to wait for the Unlock time to expire or use SSH to unlock it
  • I queried the events for the past hour and have identified the IP where the bad logons are coming from
  • The failed logon attempts (50 failed login attempts) is a global counter on the ESXi host for all bad logon attempts. I have been playing with this host, validating the scripts and writing this blog. I in fact just tried to logon 6 times with a bad password to get dellomsa user locked out. 5 attempts is default policy before locking out and 15 minutes to unlock.
  • If a local account is locked out, you will get a warning on the host in vCenter as well

EsxAcct10

As you can see using PowerCLI is a great way to manage and troubleshoot ESXi user accounts hope this has been helpful for you.

Posted in PowerShell

Verify new Spectre mitigation patches using PowerCli and vDocumentation

Intel recently announced that it has released microcode updates for 100 percent of its products launched in the past five years that require protection against Spectre/Meltdown. For virtualization admins using Intel Xeon chips, this would mean that servers with a Sandy Bridge or newer generation CPU now have a BIOS update available. Westmere and Nehalem are in Beta production status as of this writing for microcode updates.

We have published an update of vDocumentation that will assist you in validating your current environment for Spectre. The update (v2.4.1) includes enhanced versions of:

  • Get-ESXSpeculativeExecution
  • Get-VMSpeculativeExecution

UPDATE (3/28/2018) VMware recently updated security advisory VMSA-2018-0004 with the list of new patches, and Intel has also released production MCU for Westmere and Nehalem.

We have released v2.4.2 of vDocumentation to help validate your environment for these new patches. More details at the end of this post.

Get-ESXSpeculativeExecution

A better way of explaining the changes would be by demonstrating the results gathered with a few examples. First though, here are some dependencies that the script relies on, we will mention them here and cover them in more detail in paragraphs below.

The script relies on two CSV files:

  • BIOSUpdates.csv – This file is used to validate and provide guidance on the current installed BIOS version. If you do not have internet access when running the script, or wish to use a customized version of the file, then download it and use the –inputBiosFile parameter to specify the offline version path
  • Intel_MCU.csv– This file is used to report on the microcode status provided by Intel. Use the -inputMcuFile parameter to specify the offline version path

Both files are to up to date, and I will keep them current as Intel provides new updates. The BIOS update file contains information for Dell and HP servers, which is what I have in the environment I support. If you have different hardware and would like to share the pertinent information with the community, please reach out and I will update the file with your input.

SSH dependency: VMware KB52345 has all the details about the initial MCU patches that Intel released which caused reboot problems. In order to validate if the microcode revision running on an ESXi is affected you need to run a shell command; today this can only be accomplished through SSH from an automation standpoint. When running the script, you need to use the -UseSSH parameter, this uses Posh-SSH as an SSH client which you can install from the PowerShell gallery (Install-Module -Name Posh-SSH). With Posh-SSH you will be prompted for credentials, this uses Get-Credential and the PSCredential object which stores and relays the password as a secure string.

Spec0

Running the script

Make sure to run the Get-Help (Example: Get-Help Get-ESXSpeculativeExecution –ShowWindow) command to understand all the parameters and switches, we will focus more on what is gathered on this blogpost.

Syntax

Get-ESXSpeculativeExecution [[-esxi] <Object>] [[-cluster] <Object>] [[-datacenter] <Object>] [[-inputBiosFile] <Object>] [[-inputMcuFile] <Object>] [-UseSSH ] [-ReportOnVMs ] [-ExportCSV ] [-ExportExcel ] [-PassThru ] [[-folderPath] <Object>] [<CommonParameters>]

Looking at an ESXi host that has been patched for Spectre

Spec1

BIOS Guidance

BIOS Guidance relies on the BIOSUpdates.csv file which compares the current installed version. These are the possible values:

  • Blank (null) – a CSV file was not used or is unreachable so this could not be validated
  • Proper BIOS Installed – The ESXi host has the correct BIOS installed that contains the NEW microcode update from Intel
  • BIOS update available – install vxx – The ESXi host needs a BIOS update. The version required is provided here, look at a later example of a system needing an update
  • Unknown – Check with manufacturer – The hardware model is not present in the CSV file, or there has not been a microcode release for the current system model. Look at a later example of a Nehalem/Westmere system

MCU CPUID, PCID/INVPCID

A patched ESXi host will show the new CPU instructions provided by the microcode update (IPBP, IBRS, and STIBP). This is another confirmation that the hypervisor has been properly updated. KB52085 outlines steps to gather confirmation of an updated system.

PCID/INVPCID CPU instructions can reduce performance impact of Spectre mitigation (CVE-2017-5754) which are present on Haswell and newer Intel product family. Please note that this is a performance optimization and not a security weakness if the CPU doesn’t provide it. Check with your VM Guest OS vendor to determine if it’s supported, and note that VM hardware version 11 or higher is needed.

On the example above, you can see that the Intel Product family is Broadwell and PCID/INVPCID is True, for a unsupported system it will show up as False.

ESXi applied MCU, MCU BIOS rev, MCU boot rev

KB52345 covers the details around the initial microcode patches ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG that had to be pulled due to the Intel reboot issue. If you wish to check what the current microcode revision is and whether the ESXi booted with the microcode provided by the patch (or future patches, if VMware bundles the new versions) then here is where you need to use the -UseSSH parameter to obtain this information.

ESXi applied MCU

This has 2 possible values:

  • Active – The microcode revision provided by the VMware patch is being used.
  • Inactive – The microcode revision provided by the system BIOS is being used. There is no VMware patch providing a microcode update or the BIOS provided microcode is the same or newer. In the example above you can see that the value is Inactive and when comparing MCU BIOS rev and MCU boot rev; you see that both versions are the same.

MCU BIOS revision

Shows the microcode revision provided by the system BIOS. If the BIOS provided microcode revision is newer or same as the one provided by a VMware patch, then the BIOS revision will be used.

MCU boot rev

MCU boot rev is the current active microcode revision. If ESXi applied MCU = Active, then MCU boot rev will show a newer revision than MCU BIOS rev. This will be true for a system that has a BIOS version containing an older microcode than what was provided by the VMware patch. ESXi will never override a newer version of the microcode patch provided by BIOS nor will the BIOS with an older version prevent ESXi from applying the newer version. I will show you an example of this later.

Intel product, Intel MCU Status, Intel MCU(s) at risk, Intel Production MCU

This is the result of using the Intel_MCU.csv file that provides information about the microcode production status, if there were earlier affected microcode revision(s), and what the new version is. You can compare this against the results obtained previously (MCU BIOS rev/MCU boot rev) and see if you were affected. The CSV file is a dump of Server CPUs provided by Intel MCU guidance

MCU boot rev at risk, VMware MCU workaround applied

MCU boot rev. at risk

This has 2 possible values

  • True – the current active microcode revision (MCU boot rev.) is equal to Intel MCU(s) at risk. This means that the hypervisor is currently running a microcode version that has been identified by Intel to cause reboot issues
  • False – the current active microcode revision (MCU boot rev.) has not been identified by Intel to cause reboot issues.

VMware MCU workaround applied

KB52345 mentioned earlier outlines the workaround provided by VMware to disable/hide the CPU instructions from the VMs so they are not affected by the reboot issue. This consists of updating the /etc/vmware/config file with this line: cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-” on the ESXi host.

VMware MCU workaround applied has 2 possible values

  • True – /etc/vmware/config file contains line: cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-“.
  • False – /etc/vmware/config file does NOT contain line: cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-“.

Looking at an ESXi host that needs to be patched

Spec2

Note the following on this example:

  • Dell PowerEdge R730 running ESXi 5.5.0 (7504623 Build)
  • The ESXi is using a BIOS microcode that has been identified by Intel to cause reboot issues. You can easily identify it here, as MCU boot rev. at risk = True.
  • The VMware workaround is in place as VMware MCU workaround applied = True
  • ESXi applied MCU = Inactive, Dell BIOS v2.7.0 is providing the current active microcode revision(0x0000003b)
  • There is a new production microcode revision that is available for this Intel product family (Intel product, Intel MCU status, and Intel production MCU fields)
  • BIOS guidance shows that I need to install BIOS v2.7.1 to have this system patched for Spectre
  • Intel product shows that this server has a Haswell product family CPU, so this should provide PCID/INVPCID performance optimization right? Why is PCID/INVPCID = False? Remember that a VM requires hardware version 11 or higher to support this, note that the hypervisor version running here is ESXi 5.5; which only supports:
    • Up to Hardware Version 10
    • A max EVC Mode of Ivy Bridge (Max EVC mode field)

This then is yet another reason why you should consider upgrading ESXi 5.5 sooner rather than later to get the performance benefit against Spectre. I will show you another example next of a system with the same Intel CPU Family but running ESXi 6.0.

Spec3

Looking at an ESXi host, where Intel has not released a Microcode update

Spec4

Note the following on this example:

  • An Intel Westmere product Family CPU
  • Intel MCU production status is Beta
  • A microcode was never provided for this CPU family, so there are no CPUID instructions presents for Spectre mitigation (MCU CPUID is blank/null). There is no Microcode rev at risk, and no VMware workaround applied
  • BIOS guidance = Unknown – Check with manufacturer. Using the CSV file gives you the flexibility of updating the status of this field as Intel makes progress and a BIOS update becomes available without touching the code.

Looking at an ESXi host, with an AMD processor

Spec5

Note the following on this example:

  • This script is more focused on Intel powered systems; but it does provide some information when executed against an AMD system
  • I found no plans, date for Dell to provide a BIOS update for this PowerEdge R715 server, but if one is provided I will be able to leverage the BIOS_Update.csv file to validate this system again.
  • No CPU instructions present as part of Microcode update relevant to Spectre mitigation, even though the VMware patch provided microcode update is active.

ReportOnVMs/Get-VMSpeculativeExecution

-ReportOnVMs and Get- VMSpeculativeExecution produce the same output. ReportOnVMs is just an additional switch to use with Get-ESXSpeculativeExecution to validate VM hosted on the hypervisor as well.

Get-VMSpeculativeExecution

Make sure to run the Get-Help (Example: Get-Help Get-VMSpeculativeExecution -ShowWindow) command to understand all the parameters and switches, we will focus more on what is gathered on this blogpost.

Syntax

Get-VMSpeculativeExecution [-VM] <VirtualMachine[]> [<CommonParameters>]

Validating a VM

This scripts shows/validates that the new CPU instructions introduced by the Spectre microcode update are present and available to the VM Hardware. This does not prove and is not an end to end validation process including the Guest OS. Make sure you’re familiar with KB52085 and follow your VM Guest OS vendor steps to enabling and validating Spectre. Microsoft has provided a PowerShell script to validate/enable the protections on their Guest OS.

Last PoweredOn

This is based on VMware.Vim.VmPoweredOnEvent Events, where are limited to the vCenter Database retention settings; on VCSA 6.x the default is 30 days. Don’t expect all your VMs to show a timestamp for last Powered On, but the intention of including it here is for you to have an idea and compare this time stamp vs the hypervisor uptime to determine whether or not a VM has been power cycled (cold boot), which is required for the new CPUID instructions/features to get picked up by the Guest OS.

Spec6

ESXi MCU CPUID, ESXi PCID/INVPCID

Shows that an ESXi host has the new Spectre mitigations CPUID instructions present and if the ESXi is capable of providing PCID optimization to the VM Guest OS.

VM MCU CPUID, VM PCID/INVPCID

Shows if the Spectre mitigations CPUID(s) have been picked up by the VM hardware, as well if it’s PCID optimized or not. We will see a few examples of these later, but for now please note that having this present does not necessarily mean that the VM has been power cycled, because as a VM is vMotioned to another host is a way for these to show up as well. As an example, if you’re patching a cluster and as you’re evacuating the next host to patch, VMs placed on the recently patched host will show these new instructions.

Hypervisor-Assisted Guest mitigation, PCID optimization

Both of these show an ESXi/VM relation of the setting. Only powered on VMs are validated.

Hypervisor-Assisted Guest mitigation

Has the following possible values:

  • Supported/Enabled – Both the ESXi host and VM shows the Spectre mitigation CPUID(s) (STIBP,IBRS,IBPB)
  • Supported/Disabled – ESXi host has Spectre mitigation CPUID(s) present but the VM Hardware has not yet picked them up.
    • It will show as enabled after power-cycle (cold boot)
    • It will show as enabled if the VM is vMotioned to another “Supported” host
  • NotSupported/Disabled – ESXi host does not support Spectre mitigation, so the VM has no support either. This is most likely due to the ESXi host not yet been updated to be Spectre compliant.
  • Supported/Upgrade VM Hardware – ESXi host has Spectre mitigation CPUID(s) present but the VM needs to have its hardware upgraded before it can recognize the new instructions.
    • VM hardware version 9 is the minimum requirement for Hypervisor-Assisted Guest Mitigation for branch target injection (CVE-2017-5715).
  • UnknownSinceOff – The script will only validate powered on VM. All powered off will show up as UnknownSinceOff

PCID optimization

Has the following values:

  • Supported/Enabled – The CPU on the ESXi host is Haswell or newer supporting the PCID, INVPCID instructions. The VM hardware shows these instructions as well, so the Guest OS may be PCID optimized. Check with your VM Guest OS vendor for PCID Optimization support (For Microsoft, that would be Windows 2012 and newer)
  • Supported/Disabled – The CPU on the ESXi host is Haswell or newer supporting the PCID, INVPCID instructions. The VM does show to be supporting these instructions possibly because of:
    • The VM is hosted on a cluster running in EVC Mode set to a level lower than Haswell, so the instructions have been hidden from the virtual hardware, you will see an example of this later.
    • VM advanced settings in place
  • Supported/Upgrade VM Hardware – The CPU on the ESXi host is Haswell or newer supporting the PCID, INVPCID instructions. The VM does show to be supporting these instructions because it does not meet the minimum hardware version.
    • VM hardware version 11 or newer is needed to support PCID/INVPCID optimization
  • UnknownSinceOff – The script will only validate powered on VM. All powered off will show up as UnknownSinceOff
  • NotSupported/NA – The CPU on the ESXi host is older than Haswell which does not support the INVPCID instruction. VM will show up as NA, you will see an example of this later.

Looking at a Hypervisor-Assisted Guest mitigation enabled VM

 Spec7aSpec7b

Note the following on this example:

  • VM hardware v13 – ESXi host is v6.5.0 (comparing with the results of Get-ESXSpeculativeExecution)
  • PCID Optimized
  • Spectre mitigation CPUID present on both host and VM and there is a recent VM Last PowerOn event, so most likely the Guest OS is now picking up the needed CPUID(s)… given all steps provided by the Guest OS vendor was followed.

Looking at a Hypervisor-Assisted Guest mitigation disabled VM

Spec8

Note the following on this example:

  • Spectre mitigation CPUID present on the ESXi host but not on the VM.
  • Hypervisor-Assisted Guest mitigation = Supported/Upgrade VM hardware. Current VM hardware (v8) does not meet minimum requirement for Spectre mitigation.
  • Same is true for PCID optimization. To get VM to be “compliant” I will need to upgrade VM hardware to v11 or newer.

Looking at a Hypervisor-Assisted Guest mitigation disabled and PCID optimization disabled VM

Spec9a

Note the following on this example:

  • The ESXi host is running the first release of the problematic Intel microcode that is known to cause reboot issues and I have not applied the recommended VMware workaround for it!
  • A good example where results gathered through SSH are important for this analysis. We can deduce that Dell BIOS version 1.2.11 provides microcode version 0x0200002c and that the VMware installed Patch is providing the current active microcode update which is version 0x0200003a. ESXi applied MCU = Active
  • The Intel MCU(s) at risk field shows that the current active microcode version is known to cause reboot issue.
  • There is also a new production MCU available through Dell BIOS update v1.3.7 that I can apply

So why have I not done anything about fixing this potential problem? Let’s look at a VM to better understand!

Spec9b

The answer is due my cluster’s EVC mode. Go back to KB52085, under vMotion and EVC Information, states the following “In order to maintain this compatibility the new features are hidden from guests within the cluster until all hosts in the cluster are properly updated.  At that time, the cluster will automatically upgrade its capabilities to expose the new features. Unpatched ESXi hosts will no longer be admitted into the EVC cluster

  • I have a hardware v13 VM running on that host, that even though shows to be Spectre mitigation capable, the VM hardware is not picking up the new features due to the EVC mode.
  • I don’t need to worry about updating this host, until a microcode version is made available for Nehalem product family, since all my hosts in cluster needs to be patched.
  • The CPU on the ESXi host is Skylake which supports the PCID, INVPCID instructions, however the VM is not picking it up. The EVC mode in place is hiding those instructions from the virtual hardware.

Update 3/28/2018

The following Patches have been made available from VMware:

ESXi Guest OS Framework MCU v2 Patch
6.5 ESXi650-201803401-BG ESXi650-201803402-BG
6.0 ESXi600-201803401-BG ESXi600-201803402-BG
5.5 ESXi550-201803401-BG ESXi550-201803402-BG

 Important knowledge-base articles update:

KB5208 – Note that Guest framework patch is what allows the VM guest OS to use new CPU instructions (IBPB,IBRS,STIBP). The v2 MCU patch covers the Intel Generation CPU on this KB only; which does not include Westmere and Nehalem product Family. You will see examples below of patching against Westmere system.

KB52345 – If you are a user that currently have patches for v1 MCU that caused reboot issues (ESXi650-201801402-BG, ESXi600-201801402-BG, and/or ESXi550-201801401-BG), which I covered previously, and have the workaround applied then know that the Guest OS framework patch will remove the workaround line from /etc/vmware/config when applied.

V2.4.2 of vDocumentation includes the following enhancement to Get-ESXSpeculativeExecution script.

ESXi Guidance

This will validate if you have installed Guest OS Framework Patch and MCU v2 Patch. These are the possible values:

  • Framework Missing – The Framework patch has not been installed (ESXi650-201803401-BG, ESXi600-201803401-BG, or ESXi550-201803401-BG)
  • Framework Installed – The Framework patch has been installed
  • v2 MCU Missing – The v2 MCU patch has not been installed (ESXi650-201803402-BG, ESXi600-201803402-BG, or ESXi550-201803402-BG)
  • v2 MCU Installed – The v2 MCU patch has been installed

Let’s look at a few examples to demonstrate this

Installing the new VMware patches on an ESXi host

Spec10

Note the following on this example:

  • I applied the VMware workaround for previous v1 MCU patches to demonstrate what happens when installing the Framework patch. VMware MCU workaround applied = True
  • Looking at ESXi Build ID, you can see that it has not been patched (7526125)
  • BIOS guidance shows that I need to install BIOS v2.6.1 to have this system patched for Spectre
  • ESXi guidance shows both new patches are missing

Next I’ll install the Framework patch only:

Spec11

  • ESXi Build ID is now updated, showing that I have patched this system
  • ESXi guidance shows that I have only applied the framework patch.
  • MCU CPUID still blank since I have not applied the BIOS update nor the ESXi v2 MCU patch
  • ESXi applied MCU = Inactive
  • VMware MCU workaround applied = False; installing the Framework patch removed the workaround line (cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-“. ) from etc/vmware/config

Next I’ll install the v2 MCU Patch

Spec12

  • BIOS guidance still shows that I need a BIOS update, and my current BIOS version is still v2.5.4
  • ESXi guidance shows that I have applied both patches
  • MCU CPUID shows the spectre mitigation instructions are present
  • ESXi applied MCU = Active. BIOS v2.5.4 is providing microcode revision 0x710, however the v2 ESXi patch is providing a newer revision: 0x713. The ESXi is then using the microcode provided by the patch

Next I’ll install the BIOS update

Spec13

  • BIOS guidance shows that proper BIOS is installed. Current BIOS version is v2.6.1
  • ESXi applied MCU = Inactive, both microcode revisions are the same, so the BIOS provided rev is used.

Looking at an ESXi host with Westmere/Nehalem Intel CPU

Spec14

Note the following on this example:

  • ESXi guidance shows that both patches have been installed. The Build (7967664) is current as well
  • This is a Westemere Intel CPU, so the patches does not contain the v2 microcode for this product family
  • BIOS guidance shows that there is a BIOS update available: vP65 (2/22/2018)
  • MCU CPUID is blank, no spectre mitigation CPU instructions present

Next, I’ll show you a Westmere system that has the BIOS update present

Spec15

  • BIOS guidance shows that proper BIOS is installed. Current BIOS version is P65 (2/22/2018)
  • MCU CPUID shows the spectre mitigation instructions are present

Happy patching!!

Posted in PowerShell

Validating compliance of VMSA-2018-0004 (Spectre) on ESXi and VM

This is an update to  Validating Compliance of VMSA-2018-0002 and BIOS update.

VMware recently published VMSA-2018-004, which details Hypervisor-Assisted Guest Mitigation fixes as well as knowledge base (KB) article 52085 with instructions to verify the updated microcode for a Virtual Machine.

Version 2.4.0 of vDocumentation now includes enhancements to the Get-ESXSpeculativeExecution script cmdlet that will:

  • Validate if an ESXi is patched and compliant to VMSA-2018-0004
  • Validate if an ESXi is seeing the updated CPUID Microcode

If –ReportOnVMs switch is specified then this will also report on VM compliance.

This new version also includes a new script cmdlet: Get-VMSpeculativeExecution. This cmdlet accepts $VM object and is very useful for checking VM compliance. Make sure to run “get-help Get-VMSpeculativeExecution –full” to see all the help content.

 

Examples of Get-ESXSpeculativeExecution

If we run “Get-ESXSpeculativeExecution -esxi labesx001.local”, this is the output:

VMSA-2018-0004

If we run “Get-ESXSpeculativeExecution –ReportOnVMs –exportexcel”, we will generate an Excel sheet with 3 TABs: Patch_Compliance, BIOS_Compliance, and VM_Compliance

VMSA-2018-0004_2

VMSA-2018-0004_3

VMSA-2018-0004_4

Examples of Get-VMSpeculativeExecution

If we run “Get-VM “testvm39” | Get-VMSpeculativeExecution”, this is the output:

VMSA-2018-0004_5

Other Examples

“Get-VM |   Get-VMSpeculativeExecution” Will check compliance on ALL VMs

“Get-Host “Labhost13.local” | Get-VMSpeculativeExecution” Will check compliance for all VMs running on host Labhost13.local

“Get-VM -Location “Dev_Cluster” | Get-VMSpeculativeExecution” Will check compliance for all VMs running on Dev_Cluster

Get-VMHost “Labhost13.local” | Get-VM | Get-VMSpeculativeExecution | Export-Excel “VMValidation.xlsx” -WorkSheetname “VMresults” Will check compliance for all VMs running on host Labhost13.local, which will exported to “VMValidation.xlsx” Excel file.

 

Posted in PowerShell

Validating compliance of VMSA-2018-0002 and BIOS update

UPDATE: Please see validating compliance of VMSA-2018-0004 (Spectre) on ESXi and VM

VMware has published VMSA-2018-0002 that addresses vulnerability for Spectre and Meltdown (CVE-2017-5753, CVE-2017-5715) and tells you which patch should be installed.

Along with this patch, you also need the required BIOS update. However, when managing a large VMware environment it’s hard to keep track of what has been patched. It’s also hard to remember if BIOS has been updated.

For this reason, I have added a new script to vDocumentation that will check compliance against VMSA-2018-0002 and the required BIOS update.

Version 2.3.0 now includes a script Cmdlet (Get-ESXSpeculativeExecution) that will aide in validating your environment. If you are new to vDocumentation, please make sure to check the GitHub project page on how to install it from the PowerShell Gallery. (https://github.com/arielsanchezmora/vDocumentation).

Running Get-ESXSpeculativeExecution

Run “get-help Get-ESXSpeculativeExecution –full” to see all parameters and switches available; below are the most frequently used:

  • The Script will validate ESXi versions 5.5, 6.0, and 6.5
  • You can validate ESXi only (-PatchCompliance switch)
  • You can validate BIOS only (-BIOSCompliance switch)
  • You can validate both (specify no switch)
  • –esxi to validate a host
  • -cluster to validate a cluster
  • -datcenter to validate a datacenter
  • No parameter to run against the entire vCenter
  • -exportExcel to export to Excel

Examples:

If we run “Get-ESXSpeculativeExecution -esxi labesx001.local” , this is the output:

SXSpeculativeExecution001

If we run “Get-ESXSpeculativeExecution –exportexcel”, we will generate an Excel sheet with 2 TABs: Patch_Compliance and BIOS_Compliance.

SXSpeculativeExecution002

SXSpeculativeExecution003

BIOS Version Check

The BIOS Compliance validation relies on accessing the BIOSUpdates.csv file, which is hosted on the project page: https://raw.githubusercontent.com/edmsanchez/vDocumentation/master/powershell/vDocumentation/BIOSUpdates.csv

If you don’t have access to the Internet you can download the CSV file locally and specify it using the –inputfile parameter. The CSV file contains HP and Dell models, with updates from these these official sources:

If you’re working on something other than HP or Dell, do let me know via the #vdocumentation Vmware code Slack channel, Twitter or the GitHub webpage and we can update the CSV file with your model.

The SafeFromSpectre field on the BIOS_Compliance Tab will be True if both ESXi and BIOS have been patched (see below).  This indicates you have addressed the vulnerability completely.

SXSpeculativeExecution004

Posted in PowerShell, vDocumentation
Edgar Sanchez