Confirming Hypervisor-Assisted Guest Mitigations at a VM level. gives you detailed steps on how to validate if an ESXi host has both the CPU Microcode and patches applied. The step for confirming this is to examine the vmware.log file for a VM and look for the CPUID capability entry, for example based on latest KB.

Capability Found: cpuid.MDCLEAR

This however just confirms that the hypervisor has the new microcode that includes the CPUID mask (CPU Feature) to mitigate the Microarchitectural Data Sampling (MDS) Vulnerabilities. There is no easy way by just looking at the vmware.log to confirm whether or not the CPU features have been exposed to the VM hardware, what some folks do is search the vmware.log and look for the MDCLEAR entry under “Featurecompat: Requirements:”.


There is an easier way of confirming this and any future CPUID Masks. We can validate both the Hypervisor and VMs using PowerCLI, doesn’t have to be a script, just a line of code:

Validating Hypervisor

The below code, will list all the CPUID Masks enabled by the current CPU Microcode on a specific ESXi host. Note that MDCLEAR CPU Feature is present, so the ESXi host is patched and compliant

(Get-VMHost -Name "hostname").ExtensionData.Config.FeatureCapability | 
Where-Object value -eq 1 | Select-Object -ExpandProperty FeatureName 

If you wanted to validate a Cluster, to list all ESXi hosts that are MDS compliant (Have the MDCLEAR CPU Feature present):

Get-Cluster "clustername" | Get-VMHost | 
Where-Object {$_.ExtensionData.Config.FeatureCapability.value -eq 1 -and $_.ExtensionData.Config.FeatureCapability.FeatureName -match "MDCLEAR"} 

Validating VMs

The below line, will list all the CPUID Masks available to the VM hardware

(Get-VM -Name "VMname").ExtensionData.Runtime.FeatureRequirement | Select-Object -ExpandProperty FeatureName 

If you wanted to validate a cluster, to list all VMs that are MDS compliant (Have the MDCLEAR CPU Feature present):

Get-Cluster "clustername" | Get-VM | Where-Object {$_.ExtensionData.Runtime.FeatureRequirement.FeatureName -match "MDCLEAR"} 

The Vmware.log is an accurate source to show the CPUID capability (indicating that that both the CPU microcode and hypervisor are properly updated). However, you should know that just because the log indicates the CPUID capability that does not mean the VM is mitigated. I have found the log entries present just by migrating VMs during the remediation of a Cluster.

By using PowerShell we can confirm the CPUID features availability at the hypervisor level, and which VMs have the hardware support enabled.

Posted in PowerShell

Intel MDS Guidance – No plans to support Nehalem and Westmere Product Family

With the recent release of new microcode updates (MCUs) to mitigate Microarchitectural Data Sampling (MDS) vulnerabilities, Intel dropped plans to support older generation processors like Nehalem and Westmere. These were supported for L1TF, Spectre and Meltdown vulnerabilities. If you are a Dell shop then these are 11G servers (R610, R710), or G7 servers for HPE. Please review SA00233-microcode-update-guidance to make sure your environment is covered.

Impact of not having the MCU

Microarchitectural Data Sampling (MDS), name given to a collective group of vulnerabilities:

  • Microarchitectural Store Buffer Data Sampling (MSBDS) aka Fallout Attack – CVE-2018-12126
  • Microarchitectural Fill Buffer Data Sampling (MFBDS) aka ZombieLoadAttack – CVE-2018-12130
  • Microarchitectural Load Port Data Sampling (MLPDS) aka RIDL – CVE-2018-12127
  • Microarchitectural Data Sampling Uncacheable Memory (MDSUM) – CVE-2019-11091

These attack allow local attackers that are able to execute code to access portions of recently written or loaded data by using speculative execution information leak methods. They are 2 attack vectors

  • Single thread, same core or what VMware refers to as Sequential-context attack vector
  • Cross Hyper thread or what VMware refers to as Concurrent-context attack vector

Both of these attack vectors have another category when dealing with virtualization:

  • Intra-VM – attack vector in the context of the same Virtual Machine
  • Inter-VM – attack vector in the context of VM-to-VM and VM-to-Host

For full understanding and step by step guide in mitigating MDS please follow directions from VMware KB articles.

All 4 MDS vulnerabilities require CPU microcode/firmware to mitigate and to fully protect a VMware infrastructure 2 things are needed:

  • ESXi Patched which delivers the new MCU
  • Enabling the ESXi Side-Channel-Aware Scheduler. This scheduler will then schedule on only one Hyperthread of a Hyperthread-enabled core, with potentially significant consequences as hyperthreading is essentially no longer used. With the release of ESXi 6.7 U2 a new version of this scheduler is supported, please refer to VMware article by Bob Plankers that covers this topic extensively.

Do note that by enabling the ESXi Side-Channel-Aware Scheduler you’re essentially eliminating the Cross Hyper thread (Concurrent-context attack vector) attack surface, for MDS this means that you’re being protected against MSBDS and MFBDS.

Intel makes the distinction that by disabling HT alone does not provide protection against MDS, because you are still vulnerable to the single thread attack vector and here is where the new MCU is critical. The new MCU mitigates this by providing special software sequences called “MD_CLEAR” that flushes the internal CPU buffer and other structures at certain times so that critical data is no longer visible to an attacker.

So systems without the MCU are vulnerable because they don’t have the CPUID MD_CLEAR to carry out the mitigation of flushing the CPU buffers.

Mitigation and Validation

The mitigation guidelines form VMware focuses on Inter-VM as Intra-VM are Operating System-Specific Mitigations. These are applied to guest operating systems which are provided by your OS vendor.

The Inter-VM mitigation process for MDS is divided into three phases (taken from


To validate the state of protection I have 2 test environments

  1. Test VM running on ESXi 6.5 with Intel Broadwell family CPU
  2. Test VM running on ESXi 6.0 with Intel Westmere family CPU

All the vSphere environment fully remediated as outlined by the flowchart above

  • vCenter patched
  • ESXi Patched
  •  ESXi Side-Channel-Aware Scheduler enabled

On the Guest VM side, both are Windows 2012 R2, both patched and mitigations enabled (Registry keys applied) and power cycled.

Validate VMs using MDS tool from RIDL team

Broadwell CPU VM – has MDS MCU available


Note the following:

  • CPU is vulnerable to MDS
  • MD_CLEAR CPUID present so this means the ESXi has the new MCU and the Hypervisor-Assisted Guest Mitigations is in place exposing the hardware feature to the VM

Running the Microsoft PowerShell module Get-SpeculationControlSettings to validate mitigation compliance in the guest OS


The result confirms that the VM is being protected against MDS



Note the following:

  • CPU is vulnerable to MDS
  • MD_CLEAR CPUID NOT present. ESXi fully patched but since there is no available MCU for this CPU family, then there is no support for MD_CLEAR

Running the Get-SpeculationControlSettings module reveals the following:


OS support for MDS mitigation is present, but since the MD_CLEAR CPUID instruction is not available then MDS cannot be mitigated even if the registry keys are in place.

What MCUs are available

As of this writing Intel has released production MCUs supporting all systems manufactured recently (last 5 yrs.), from a server perspective what is still pending is Sandy Bridge family (servers manufactured in ~2012). Dell advisory has an expected release date for the new BIOS version at 09/2019, these are all 12G servers. The VMware VMSA-2019-0008 security advisory makes this distinction as well: “These patches contain updated microcode.  At the time of this publication Sandy Bridge DT/EP Microcode Updates (MCUs) had not yet been provided to VMware. Customers on this microarchitecture may request MCUs from their hardware vendor in the form of a BIOS update. This microcode will be included in future releases of ESXi.”

Basically if you are running Ivy bridge and above in your server environment you are covered from a MCU perspective. Sandy Bridge you need to wait for new MCU, Westmere or older, no plans for new MCUs.


If you are running Westmere or Nehalem platforms like Dell R710/R610/m610 or HPE Gen7, YOU ARE NO LONGER IN A SUPPORTED CONFIGURATION BY INTEL. For MDS mitigation you will have to make a business case for new hardware and upgrade to ESXi 6.7 where you can also benefit from the v2 of the scheduler.

If you need more information about MDS, here are some articles to look at:

Posted in PowerShell

Use PowerCLI to manage users on an ESXi host

Starting with vSphere 6.0 VMware introduced a new set of ESXCLI commands to manage the life cycle of local accounts and permissions.

I have created ESXiAccountManagement.ps1 script that includes the following functions:

  • Get-ESXiAccount: List all local accounts and their role
  • Add-ESXiAccount: Create local accounts and assign one of the predefined roles
  • Set-ESXiAccount: Updates user’s Description, permission (one of the predefined roles) and most popular reset the user password.
  • Remove-ESXiAccount: Removes a local account
  • Get-ESXiAccountSecurity: Get ESXi host local account security settings and can also assist in checking events for account locked and bad logon events (past 1, 2 or 24 hrs)

Running the script

Make sure to run the Get-Help (Example: Get-Help Get-ESXiAccount –ShowWindow) command to understand all the parameters and switches. All these functions make use of the new ESXCLI commands so the script will only run against ESXi 6.x hosts. The script also assumes that you already have a connection to a vCenter, let’s walk through a few examples to show the script in action.


A script function to list all local accounts, can be run against a host, cluster, or datacenter


Get-ESXiAccount [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]



A script function to add/create a new local user. The script will first check if a User ID with the same name already exists before creating it. You are prompted for the user’s password during the script execution, this uses Get-Credential object which stores and relays the password as a secure string.

The –Permission parameter will show you a list of options, which are the predefined roles supported (Admin, ReadOnly, NoAccess)  to grant a user.


Add-ESXiAccount [-Name] <String> [[-Description] <String>] [[-Permission] <Object>] [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

Creating a user account

Let’s create a monitoring service account needed on the ESXi host, user ID: sysmonitoring


Note the following on this example:

  • I’m specifying the User ID, Description and Permission for the new user (sysmonitoring)
  • I was prompted for the password
  • User creation failed because the password I used did not meet the password quality control policy
  • The password policy in place is the default, and is displayed as part of the feedback on the error message

Let’s try that again using a stronger password



A script function to reset a user’s password, update description, or update permission. Same as Add-ESXiAccount, you will be prompted for the new password when resetting a user’s password and the password strength must meet the policy requirements.


Set-ESXiAccount [-Name] <String> [[-Description] <String>] [[-Permission] <Object>] [-ResetPassword ] [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

Recovering the root user on a ESXi host

The most popular use case for this is when you have forgotten the root user password on the ESXi host. You can stop using host profiles, SSH keys, etc, and just do it through PowerCLI!



Note the following on this example:

  • I was able to recover/reset the root password using PowerCLI
  • Password complexity applies, so keep that in mind or else you will get the error as the previous example when adding a new user

Changing user description and permissions

Now that I have created sysmonitoring user, I want to update the description and also found out that it needs admin privileges instead of ReadOnly (hypothetical example).



A script function to delete an ESXi local user, can be run against a host, cluster, or datacenter. This will remove user created accounts, not system accounts (root for example)


Remove-ESXiAccount [-Name] <String> [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

Trying to remove a system account (root)


Note the following on this example:

  • System accounts are protected against deletion
  • Some of these scripts functions are powerful, use with caution

Removing sysmonitoring user ID (user created account)

Continuing with our example, I now don’t have the need to keep sysmonitoring any longer, so I shall remove him


Note the following on this example:

  • Sysmonitoring user was successfully removed
  • A listing of all local accounts on the ESXi host shows that sysmonitoring user is no longer present


A script function to list account lockout policy, password quality control policy and to gather events for troubleshooting account lockouts and bad logons


Get-ESXiAccountSecurity [[-EventsPastHrs] <Int32>] [[-VMhost] <Object>] [[-Cluster] <Object>] [[-Datacenter] <Object>] [<CommonParameters>]

-EventsPastHrs is the switch used to query for account lockout , and bad logon events. It has 3 predefined options (1, 2, 24) hours.

Troubleshooting account logon scenario

Let’s look at a hypothetical example:

  • Dellomsa was created and given an admin role
  • This account has stopped working, I cannot logon with it. Suspecting a bad password, I went ahead and did a password reset
  • After resetting the password, I’m still not able to login

To troubleshot further I then looked for account events


Note the following on this example:

  • System has the default settings for Lock Failures, Unlock Time, and complexity
  • I cannot login with dellomsa because its locked out, so I need to wait for the Unlock time to expire or use SSH to unlock it
  • I queried the events for the past hour and have identified the IP where the bad logons are coming from
  • The failed logon attempts (50 failed login attempts) is a global counter on the ESXi host for all bad logon attempts. I have been playing with this host, validating the scripts and writing this blog. I in fact just tried to logon 6 times with a bad password to get dellomsa user locked out. 5 attempts is default policy before locking out and 15 minutes to unlock.
  • If a local account is locked out, you will get a warning on the host in vCenter as well


As you can see using PowerCLI is a great way to manage and troubleshoot ESXi user accounts hope this has been helpful for you.

Posted in PowerShell

Verify new Spectre mitigation patches using PowerCli and vDocumentation

Intel recently announced that it has released microcode updates for 100 percent of its products launched in the past five years that require protection against Spectre/Meltdown. For virtualization admins using Intel Xeon chips, this would mean that servers with a Sandy Bridge or newer generation CPU now have a BIOS update available. Westmere and Nehalem are in Beta production status as of this writing for microcode updates.

We have published an update of vDocumentation that will assist you in validating your current environment for Spectre. The update (v2.4.1) includes enhanced versions of:

  • Get-ESXSpeculativeExecution
  • Get-VMSpeculativeExecution

UPDATE (3/28/2018) VMware recently updated security advisory VMSA-2018-0004 with the list of new patches, and Intel has also released production MCU for Westmere and Nehalem.

We have released v2.4.2 of vDocumentation to help validate your environment for these new patches. More details at the end of this post.


A better way of explaining the changes would be by demonstrating the results gathered with a few examples. First though, here are some dependencies that the script relies on, we will mention them here and cover them in more detail in paragraphs below.

The script relies on two CSV files:

  • BIOSUpdates.csv – This file is used to validate and provide guidance on the current installed BIOS version. If you do not have internet access when running the script, or wish to use a customized version of the file, then download it and use the –inputBiosFile parameter to specify the offline version path
  • Intel_MCU.csv– This file is used to report on the microcode status provided by Intel. Use the -inputMcuFile parameter to specify the offline version path

Both files are to up to date, and I will keep them current as Intel provides new updates. The BIOS update file contains information for Dell and HP servers, which is what I have in the environment I support. If you have different hardware and would like to share the pertinent information with the community, please reach out and I will update the file with your input.

SSH dependency: VMware KB52345 has all the details about the initial MCU patches that Intel released which caused reboot problems. In order to validate if the microcode revision running on an ESXi is affected you need to run a shell command; today this can only be accomplished through SSH from an automation standpoint. When running the script, you need to use the -UseSSH parameter, this uses Posh-SSH as an SSH client which you can install from the PowerShell gallery (Install-Module -Name Posh-SSH). With Posh-SSH you will be prompted for credentials, this uses Get-Credential and the PSCredential object which stores and relays the password as a secure string.


Running the script

Make sure to run the Get-Help (Example: Get-Help Get-ESXSpeculativeExecution –ShowWindow) command to understand all the parameters and switches, we will focus more on what is gathered on this blogpost.


Get-ESXSpeculativeExecution [[-esxi] <Object>] [[-cluster] <Object>] [[-datacenter] <Object>] [[-inputBiosFile] <Object>] [[-inputMcuFile] <Object>] [-UseSSH ] [-ReportOnVMs ] [-ExportCSV ] [-ExportExcel ] [-PassThru ] [[-folderPath] <Object>] [<CommonParameters>]

Looking at an ESXi host that has been patched for Spectre


BIOS Guidance

BIOS Guidance relies on the BIOSUpdates.csv file which compares the current installed version. These are the possible values:

  • Blank (null) – a CSV file was not used or is unreachable so this could not be validated
  • Proper BIOS Installed – The ESXi host has the correct BIOS installed that contains the NEW microcode update from Intel
  • BIOS update available – install vxx – The ESXi host needs a BIOS update. The version required is provided here, look at a later example of a system needing an update
  • Unknown – Check with manufacturer – The hardware model is not present in the CSV file, or there has not been a microcode release for the current system model. Look at a later example of a Nehalem/Westmere system


A patched ESXi host will show the new CPU instructions provided by the microcode update (IPBP, IBRS, and STIBP). This is another confirmation that the hypervisor has been properly updated. KB52085 outlines steps to gather confirmation of an updated system.

PCID/INVPCID CPU instructions can reduce performance impact of Spectre mitigation (CVE-2017-5754) which are present on Haswell and newer Intel product family. Please note that this is a performance optimization and not a security weakness if the CPU doesn’t provide it. Check with your VM Guest OS vendor to determine if it’s supported, and note that VM hardware version 11 or higher is needed.

On the example above, you can see that the Intel Product family is Broadwell and PCID/INVPCID is True, for a unsupported system it will show up as False.

ESXi applied MCU, MCU BIOS rev, MCU boot rev

KB52345 covers the details around the initial microcode patches ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG that had to be pulled due to the Intel reboot issue. If you wish to check what the current microcode revision is and whether the ESXi booted with the microcode provided by the patch (or future patches, if VMware bundles the new versions) then here is where you need to use the -UseSSH parameter to obtain this information.

ESXi applied MCU

This has 2 possible values:

  • Active – The microcode revision provided by the VMware patch is being used.
  • Inactive – The microcode revision provided by the system BIOS is being used. There is no VMware patch providing a microcode update or the BIOS provided microcode is the same or newer. In the example above you can see that the value is Inactive and when comparing MCU BIOS rev and MCU boot rev; you see that both versions are the same.

MCU BIOS revision

Shows the microcode revision provided by the system BIOS. If the BIOS provided microcode revision is newer or same as the one provided by a VMware patch, then the BIOS revision will be used.

MCU boot rev

MCU boot rev is the current active microcode revision. If ESXi applied MCU = Active, then MCU boot rev will show a newer revision than MCU BIOS rev. This will be true for a system that has a BIOS version containing an older microcode than what was provided by the VMware patch. ESXi will never override a newer version of the microcode patch provided by BIOS nor will the BIOS with an older version prevent ESXi from applying the newer version. I will show you an example of this later.

Intel product, Intel MCU Status, Intel MCU(s) at risk, Intel Production MCU

This is the result of using the Intel_MCU.csv file that provides information about the microcode production status, if there were earlier affected microcode revision(s), and what the new version is. You can compare this against the results obtained previously (MCU BIOS rev/MCU boot rev) and see if you were affected. The CSV file is a dump of Server CPUs provided by Intel MCU guidance

MCU boot rev at risk, VMware MCU workaround applied

MCU boot rev. at risk

This has 2 possible values

  • True – the current active microcode revision (MCU boot rev.) is equal to Intel MCU(s) at risk. This means that the hypervisor is currently running a microcode version that has been identified by Intel to cause reboot issues
  • False – the current active microcode revision (MCU boot rev.) has not been identified by Intel to cause reboot issues.

VMware MCU workaround applied

KB52345 mentioned earlier outlines the workaround provided by VMware to disable/hide the CPU instructions from the VMs so they are not affected by the reboot issue. This consists of updating the /etc/vmware/config file with this line: cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-” on the ESXi host.

VMware MCU workaround applied has 2 possible values

  • True – /etc/vmware/config file contains line: cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-“.
  • False – /etc/vmware/config file does NOT contain line: cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-“.

Looking at an ESXi host that needs to be patched


Note the following on this example:

  • Dell PowerEdge R730 running ESXi 5.5.0 (7504623 Build)
  • The ESXi is using a BIOS microcode that has been identified by Intel to cause reboot issues. You can easily identify it here, as MCU boot rev. at risk = True.
  • The VMware workaround is in place as VMware MCU workaround applied = True
  • ESXi applied MCU = Inactive, Dell BIOS v2.7.0 is providing the current active microcode revision(0x0000003b)
  • There is a new production microcode revision that is available for this Intel product family (Intel product, Intel MCU status, and Intel production MCU fields)
  • BIOS guidance shows that I need to install BIOS v2.7.1 to have this system patched for Spectre
  • Intel product shows that this server has a Haswell product family CPU, so this should provide PCID/INVPCID performance optimization right? Why is PCID/INVPCID = False? Remember that a VM requires hardware version 11 or higher to support this, note that the hypervisor version running here is ESXi 5.5; which only supports:
    • Up to Hardware Version 10
    • A max EVC Mode of Ivy Bridge (Max EVC mode field)

This then is yet another reason why you should consider upgrading ESXi 5.5 sooner rather than later to get the performance benefit against Spectre. I will show you another example next of a system with the same Intel CPU Family but running ESXi 6.0.


Looking at an ESXi host, where Intel has not released a Microcode update


Note the following on this example:

  • An Intel Westmere product Family CPU
  • Intel MCU production status is Beta
  • A microcode was never provided for this CPU family, so there are no CPUID instructions presents for Spectre mitigation (MCU CPUID is blank/null). There is no Microcode rev at risk, and no VMware workaround applied
  • BIOS guidance = Unknown – Check with manufacturer. Using the CSV file gives you the flexibility of updating the status of this field as Intel makes progress and a BIOS update becomes available without touching the code.

Looking at an ESXi host, with an AMD processor


Note the following on this example:

  • This script is more focused on Intel powered systems; but it does provide some information when executed against an AMD system
  • I found no plans, date for Dell to provide a BIOS update for this PowerEdge R715 server, but if one is provided I will be able to leverage the BIOS_Update.csv file to validate this system again.
  • No CPU instructions present as part of Microcode update relevant to Spectre mitigation, even though the VMware patch provided microcode update is active.


-ReportOnVMs and Get- VMSpeculativeExecution produce the same output. ReportOnVMs is just an additional switch to use with Get-ESXSpeculativeExecution to validate VM hosted on the hypervisor as well.


Make sure to run the Get-Help (Example: Get-Help Get-VMSpeculativeExecution -ShowWindow) command to understand all the parameters and switches, we will focus more on what is gathered on this blogpost.


Get-VMSpeculativeExecution [-VM] <VirtualMachine[]> [<CommonParameters>]

Validating a VM

This scripts shows/validates that the new CPU instructions introduced by the Spectre microcode update are present and available to the VM Hardware. This does not prove and is not an end to end validation process including the Guest OS. Make sure you’re familiar with KB52085 and follow your VM Guest OS vendor steps to enabling and validating Spectre. Microsoft has provided a PowerShell script to validate/enable the protections on their Guest OS.

Last PoweredOn

This is based on VMware.Vim.VmPoweredOnEvent Events, where are limited to the vCenter Database retention settings; on VCSA 6.x the default is 30 days. Don’t expect all your VMs to show a timestamp for last Powered On, but the intention of including it here is for you to have an idea and compare this time stamp vs the hypervisor uptime to determine whether or not a VM has been power cycled (cold boot), which is required for the new CPUID instructions/features to get picked up by the Guest OS.



Shows that an ESXi host has the new Spectre mitigations CPUID instructions present and if the ESXi is capable of providing PCID optimization to the VM Guest OS.


Shows if the Spectre mitigations CPUID(s) have been picked up by the VM hardware, as well if it’s PCID optimized or not. We will see a few examples of these later, but for now please note that having this present does not necessarily mean that the VM has been power cycled, because as a VM is vMotioned to another host is a way for these to show up as well. As an example, if you’re patching a cluster and as you’re evacuating the next host to patch, VMs placed on the recently patched host will show these new instructions.

Hypervisor-Assisted Guest mitigation, PCID optimization

Both of these show an ESXi/VM relation of the setting. Only powered on VMs are validated.

Hypervisor-Assisted Guest mitigation

Has the following possible values:

  • Supported/Enabled – Both the ESXi host and VM shows the Spectre mitigation CPUID(s) (STIBP,IBRS,IBPB)
  • Supported/Disabled – ESXi host has Spectre mitigation CPUID(s) present but the VM Hardware has not yet picked them up.
    • It will show as enabled after power-cycle (cold boot)
    • It will show as enabled if the VM is vMotioned to another “Supported” host
  • NotSupported/Disabled – ESXi host does not support Spectre mitigation, so the VM has no support either. This is most likely due to the ESXi host not yet been updated to be Spectre compliant.
  • Supported/Upgrade VM Hardware – ESXi host has Spectre mitigation CPUID(s) present but the VM needs to have its hardware upgraded before it can recognize the new instructions.
    • VM hardware version 9 is the minimum requirement for Hypervisor-Assisted Guest Mitigation for branch target injection (CVE-2017-5715).
  • UnknownSinceOff – The script will only validate powered on VM. All powered off will show up as UnknownSinceOff

PCID optimization

Has the following values:

  • Supported/Enabled – The CPU on the ESXi host is Haswell or newer supporting the PCID, INVPCID instructions. The VM hardware shows these instructions as well, so the Guest OS may be PCID optimized. Check with your VM Guest OS vendor for PCID Optimization support (For Microsoft, that would be Windows 2012 and newer)
  • Supported/Disabled – The CPU on the ESXi host is Haswell or newer supporting the PCID, INVPCID instructions. The VM does show to be supporting these instructions possibly because of:
    • The VM is hosted on a cluster running in EVC Mode set to a level lower than Haswell, so the instructions have been hidden from the virtual hardware, you will see an example of this later.
    • VM advanced settings in place
  • Supported/Upgrade VM Hardware – The CPU on the ESXi host is Haswell or newer supporting the PCID, INVPCID instructions. The VM does show to be supporting these instructions because it does not meet the minimum hardware version.
    • VM hardware version 11 or newer is needed to support PCID/INVPCID optimization
  • UnknownSinceOff – The script will only validate powered on VM. All powered off will show up as UnknownSinceOff
  • NotSupported/NA – The CPU on the ESXi host is older than Haswell which does not support the INVPCID instruction. VM will show up as NA, you will see an example of this later.

Looking at a Hypervisor-Assisted Guest mitigation enabled VM


Note the following on this example:

  • VM hardware v13 – ESXi host is v6.5.0 (comparing with the results of Get-ESXSpeculativeExecution)
  • PCID Optimized
  • Spectre mitigation CPUID present on both host and VM and there is a recent VM Last PowerOn event, so most likely the Guest OS is now picking up the needed CPUID(s)… given all steps provided by the Guest OS vendor was followed.

Looking at a Hypervisor-Assisted Guest mitigation disabled VM


Note the following on this example:

  • Spectre mitigation CPUID present on the ESXi host but not on the VM.
  • Hypervisor-Assisted Guest mitigation = Supported/Upgrade VM hardware. Current VM hardware (v8) does not meet minimum requirement for Spectre mitigation.
  • Same is true for PCID optimization. To get VM to be “compliant” I will need to upgrade VM hardware to v11 or newer.

Looking at a Hypervisor-Assisted Guest mitigation disabled and PCID optimization disabled VM


Note the following on this example:

  • The ESXi host is running the first release of the problematic Intel microcode that is known to cause reboot issues and I have not applied the recommended VMware workaround for it!
  • A good example where results gathered through SSH are important for this analysis. We can deduce that Dell BIOS version 1.2.11 provides microcode version 0x0200002c and that the VMware installed Patch is providing the current active microcode update which is version 0x0200003a. ESXi applied MCU = Active
  • The Intel MCU(s) at risk field shows that the current active microcode version is known to cause reboot issue.
  • There is also a new production MCU available through Dell BIOS update v1.3.7 that I can apply

So why have I not done anything about fixing this potential problem? Let’s look at a VM to better understand!


The answer is due my cluster’s EVC mode. Go back to KB52085, under vMotion and EVC Information, states the following “In order to maintain this compatibility the new features are hidden from guests within the cluster until all hosts in the cluster are properly updated.  At that time, the cluster will automatically upgrade its capabilities to expose the new features. Unpatched ESXi hosts will no longer be admitted into the EVC cluster

  • I have a hardware v13 VM running on that host, that even though shows to be Spectre mitigation capable, the VM hardware is not picking up the new features due to the EVC mode.
  • I don’t need to worry about updating this host, until a microcode version is made available for Nehalem product family, since all my hosts in cluster needs to be patched.
  • The CPU on the ESXi host is Skylake which supports the PCID, INVPCID instructions, however the VM is not picking it up. The EVC mode in place is hiding those instructions from the virtual hardware.

Update 3/28/2018

The following Patches have been made available from VMware:

ESXi Guest OS Framework MCU v2 Patch
6.5 ESXi650-201803401-BG ESXi650-201803402-BG
6.0 ESXi600-201803401-BG ESXi600-201803402-BG
5.5 ESXi550-201803401-BG ESXi550-201803402-BG

 Important knowledge-base articles update:

KB5208 – Note that Guest framework patch is what allows the VM guest OS to use new CPU instructions (IBPB,IBRS,STIBP). The v2 MCU patch covers the Intel Generation CPU on this KB only; which does not include Westmere and Nehalem product Family. You will see examples below of patching against Westmere system.

KB52345 – If you are a user that currently have patches for v1 MCU that caused reboot issues (ESXi650-201801402-BG, ESXi600-201801402-BG, and/or ESXi550-201801401-BG), which I covered previously, and have the workaround applied then know that the Guest OS framework patch will remove the workaround line from /etc/vmware/config when applied.

V2.4.2 of vDocumentation includes the following enhancement to Get-ESXSpeculativeExecution script.

ESXi Guidance

This will validate if you have installed Guest OS Framework Patch and MCU v2 Patch. These are the possible values:

  • Framework Missing – The Framework patch has not been installed (ESXi650-201803401-BG, ESXi600-201803401-BG, or ESXi550-201803401-BG)
  • Framework Installed – The Framework patch has been installed
  • v2 MCU Missing – The v2 MCU patch has not been installed (ESXi650-201803402-BG, ESXi600-201803402-BG, or ESXi550-201803402-BG)
  • v2 MCU Installed – The v2 MCU patch has been installed

Let’s look at a few examples to demonstrate this

Installing the new VMware patches on an ESXi host


Note the following on this example:

  • I applied the VMware workaround for previous v1 MCU patches to demonstrate what happens when installing the Framework patch. VMware MCU workaround applied = True
  • Looking at ESXi Build ID, you can see that it has not been patched (7526125)
  • BIOS guidance shows that I need to install BIOS v2.6.1 to have this system patched for Spectre
  • ESXi guidance shows both new patches are missing

Next I’ll install the Framework patch only:


  • ESXi Build ID is now updated, showing that I have patched this system
  • ESXi guidance shows that I have only applied the framework patch.
  • MCU CPUID still blank since I have not applied the BIOS update nor the ESXi v2 MCU patch
  • ESXi applied MCU = Inactive
  • VMware MCU workaround applied = False; installing the Framework patch removed the workaround line (cpuid.7.edx = “—-:00–:—-:—-:—-:—-:—-:—-“. ) from etc/vmware/config

Next I’ll install the v2 MCU Patch


  • BIOS guidance still shows that I need a BIOS update, and my current BIOS version is still v2.5.4
  • ESXi guidance shows that I have applied both patches
  • MCU CPUID shows the spectre mitigation instructions are present
  • ESXi applied MCU = Active. BIOS v2.5.4 is providing microcode revision 0x710, however the v2 ESXi patch is providing a newer revision: 0x713. The ESXi is then using the microcode provided by the patch

Next I’ll install the BIOS update


  • BIOS guidance shows that proper BIOS is installed. Current BIOS version is v2.6.1
  • ESXi applied MCU = Inactive, both microcode revisions are the same, so the BIOS provided rev is used.

Looking at an ESXi host with Westmere/Nehalem Intel CPU


Note the following on this example:

  • ESXi guidance shows that both patches have been installed. The Build (7967664) is current as well
  • This is a Westemere Intel CPU, so the patches does not contain the v2 microcode for this product family
  • BIOS guidance shows that there is a BIOS update available: vP65 (2/22/2018)
  • MCU CPUID is blank, no spectre mitigation CPU instructions present

Next, I’ll show you a Westmere system that has the BIOS update present


  • BIOS guidance shows that proper BIOS is installed. Current BIOS version is P65 (2/22/2018)
  • MCU CPUID shows the spectre mitigation instructions are present

Happy patching!!

Posted in PowerShell

Validating compliance of VMSA-2018-0004 (Spectre) on ESXi and VM

This is an update to  Validating Compliance of VMSA-2018-0002 and BIOS update.

VMware recently published VMSA-2018-004, which details Hypervisor-Assisted Guest Mitigation fixes as well as knowledge base (KB) article 52085 with instructions to verify the updated microcode for a Virtual Machine.

Version 2.4.0 of vDocumentation now includes enhancements to the Get-ESXSpeculativeExecution script cmdlet that will:

  • Validate if an ESXi is patched and compliant to VMSA-2018-0004
  • Validate if an ESXi is seeing the updated CPUID Microcode

If –ReportOnVMs switch is specified then this will also report on VM compliance.

This new version also includes a new script cmdlet: Get-VMSpeculativeExecution. This cmdlet accepts $VM object and is very useful for checking VM compliance. Make sure to run “get-help Get-VMSpeculativeExecution –full” to see all the help content.


Examples of Get-ESXSpeculativeExecution

If we run “Get-ESXSpeculativeExecution -esxi labesx001.local”, this is the output:


If we run “Get-ESXSpeculativeExecution –ReportOnVMs –exportexcel”, we will generate an Excel sheet with 3 TABs: Patch_Compliance, BIOS_Compliance, and VM_Compliance




Examples of Get-VMSpeculativeExecution

If we run “Get-VM “testvm39” | Get-VMSpeculativeExecution”, this is the output:


Other Examples

“Get-VM |   Get-VMSpeculativeExecution” Will check compliance on ALL VMs

“Get-Host “Labhost13.local” | Get-VMSpeculativeExecution” Will check compliance for all VMs running on host Labhost13.local

“Get-VM -Location “Dev_Cluster” | Get-VMSpeculativeExecution” Will check compliance for all VMs running on Dev_Cluster

Get-VMHost “Labhost13.local” | Get-VM | Get-VMSpeculativeExecution | Export-Excel “VMValidation.xlsx” -WorkSheetname “VMresults” Will check compliance for all VMs running on host Labhost13.local, which will exported to “VMValidation.xlsx” Excel file.


Posted in PowerShell

Validating compliance of VMSA-2018-0002 and BIOS update

UPDATE: Please see validating compliance of VMSA-2018-0004 (Spectre) on ESXi and VM

VMware has published VMSA-2018-0002 that addresses vulnerability for Spectre and Meltdown (CVE-2017-5753, CVE-2017-5715) and tells you which patch should be installed.

Along with this patch, you also need the required BIOS update. However, when managing a large VMware environment it’s hard to keep track of what has been patched. It’s also hard to remember if BIOS has been updated.

For this reason, I have added a new script to vDocumentation that will check compliance against VMSA-2018-0002 and the required BIOS update.

Version 2.3.0 now includes a script Cmdlet (Get-ESXSpeculativeExecution) that will aide in validating your environment. If you are new to vDocumentation, please make sure to check the GitHub project page on how to install it from the PowerShell Gallery. (

Running Get-ESXSpeculativeExecution

Run “get-help Get-ESXSpeculativeExecution –full” to see all parameters and switches available; below are the most frequently used:

  • The Script will validate ESXi versions 5.5, 6.0, and 6.5
  • You can validate ESXi only (-PatchCompliance switch)
  • You can validate BIOS only (-BIOSCompliance switch)
  • You can validate both (specify no switch)
  • –esxi to validate a host
  • -cluster to validate a cluster
  • -datcenter to validate a datacenter
  • No parameter to run against the entire vCenter
  • -exportExcel to export to Excel


If we run “Get-ESXSpeculativeExecution -esxi labesx001.local” , this is the output:


If we run “Get-ESXSpeculativeExecution –exportexcel”, we will generate an Excel sheet with 2 TABs: Patch_Compliance and BIOS_Compliance.



BIOS Version Check

The BIOS Compliance validation relies on accessing the BIOSUpdates.csv file, which is hosted on the project page:

If you don’t have access to the Internet you can download the CSV file locally and specify it using the –inputfile parameter. The CSV file contains HP and Dell models, with updates from these these official sources:

If you’re working on something other than HP or Dell, do let me know via the #vdocumentation Vmware code Slack channel, Twitter or the GitHub webpage and we can update the CSV file with your model.

The SafeFromSpectre field on the BIOS_Compliance Tab will be True if both ESXi and BIOS have been patched (see below).  This indicates you have addressed the vulnerability completely.


Posted in PowerShell, vDocumentation
Edgar Sanchez