Although Microsoft Hyper-V is a mature, stable, and reliable server virtualization platform, problems can, and sometimes do occur. That being the case, this eBook has been written in an effort to help Hyper-V administrators to diagnose various problems with the hypervisor and Hyper-V virtual machines.
Because it is nearly impossible to discuss every problem that could potentially occur in a Hyper-V environment, this book focuses on the more common problems that Hyper-V admins are likely to encounter. For each problem that is discussed, you will find a short description of the problem and one or more step by step solutions to the problem.
The solutions presented within this book are grouped by hardware components (storage related problems, network related problems, etc.). It is worth noting however, that problems can sometimes have multiple causes, so if you don't initially find any mention of the problem that you are experiencing, then you might try looking in a different section.
Many of the most common problems that Hyper-V administrators tend to experience are storage related. A variety of problems ranging from poor virtual machine performance to issues with high availability can potentially be related to performance.
Poor disk performance tends to be one of the more difficult problems to troubleshoot in a Hyper-V environment. This is because there are so many different things that can cause slow storage performance.
One common cause is resource contention. In any virtualized environment, there is a finite amount of storage I/O available, and this I/O must be shared among the virtual machines. If the I/O demand exceeds (or approaches) the I/O capacity, then performance will suffer. In these types of situations, it is usually advisable to move some virtual machines to an alternate Hyper-V host (assuming that local storage is being used). However, there are some other things that you can check to ensure optimal storage performance.
The first thing that you should check is virtual machine component placement. Virtual machine components should reside on dedicated storage, where they will not have to compete with the operating system for storage I/O. Microsoft recommends that this volume be structured as RAID 1+0 (sometimes referred to as RAID 10). RAID 1+0 delivers the performance of striping, with the redundancy of mirroring. Although some organizations use RAID 5 or RAID 6 volumes, RAID 5 and 6 are poor choices for write intensive virtual machines, because of the overhead involved in writing parity data.
While you are assessing your storage architecture, it is also a good idea to evaluate your storage connectivity. In some cases, the Fibre Channel or iSCSI connection between a storage array and the Hyper-V hosts may not perform as well as the actual storage array, and the connection can become a bottleneck.
Microsoft also recommends using VHDX based virtual hard disks, and virtual hard disks in VHD format should be converted to VHDX (https://blogs.technet.microsoft.com/ askpfeplat/2013/03/10/windows-server-2012-hyper-v-best-practices-in-easy-checklist-form/). Microsoft also recommends the use of fixed length virtual hard disks (which are sometimes referred to as fixed size virtual hard disks), as opposed to dynamically expanding virtual hard disks (https://technet.microsoft.com/en-us/library/mt589658.aspx).
Dynamically expanding virtual hard disks make use of thin provisioning. This means that the virtual hard disk initially consumes only a small amount of physical storage space, regardless of the virtual hard disk's capacity. When dynamically expanding virtual hard disks are used, Hyper-V only claims physical disk space as data is added to the virtual hard disk. When data is written to a dynamically expanding virtual hard disk, the virtual hard disk file grows to accommodate the new data. Conversely, fixed length virtual hard disks claim the full amount of required physical disk space upon being created.
If you were to create a 100 GB virtual hard disk, then the amount of physical disk space that would initially be consumed would be determined by the type of virtual hard disk that was created. A fixed length virtual hard disk would claim the full 100 GB of space right away. A dynamically expanding virtual hard disk might eventually consume 100 GB of space if the disk is filled to capacity, but would initially consume only 4 MB of space.
The primary advantage to creating dynamically expanding virtual hard disks is that it allows administrators to create virtual hard disks with little regard for physical storage availability. Doing so works well for situations in which it is difficult to estimate how much storage space a workload will ultimately need, or in situations in which physical storage upgrades are planned, but have not yet been performed. An administrator might for example, create a virtual hard disk that is larger than the available physical disk space, knowing that physical disk space can always be added later on in order to accommodate growth.
In spite of their advantages, dynamically expanding virtual hard disks do have their disadvantages. First, the physical storage can be over committed, and it is possible for virtual hard disk growth to exceed the physical hardware's capacity.
Another disadvantage is that dynamically expanding virtual hard disks do not perform quite as well as fixed length virtual hard disks. This is because of the overhead involved in expanding the virtual hard disk file each time that new data is written to the virtual hard disk. In some situations, performance may also suffer as a result of virtual hard disk file fragmentation.
When you create a virtual machine through Hyper-V Manager, the virtual machine uses dynamically expanding virtual hard disks by default.
You can verify a virtual machine's virtual hard disk type by completing these steps:
Figure 1 | Select the virtual hard disk that you want to inspect.
Figure 2 | The Virtual Hard Disk Properties dialog box lists the virtual hard disk type.
If you prefer to use PowerShell, then you can do so by using the Get-VHD cmdlet. The only parameter that you will have to specify is the Path. However, it can be helpful to append the Select-Object cmdlet as a way of filtering the output so that the cmdlet shows you only the information that you are interested in. For example, the following command displays the virtual hard disk's path, format, and type:
Get-VHD –Path <virtual hard disk path and filename> | Select-Object Path, VhdFormat, VhdType
You can see what this command looks like in action in Figure 3.
Figure 3 | You can use the Get-VHD cmdlet to retrieve information about a virtual hard disk.
Even if you discover that a virtual hard disk is configured to use thin provisioning, it does not necessarily mean that you have a problem. Thin provisioning does have its place. However, if you do decide that it is in your best interest to convert a dynamically expanding virtual hard disk into a fixed length virtual hard disk, then you can do so by completing these steps:
Figure 4 | Click on the virtual hard disk, and then click the Edit button.
Figure 5 | Choose the Fixed Size option, and click Next.
NOTE: Hyper-V does not actually convert the virtual hard disk to a new format, but rather creates a brand new virtual hard disk. As such, you will have to provide a virtual hard disk name that differs from the name that was originally used. Be sure to pay close attention to the path, because Hyper-V will attempt to place the virtual hard disk on the server's system drive by default. It is also worth noting that you will need to have a sufficient amount of free physical disk space to accommodate both copies of the virtual hard disk.
NOTE: As a best practice, you should shut down the virtual machine before attempting this process.
If you prefer, you can use Windows PowerShell to convert a dynamically expanding virtual hard disk into a fixed length virtual hard disk. In order to do so, you will need to know the path and filename used by the original virtual hard disk. You will also have to provide a destination path and filename for the new virtual hard disk. The cmdlet that is used to convert the virtual hard disk is Convert-VHD. The syntax is:
Convert-VHD –Path <path and filename of the virtual hard disk to be replaced> -DestinationPath <path and filename of the new virtual hard disk> -VHDType Fixed
You can see an example of such a conversion in Figure 6.
Figure 6 | You can use the Convert-VHD cmdlet to convert a virtual hard disk.
Keep in mind however, that there is usually more to the process than simply converting the virtual hard disk. You will typically need to shut down the virtual machine, convert the virtual hard disk, and then remove the old virtual hard disk and replace it with the newly created virtual hard disk.
This process requires you to know the virtual hard disk's controller type, controller number, and controller location, so you will have to look this information up as a part of the process. The commands used in doing so are:
Stop-VM –Name <virtual machine name>
Convert-VHD –Path <path and filename of the virtual hard disk to be replaced> -DestinationPath <path and filename of the new virtual hard disk> -VHDType Fixed
Get-VMHardDiskDrive –VMName <virtual machine name>
Remove-VMHardDiskDrive -VMName <virtual machine name> –ControllerType <IDE or SCSI> -ControllerNumber <disk controller number> -ControllerLocation <virtual hard disks location on the controller>
Get-VMHardDiskDrive –VMName <virtual machine name>
ADD-VMHardDiskDrive -VMName <virtual machine name> -Path <virtual hard disk path and filename>
Get-VMHardDiskDrive –VMName <virtual machine name>
You can see the entire process in action in Figure 7:
Figure 7 | You can use PowerShell to convert and replace a virtual hard disk.
The existence of checkpoints also impacts virtual hard disk performance. Furthermore, Microsoft advises against the use of checkpoints in production environments until the release of Windows Server 2016, which will feature a production checkpoint feature. The reason for this is because prior to the release of Windows Server 2016, the Hyper-V checkpoint feature (or snapshot feature as it was known in Windows Server 2008 and 2008 R2) was not application aware. As such, creating and then later applying a checkpoint on an application server had the potential to cause corruption or data loss. The Windows Server 2016 production checkpoint feature will utilize the Volume Shadow Copy Services (VSS) as a part of the checkpoint process, so as to ensure that VSS aware applications can be safely check pointed. VSS is the same mechanism that backup applications use to safely back up Microsoft application servers.
Checkpoints don't usually have a major impact on write performance, assuming that the checkpoints are stored on a high speed volume. However, checkpoints can have a significant impact on read performance. A virtual machine's read performance tends to correspond directly to the number of checkpoints that exist for the virtual machine. The reason for this is that each checkpoint represents a differencing disk. When a virtual machine receives a read instruction, Hyper-V has to read each differencing disk in sequence, until it finds the requested data. You can remove unwanted checkpoints by completing these steps:
Figure 8 | You can improve a virtual machine's read performance by removing unwanted checkpoints.
It is also possible to remove an unwanted virtual machine checkpoint by using PowerShell. To do so, you will have to use the Remove-VMSnapshot cmdlet. Checkpoints were referred to as snapshots prior to the release of Windows Server 2012, hence the reference to snapshots in the cmdlet.
Unless you know the name of the checkpoint that you want to remove, you will need to retrieve a list of checkpoints from the server. You can accomplish this task by using the GetVMSnapshot cmdlet, and using the –VMName switch to provide the name of the virtual machine that you want to examine. The commands used to retrieve a list of checkpoints and then remove a specific checkpoint are:
Get-VMSnapshot –VMName <virtual machine name>
Get-VMSnapshot –VMName <virtual machine name> –Name <checkpoint name> | Remove-VMSnapshot
Get-VMSnapshot –VMName <virtual machine name>
Figure 9 | It is possible to use PowerShell to remove a checkpoint.
Incidentally, checkpoint files tend to grow over time. Believe it or not, this behavior is by design. When an administrator creates a checkpoint, Hyper-V creates a differencing disk. From that point on, all of the virtual machine's write operations are directed to the differencing disk, rather than to the virtual machine's original virtual hard disk. The original virtual hard disk becomes read only, and is left in the state in which it existed when the checkpoint was created. It is this behavior that makes it possible to easily roll a virtual machine back to an earlier point in time. Because the original virtual hard disk becomes read only, and because all future write operations are directed to the differencing disk, the checkpoint's size will continue to grow until the checkpoint is either removed, or replaced with a newer checkpoint.
iSCSI storage is a popular choice for SMBs that want to create a cluster shared volume, without the cost or complexity of Fibre Channel. Although the iSCSI configuration process tends to be relatively straightforward, there are some common mistakes that can cause iSCSI multipath to not work correctly.
The Windows Server operating system has built-in support for multi-path I/O for the purpose of providing high availability. Multi-path I/O works by enabling multiple connections (sessions) to a storage array. This makes it possible for each node in a Windows failover cluster to have its own connection to the storage hardware that hosts the cluster shared volume. As such, each node in the cluster has access to exactly the same storage resources.
Although this book addresses multi-path I/O from an iSCSI standpoint, it is important to understand that multi-path I/O is not unique to iSCSI. The Microsoft multi-path I/O implementation supports the use of iSCSI, Fibre Channel, and Serial Attached Storage (SAS).
Microsoft's multi-path I/O feature uses a Device Specific Module (DSM) to provide support for storage arrays that natively support the Asymmetric Logical Unit Access (ALUA) model, as defined by SPC-3. Also supported are hardware arrays that adhere to the Active / Active controller model. Microsoft multi-path I/O feature also allows for hardware independent multi-path I/O at the iSCSI level.
The first step in troubleshooting any multi-path I/O related problem is to verify that Multi-Path I/O is enabled on the server. The easiest way to accomplish this is to use Windows PowerShell. To do so, open an elevated PowerShell window, and enter the following command:
Get-WindowsFeature –Name 'Multipath-IO'
As you can see in Figure 10, this command will show you the installation state for the MultiPath I/O feature.
Figure 10 | You can use the Get-WindowsFeature cmdlet to check whether or not the Multi-Path I/O feature is installed.
If you determine that the Multi-Path I/O feature is not installed, then you can easily install it from within PowerShell. Keep in mind that the Multi-Path I/O feature must be installed onto each server that will need multi-path access to the storage array. The command used to install the multi-path I/O feature is:
Enable-WindowsOptionalFeature –Online –FeatureName MultiPathIO
You can see what this process looks like in Figure 11.
Figure 11 | You can use PowerShell to enable multi-path I/O
Installing the multi-path I/O feature results in the Microsoft Device Specific Module (DSM) also being installed. In addition, Windows also installs the MPIO Control Panel, which allows administrators to configure MPIO functionality, create MPIO configuration reports, and install additional DSMs (to support additional storage products). You can access the MPIO Control Panel by navigating through the Windows Control Panel to System and Security \ Administrative Tools \ MPIO.
NOTE: In a Server Core environment, you can launch the MPIO Control Panel by running MPIOCPL.EXE command.
The MPIO Control Panel, which you can see in Figure 12, is divided into four tabs – MPIO Devices, Discover Multi-Paths, DSM Install, and Configuration Snapshot.
Figure 12 | This is the MPIO Control Panel.
The MPIO Devices tab allows admins to add support for new storage devices. Simply click the Add button and enter the vendor and product IDs. The vendor ID is an eight-character string, and the product ID is a sixteen-character string. If you need to add multiple devices, you can do so by separating devices with a semi-colon.
The Discover Multi-Paths tab can be used to add device IDs for Fibre Channel devices that use Microsoft DSM. This tab can also be used to determine whether or not multiple instances actually point to a common Logical Unit Number (LUN).
The DSM Install tab is used to add support for storage devices that do not support the MPIO architecture. Most SPC-3 compliant arrays will work with the DSM provided by Microsoft, but some hardware vendors so provide their own DSM. You can add a third party DSM by using the Browse button to locate the corresponding .INF file, and then clicking the Install button.
The Configuration Snapshot tab exists primarily for troubleshooting purposes. By using this tab, administrators can save the MPIO configuration to a text file for review. You can see this tab and a configuration snapshot (taken from a system with no attached storage) in Figure 13.
Figure 13 | The Configuration Snapshot tab produces a report detailing registered DSMs, devices, and load balancing policies.
As you can see in the figure above, the MPIO Storage Snapshot report provides information about registered DSMs, controlled devices, and target load balancing policies. This report can be useful for troubleshooting, but the report tends to be even more useful if a comparison can be made between a report that was created when the system was functioning properly, and a report that was created after a problem started.
The most common iSCSI configuration issues involve multipath not being enabled, and the iSCSI initiator not being allowed to access the iSCSI target. The iSCSI target configuration process varies by vendor, but typically the target is treated as being multipath enabled, so long as each of the iSCSI initiators have permission to access the target. When the first iSCSI initiator establishes connectivity however, the initiator must be configured for multipath support. Otherwise, the other cluster nodes will be unable to connect to the iSCSI target.
As previously noted, the iSCSI configuration steps vary depending on which vendor's solution you are using. Assuming that you are using the Windows Server 2012 R2 iSCSI Initiator and iSCSI Target, you can verify the permissions required for multipath by completing these steps:
Figure 14 | Document the name of each iSCSI Imitator.
Figure 15 | Each initiator must have permission to connect to the target.
Figure 16 | You must select the Enable Multi Path checkbox.
One problem that many Hyper-V administrators have reported is that of extremely slow file transfers. In some situations, the file transfers can be so slow that the file copy process times out before the file transfer can be complete.
Before you begin diagnosing the file copy process, it is worth noting that the problem may not be related to Hyper-V itself. Slow file copy operations can be caused by any number of factors (such as resource contention, hypervisor level storage throttling, etc.), so it is important to use performance monitoring metrics to determine the underlying cause of the problem prior to trying to make any corrections.
If you do determine that a Hyper-V host or a virtual machine is the source of your file copy performance problems, then there are two things that you should check.
First, make sure that your Hyper-V servers are running all of the latest patches. Early on, Windows Server 2012 and Windows Server 2012 R2 were plagued by extremely slow file transfers (for certain hardware configurations). It was possible to work around these problems by disabling SMB signing, However, Microsoft eventually corrected that particular problem with a patch.
Another common reason for extremely slow file copy is an incorrectly configured network adapter. In some situations, the Virtual Machine Queue (VMQ) may be enabled by default. When the VMQ feature is enabled, Hyper-V creates a dedicated queue on the physical network adapter for each virtual network adapter that requests a queue (https://technet. microsoft.com/en-us/library/gg162704(v=ws.10).aspx).
According to Microsoft (https://technet.microsoft.com/en-us/library/gg162704(v=ws.10).aspx), virtual machine queues should only be enabled for virtual machines that experience heavy volumes of inbound traffic. More importantly, Microsoft recommends that VMQ be disabled for 1 gigabit Ethernet NICs. Gigabit Ethernet NICs cannot derive any significant benefit from VMQ, and the use of VMQ on such NICs has sometimes been known to performance and availability issues (https://www.petri.com/hyper-v-network-issues-1-gbe-nics).
It is important to note that even if you manually disable VMQ, some vendors automatically reenable it as a part of the driver update process. Therefore, it is a good idea to check the VMQ status, even if VMQ was previously disabled.
A NIC's virtual machine queue can be disabled by completing the following steps:
Figure 17 | Locate the problematic network adapter within the Device Manager.
Figure 18 | Locate and disable the virtual machine queue.
If you prefer to use PowerShell, you can check to see whether VMQ is enabled on your network adapters by entering the following command:
This cmdlet will display the VMQ state for each network adapter, as shown in Figure 19.
Figure 19 | The Get-NetAdapterVMQ cmdlet will display the VMQ state for each network adapter.
If PowerShell reveals that VMQ is enabled for a particular network adapter, and you wish to disable VMQ, then you can do so by using the following command:
Disable-NetAdapterVMQ –Name <network adapter name>
You will notice that this command requires you to supply the network adapter name. The name of each network adapter is revealed when you enter the Get-NetAdapterVMQ cmdlet. For example, the network adapter shown in the previous figure uses the name MyNICTeam, and that is the name that you would use if you were disabling VMQ for that particular network adapter. Figure 20 shows what this process looks like. In this case, VMQ is already disabled, but you can still see how to use the various commands.
Figure 20 | You can use the Disable-NetAdapterVMQ cmdlet to disable VMQ for a specified network adapter.
Another somewhat common problem that is often storage related involves a situation in which virtual machines cannot be powered on. Although this problem can sometimes be permissions related, it is more often related to the inability of Hyper-V to communicate with the storage on which virtual machine components reside. When this problem occurs, the Hyper-V Manager will hide the option to power on the virtual machine, and will also display a status message indicating that Hyper-V cannot connect to virtual machine storage. You can see an example of such a situation in Figure 21.
Figure 21 | Virtual machines cannot be powered on.
As you look at the figure above, you will notice that some of the virtual machines are listed as being Off-Critical, while others have a state of Saved-Critical. You will also notice that the status indicates that Hyper-V cannot connect to virtual machine configuration storage.
This problem can be caused by a storage failure, a storage disconnection, or by a change to a drive mapping. The problem can also stem from storage permissions being changed in a way that prevent Hyper-V from being able to read and write to the storage medium.
In this particular example, the problem was caused by a detached virtual disk within Windows Storage Spaces (not Hyper-V itself). Keep in mind however, that the same problem can occur even if you are not using Storage Spaces. Virtual machines that use Direct Attached Storage (DAS), iSCSI, Fibre Channel, etc., can all experience the same problem if a storage array or the connectivity to that array were to fail.
If you are using Windows Storage Spaces, then the Server Manager will often show you the cause of the problem. If you look at Figure 22 for example, you can see that the virtual disk is displaying a warning, and that right clicking on the disk provides the option to attach the virtual disk.
Figure 22 | You can use the Disable-NetAdapterVMQ cmdlet to disable VMQ for a specified network adapter. The virtual disk that Hyper-V is using for virtual machine storage is detached.
It is worth noting that although a virtual disk can be attached within a matter of a couple of seconds, the Hyper-V virtual machines do not become immediately available. It can take a minute or two for Windows to re-establish the drive mappings for the recently attached virtual disk. Hyper-V will eventually recognize the change and make the virtual machines available (it can take up to about five minutes), but you may be able to speed things up by using the Refresh buttons found within Server Manager and Hyper-V Manager. Similar behavior can be expected when reattaching other forms of storage, even if Windows Storage Spaces is not used. It may take Hyper-V a few minutes to acknowledge that the problem has been fixed.
Often times problems that occur within Hyper-V can be attributed to permissions problems. These problems may result in Access Denied errors, and a variety of other error messages. In fact, there are too many different types of permissions related errors to be able to list them all here. Even so, the vast majority of these errors can be attributed to one of two things:
The first type of error that you could potentially receive indicates that an error has occurred while attempting to connect to the server. The actual text of this error message can vary depending on the version of Windows that you are using and on the conditions that caused the error to be displayed, but you can see an example of such an error in Figure 23.
Figure 23 | The Hyper-V Manager is unable to connect to Hyper-V.
This error usually occurs when an administrator attempts to access a remote Hyper-V server, but under the right circumstances, it can occur even if the Hyper-V Manager is being run directly on the Hyper-V Server. This error is often related to a permission problem. The administrator who is launching the Hyper-V Manager may lack permissions to the remote Hyper-V server. Before you can assume that a permissions problem is to blame however, you should check three things.
Figure 24 | Make sure that the Hyper-V Virtual Machine Management Service is running.
If you prefer to check this service by using PowerShell, you can do so by entering this command:
If you find the service to be stopped, then you can start the service by entering this command:
You can see these commands being used in Figure 25.
Figure 25 | You can use PowerShell to start the Hyper-V Virtual Machine Management Service.
As previously mentioned, an incorrectly configured firewall can make it impossible to remotely manage a Hyper-V host. The firewall ports that are required vary depending on whether you are using the Hyper-V Manager or System Center Virtual Machine Manager. The sections below list the required firewall ports.
Hyper-V Ports: Hyper-V uses the following firewall ports:
Hyper-V WMI (Async-In)
Hyper-V WMI (DCOM-In)
Hyper-V WMI (TCP-In)
RPC Dynamic Ports
RPC Endpoint Mapper
Hyper-V Management Clients – WMI (Async-In)
Hyper-V Management Clients – WMI (DCOM-In)
Hyper-V Management Clients – WMI (TCP-In)
Hyper-V Replica HTTP Listener (TCP-In)
Hyper-V Replica HTTPS Listener (TCP-In)
You can see a listing of Microsoft's firewall rules for Hyper-V in Figure 26.
Figure 26 | These are the Hyper-V related firewall rules that exist within Windows Server 2012 R2.
System Center Virtual Machine Manager Ports: Microsoft defines a separate list of firewall ports for use with System Center 2012 R2 Virtual Machine Manager. These ports are outlined in a TechNet article at: https://technet.microsoft.com/en-us/library/cc764268.aspx
The required ports are:
VMM server to VMM agent on Windows Server– based host (control)
VMM server to VMM agent on Windows Server– based host (file transfers)
HTTPS (using BITS)
443 (Maximum value: 32768)
VMM server to remote Microsoft SQL Server database
VMM server to P2V source agent
VMM Administrator Console to VMM server
VMM Self-Service Portal Web server to VMM server
VMM Self-Service Portal to VMM self-service Web server
VMM library server to hosts
443 (Maximum value: 32768)
VMM host-to-host file transfer
443 (Maximum value: 32768)
VMRC connection to Virtual Server host
VMConnect (RDP) to Hyper-V hosts
Remote Desktop to virtual machines
VMware Web Services communication
SFTP file transfer from VMWare ESX Server 3.0 and VMware ESX Server 3.5 hosts
SFTP file transfer from VMM server to VMWare ESX Server 3i hosts
Assuming that DNS name resolution is configured correctly, the correct firewall ports are open, and the Hyper-V Virtual Machine Management Service is running, then issues with accessing Hyper-V from the Hyper-V Manager are probably going to be permissions related. To correct the problem (in a non-domain environment), you will need to use the Component Services tool (Dcomcnfg.exe). The reason why you will need to use this tool is because permissions are applied at the Component Object Model (COM) level, and the Component Services tool allows you to modify COM level permissions.
You can modify the COM permissions by completing these steps:
Figure 27 | Right click on the My Computer container.
Figure 28 | Locate the Access Permissions section, and click the Edit Limits button.
Figure 29 | Enable anonymous remote access.
It is worth noting that although this technique works, it does decrease security. It is best to avoid using this technique if possible. For example, rather than running the Hyper-V Manager from a non-domain joined workstation (on which this fix would most likely be required), an administrator might instead opt to use a domain joined workstation, or establish an RDP session to a Hyper-V Server and run the Hyper-V Manager from there.
If you are using System Center Virtual Machine Manager for Hyper-V Management, then access denied errors can sometimes occur as a result of a RunAs account that has been removed, disabled, or that lacks the appropriate permissions. When an administrator performs an action within System Center Virtual Machine Manager, the server treats the action as a scheduled job, even if the action is to occur immediately. As such, many actions require the use of a RunAs account. This account's permissions are used in place of the administrator's permissions. You can check for the existence of a RunAs account by completing these steps:
Figure 30 | Make sure that your RunAs account appears on the list of RunAs accounts.
One of the more common Hyper-V problems involves the inability to live migrate a virtual machine from one host to another. There are countless issues that can cause live migrations to fail, but the three most common are:
Administrators must give Hyper-V hosts permission to receive inbound live migrations. To verify that the necessary permissions are in place, complete the following steps:
Figure 31 | Make sure that live migrations are enabled.
As an alternative, you can use PowerShell to check to see if live migration is enabled. The command used for doing so is:
(Get-VMHost <virtual machine name>).VirtualMachineMigrationEnabled
If live migration is enabled, then this command should return a value of True, as shown in Figure 32.
Figure 32 | You can use PowerShell to check to see if live migration is enabled for a host.
If you need to enable live migration, you can do so by using the Enable-VMMigration cmdlet. You will typically also need to configure the migration network and the migration protocol (the migration protocol will be discussed in the next section). The commands used for these steps are:
Set-VMMigrationNetwork <network IP>
Set-VMHost –VirtualMachineMigrationAuthenticationType <authentication protocol>
An administrator's choice of authentication protocol can cause live migrations to fail. The source and destination servers must use the same authentication protocol, but there is more to it than that.
Hyper-V gives you two authentication protocol choices – CredSSP and Kerberos. The CredSSP protocol has a hop count limit of one. This limitation means that if the CredSSP protocol is being used, then you must be initiating the live migration locally on the Hyper-V server that contains the virtual machine that you want to live migrate. If you are using CredSSP and initiate live migration from another host, then the live migration will fail.
The Kerberos protocol is much more flexible, and more secure. However, Microsoft recommends enabling constrained delegation if you are going to use Kerberos.
You can verify a Hyper-V server's live migration authentication protocol by completing these steps:
Figure 33 | The Authentication Protocol options exist on the Advanced Features container.
If you prefer, you can verify the live migration authentication type using PowerShell. The command for doing so is:
(Get-VMHost <virtual machine name>).VirtualMachineMigrationAuthenticationType
As you can see in Figure 34, this command causes the authentication protocol to be displayed.
Figure 34 | PowerShell can tell you which authentication protocol is being used for live migrations.
If you need to switch to a different authentication protocol, then you can do so by using this command:
Set-VMHost –VirtualMachineMigrationAuthenticationType <authentication protocol>
Figure 35 shows the process of switching from CredSSP authentication to Kerberos authentication.
Figure 35 | You can use PowerShell to change the live migration authentication type.
There is no rule that says that your Hyper-V servers must be identical in every way, but if you are going to be performing a live migration then the source and destination must be similar enough to allow for a successful migration.
The hardware component that tends to cause the most problems with live migrations is the CPU. If your source and destination servers contain different CPU architectures, then you can sometimes force a live migration to succeed by disabling advanced CPU features. Depending on the hardware that you are using, it may be possible to re-enable the advanced CPU features once the live migration is complete.
You can disable a virtual machine's advanced CPU features by following these steps:
Figure 36 | Select the Migrate to a Physical Computer with a Different Processor Version checkbox to disable the CPU's advanced features.
Depending on which tool is being used to perform a live migration, live migrations can also fail if the source and destination servers do not contain identically named virtual switches. You can verify a virtual machine's virtual switch name by completing these steps:
Figure 37 | Make note of the virtual switch name, and create an identically named virtual switch on the destination host if necessary.
If you prefer to use PowerShell, then you can use the Compare-VM cmdlet to test the suitability of a Hyper-V host to receive a particular virtual machine. This cmdlet requires you to provide the virtual machine name and the name of the destination host. The syntax for this command is:
Compare-VM –Name <virtual machine name> -DestinationHost <destination Hyper-V host server name>
Figure 38 shows an example of how this command can be used.
Figure 38 | You can use the Compare-VM cmdlet to check Hyper-V's ability to live migrate a virtual machine.
As you can see in the figure above, Compare-VM reported some errors, but those errors are in numerical form, and require further explanation. To get an explanation of errors that you might receive, enter the following commands:
$Report = Compare-VM –Name <virtual Machine Name> -DestinationHost <Hyper-V Destination Host Server>
$Report.Incompatibilities | Format-Table –AutoSize
You can see an example of the output from these commands in Figure 39. In this particular case, there are two main problems occurring. First, the destination server does not contain a virtual switch with a name that matches that of the virtual switch that is currently being used by the VM. Second, four virtual hard disks cannot be found. The virtual hard disk errors are occurring because some of the virtual hard disks that the virtual machine is configured to use have been intentionally deleted for the sake of demonstration.
Figure 39 | This is what an incompatibility report looks like.
Many of the commonly reported Hyper-V problems are related to an inability to back up Hyper-V virtual machines. Hyper-V can be backed up at the host level or at the guest level. Host level backups are generally the preferred method of backing up Hyper-V, because they occur at the Hyper-V Layer, which allows for more efficient backup operations, and removes the need for software agents. However, It is worth noting, that it is still relatively common to see backup vendors still focused on backing up Hyper-V virtual machines from within the guest, via a software agent. Let's cover each scenario briefly
Guest level backups are made at the virtual machine level, and usually involve installing a backup agent directly onto the virtual machine. Although guest level backups do have their place, there are also some disadvantages to using them. One disadvantage is that guest level backups can be more difficult to manage than host level backups, because most organizations have far more virtual machines than Hyper-V hosts.
A second disadvantage to guest level backups is that the backup application backs up the virtual machine contents, not the full virtual machine. In other words, the backup application will back up the contents of the virtual machine's virtual hard disk, but does not back up external virtual machine components such as checkpoints or hardware configurations. The reason for this is that the backup agent is running inside of a virtual machine, and is therefore unaware of that the operating system is running on virtual hardware.
When a backup application performs a host level backup, there are two methods that are commonly used to back up the virtual machines. One method is known as the saved state method. The backup application momentarily places the virtual machine into a saved state and then creates a checkpoint. Once the checkpoint has been created, the virtual machine is returned to a running state and a backup is created. The checkpoint allows the backup software to back up the virtual machine without having to worry that the virtual hard disk contents will be modified before the backup can be completed. After the backup completes, the checkpoint is deleted.
The other backup method is known as the Child VM Snapshot Method. This method also relies on the use of checkpoints, but is the preferred backup method because it does not require the virtual machine to be placed into a saved state. This method uses the Volume Shadow Copy Services (VSS) to back up a virtual machine. VSS will be discussed in the next section.
It is worth noting that a backup operator will not typically be asked to choose between backup methods. The backup software examines each virtual machine to see if it meets the criteria for a Child VM Snapshot Backup. If the criteria cannot be met, then a Saved State backup is created instead. It is normal for some virtual machines to be backed up using the Child VM Snapshot Method while others are backed up using the Saved State method.
There are a number of criteria that must be met if a virtual machine is to be backed up using the Child VM Snapshot Method:
Figure 40 | The Backup (Volume Checkpoint) integration service must be enabled.
Figure 41 | The Disk Management Console will show you the guest operating system's storage configuration.
The Volume Shadow Copy Services (VSS), which is the mechanism used to create Child VM Snapshot Backups, make use of three main parts. These parts include:
The VSS Requestor (the backup application) initiates the backup process. The requestor announces to the server that it needs to create a snapshot. It then queries the server to determine which VSS writers are installed. Each VSS aware application has its own writer. The requestor then instructs each writer to prepare for a backup. The writers perform a quiescence, in order to in order to get the data ready to be backed up. Once this process is complete, the VSS requestor instructs each writer to create a snapshot. Once this process has finished, the provider tells the requestor where to locate the data for use in the backup process. When the backup finishes, the VSS requestor informs the VSS writers that the backup is complete, and the writers will then perform any required post-backup tasks, and then allow the server to resume normal operations. You can see a diagram of this process in Figure 42.
Figure 42 | These are the primary VSS components.
Troubleshooting the Volume Shadow Copy Service:
The primary tool for diagnosing VSS problems is a command line tool called VSSAdmin. You can for example, use the VSSAdmin List Writers command to verify that each of the writers is stable, and is not in an error condition. You can see an example of this in Figure 43. You can see other VSSAdmin command line switches by entering the VSSAdmin /? command.
Figure 43 | You can use the VSSAdmin List Writers command to verify writer health.
Hopefully, your writers are in a healthy state and no errors are reported. If however, errors are shown, then Dell provides guidance for correcting the problem at: https://support.software. dell.com/kb/117647
In a Hyper-V environment, VSS-based host level backups of virtual machines are made possible through the Hyper-V Integration Services. If a Windows virtual machine is not being backed up correctly, then it is a good idea to make sure that the Backup Integration Service is enabled for the virtual machine. You can do so by completing these steps:
Figure 44 | Make sure that the Backup (Volume Checkpoint) integration service is enabled.
If you would like to use PowerShell to check to see whether the Backup (Volume Checkpoint) integration Service is enabled, you can do so by entering the following command:
Get-VM –Name <virtual machine name> | Get-VMIntegrationService –Name VSS
You can see what this command looks like in Figure 45. In this case, the Enabled column shows a value of True, indicating that the Backup (Volume Checkpoint) integration service is enabled.
Figure 45 | You can use PowerShell to test whether the Backup (Volume Checkpoint) integration service is enabled.
One of the nice things about PowerShell is that it makes it easy to verify the existence of the Backup (Volume Checkpoint) integration service across a large number of virtual machines. If for example, you wanted to check the status of the Backup (Volume Checkpoint) integration service for every virtual machine on a particular host, you could do so by using the following command:
Get-VM | Get-VMIntegrationService –Name VSS
You can see the command's output in Figure 46.
Figure 46 | You can check the integration services on multiple virtual machines.
Another problem that Hyper-V administrators sometimes encounter is that the backup process creates checkpoints (snapshots), and these checkpoints do not always go away when the backup completes.
Dealing with a lingering backup checkpoint requires extreme caution. Typically, if a checkpoint remains following a backup, it means that the backup application is still using the checkpoint, or that a failure occurred within a backup, even if that failure was unreported. Although it is possible to manually delete a backup checkpoint, you should always contact your backup vendor's support department first, in an effort to confirm that it is safe to remove the checkpoint. Otherwise, removing the checkpoint could break the backup application.
In order to understand why this problem happens, and how the fix works, it is necessary to understand how checkpoints are used by the backup process. A checkpoint is really nothing more than a differencing disk that has a parent / child relationship with a virtual machine. When you back up a virtual machine, Hyper-V may (depending on the type of backup that you are creating) create a differencing disk. This differencing disk intercepts any write operations that might occur during the backup process. That way, Hyper-V does not have to worry about the virtual hard disk's contents being modified during the backup process. After the backup process completes, then the differencing disk's contents are merged with the primary virtual hard disk, and the differencing disk is deleted. At least that's the way that the process is supposed to work. If the backup fails, then the checkpoint can remain in place.
Unfortunately, there is no easy way of removing this checkpoint. Most of the time, the only real option is to delve into PowerShell.
To remove the unwanted checkpoint, complete these steps:
Figure 47 | The Get-VM cmdlet displays a list of the virtual machines that exist on the host.
Figure 48 | PowerShell returns information about the snapshot.
Figure 49 | The checkpoint no longer exists.
Although Hyper-V can be run on a standalone server, Hyper-V servers are often configured to use the Windows Failover Clustering feature, so as to make virtual machines highly available. This section discusses some of the more common issues that occur in clustered deployments.
Although some organizations create single server Hyper-V deployments, production Hyper-V deployments need to be clustered. The reason for this is simple. A production Hyper-V host typically hosts multiple production workloads. If that host server were to fail, then all of the virtual machines that are running on the host would also fail, resulting in a major outage.
One somewhat common problem that Hyper-V administrators encounter is that clusters do not behave as expected. Virtual machines may for example, fail to failover to another Hyper-V host. There are a number of different things that can cause this problem.
One common cause of this problem is that the virtual machines have not been made highly available. It isn't enough to deploy the Windows Failover Clustering Feature onto your Hyper-V Servers. You must make the individual virtual machines highly available by configuring the virtual machines to act as a clustered role. You can determine whether or not a virtual machine has been made highly available by completing these steps:
Figure 50 | Make sure that your virtual machines are listed as clustered roles.
If you discover that your virtual machines are not listed, then you can click on the Configure Role link, found in the console's Actions pane. This will cause Windows to Launch the High Availability Wizard. Now, you must simply follow the wizard's prompts and choose Virtual Machine as the role type and then choose the virtual machine that you wish to make highly available.
If you work through the High Availability Wizard and discover that the virtual machines that you wish to make highly available are not listed, then there are a couple of possible things to check. Make sure that the virtual machine resides on a Hyper-V server that acts as a part of the cluster.
Even if a failover cluster is configured correctly, and virtual machines have been configured to be highly available, the failover process can still fail if there are not sufficient system resources available on the destination host. The destination host must have adequate CPU, memory, and other hardware resources available if it is to be able to run virtual machines that were previously running on a different host. This can be a problem for organizations that attempt to maximize virtual machine density, because the Hyper-V servers may already be running so many virtual machines that they simply do not have the resources to accommodate other virtual machines in the event of a failover.
Even if the destination host has plenty of resources available for running some of the virtual machines from the failed host, those resources might not be adequate for hosting every virtual machine that was previously running on the failed host.
One way to check for this condition is to examine the host's even log entries. Windows Server stores log entries related to virtual machine high availability in the Microsoft-Windows-HyperV-High-Availability event log. You can access this event log by opening the Event Viewer and navigating to Application and Service Logs \ Microsoft \ Windows \ Hyper-V-High-Availability.
One event to watch out for is:
Event ID: 21502
Description: Virtual Machine <virtual machine name> live migration did not succeed at the source. Migration failed.
This error can occur is there isn't enough physical memory on the destination host to accommodate the virtual machine that is being migrated. Memory is often the limiting resource when it comes to live migrations or failovers.
In situations in which the destination host does not have adequate resources to accommodate all of the inbound virtual machines, you can prioritize virtual machine failovers based on each individual virtual machine's importance.
To configure a priority for a virtual machine, complete these steps:
Figure 51 | Set a priority for the virtual machine and click OK.
Virtual machines can also be prioritized through the Virtual Machine Manager console. To do so, follow these steps:
Figure 52 | You can configure virtual machine priorities through the Virtual Machine Manager console.
Hyper-V is a very stable and reliable, enterprise class hypervisor. Even so, problems can, and sometimes do occur. In many cases, these problems are caused by simple configuration issues. In other cases, problems may be attributed to resource contention.
If you experience Hyper-V problems that have not been discussed in this book, then you might consider looking at Microsoft's Hyper-V resources on TechNet: https://technet.microsoft.com/ en-us/library/mt169373(v=ws.11).aspx. For more serious problems, you can contact Microsoft Support (https://technet.microsoft.com/en-us/windowsserver/hh553005.aspx).
Brien Posey is a 14 time Microsoft MVP with over two decades of IT experience. Prior to going freelance, Brien worked as CIO for a national chain of hospitals and healthcare facilities. He has also served as a network engineer for the United States Department of Defense at Fort Knox and as a network administrator for some of the country's largest insurance companies. In addition to his work in IT, Brien is currently training to be a civilian astronaut. Fore more information visit brienposey.com or twitter.com/BrienPosey
Altaro Software (www.altaro.com) is a fast growing developer of easy to use backup solutions used by over 30,000 customers to back up and restore both Hyper-V and VMware-based virtual machines, built specifically for SMBs with up to 50 host servers. Altaro take pride in their software and their high level of personal customer service and support, and it shows; Founded in 2009, Altaro already service over 30,000 satisfied customers worldwide and are a Gold Microsoft Partner for Application Development and Technology Alliance VMware Partner.