Advanced : VMware HA Important Points

Advanced : VMware HA Important Points

post source : isupportyou

VMWare HA Important Points

  • HA maintains the high availability of virtual machines when an event of host failure / isolation occurs by powering on them on running hosts.
  • Every host in cluster exchanges its heartbeat with other hosts to notify them that it is alive.
  • A host is declared as isolated when its heartbeat is not received within 12 seconds.
  • A host is declared as dead when its heartbeat is not received within 15 seconds, we can increase this duration to avoid false positives by defining an advanced setting das.failuredetectioninterval in vCenter.
  • If we set das.failuredetectioninterval to 60 seconds we can avoid false isolations, which means if an isolated host comes back within 60 seconds VM’s will continues to run on the same host, which means HA will never interfere.
  • When a host is declared as isolated after defined interval isolation response will be executed on that host.
    • If isolated response is set to “Leave Powered on” the vm’s will continues to run on the isolated, however if another host tries to power on the same vm on other host it may not be possible to do so, because of underlying lock mechanism provided by vmfs.
    • If isolated response is set to “Shutdown” the vm’s will undergo clean shutdown by the host and then they will be powered on other host.
    • If isolated response is set to “Power off” the vm’s will powered off immediately by the host and then they will be powered on other host. At the same time if the host has VM’s which are shutdown gracefully, they will never powered on new host. Only abnormally powered off vm’s will be powered on other hosts to avoid service interruptions.
  • When a host is declared as dead all the vm’s running on it will be powered on immediately on other hosts following admission control policies and restart priorities.
  • Admission control policies are defined to ensure sufficient resources are available to virtual machines when a host failure/isolation occurs, in other words HA will reserve some resources to provide room for virtual machines in worst case scenario. There are three policies available to make these reservations.
    • Host failures a cluster can tolerate
      • If we select this option, HA will reserve resources based on a concept called slots. “A slot is a logical representation of highest CPU and Memory reservations that satisfy requirements for any powered on vm in the cluster”. When slots are calculated a host with highest slots will be removed out of equation, this decision ensures resources for all the VM’s if any other host fails throughout the cluster.
      • A slot is defined from highest CPU and memory reservations in the cluster, if a VM has 4GHZ CPU and 1GB RAM and other VM has 2GHZ CPU and 4GBRAM, so 4GHZ and 4GB is defined as slot.
      • Number of slots available on a host will be calculated based on most restrictive number, for example if a host has 256GB of RAM and 16GHZ of CPU, it has 64 slots for memory and 4 slots for CPU, so the host will be considered to have 4 slots for failovers, which means it can accommodate 4 VM’s in an event of failure.
      • A custom slot size also can be defined using advanced options to increase number of slots and avoid resource wastage in case of a cluster has a VM with high amount  of resource reservations.
      • An example calculation of slots, Example 1
        • Host A -> 12GHZ + 12 GB
        • Host B ->  8 GHZ + 8 GB
        • Host C -> 12GHZ + 12 GB
        • VM 1 -> 2GHZ + 4 GB
        • VM 2 -> 1GHZ + 2 GB
        • VM 3 -> 4GHZ + 2 GB
        • VM 4 -> 4GHZ + 1 GB
        • VM 5 -> 2GHZ + 3 GB
        • VM 6 -> 3GHZ + 3 GB

So this cluster has 3 hosts with different set of resources, it has 32 GHZ CPU and 32 GB RAM throughout the cluster. As discussed earlier slot is calculated from the largest reservations of CPU and RAM assigned to the VM’s in this cluster. In our case the slot size would be 4GHZ and 4GB, it can satisfy the requirements for any powered on VM in this cluster.

So Host A has 3 slots, Host B has 2 slots and Host C also has 3 slots. A total of 8 slots available in this cluster.

If Host B fails this cluster will have only 5 slots available, this is called current capacity. Which does not satisfy to power on all VM’s so VM’s will be never powered or migrated to these hosts.

  • Another example 2
    • Host A -> 9GHZ + 9 GB
    • Host B -> 9GHZ + 6 GB
    • Host C -> 6GHZ + 6 GB
    • VM 1 -> 2GHZ + 1 GB
    • VM 2 -> 2GHZ + 1 GB
    • VM 3 -> 1GHZ + 2 GB
    • VM 4 -> 1GHZ + 1 GB
    • VM 5 -> 1GHZ + 1 GB

So this cluster has 3 hosts with different set of resources, it has 24GHZ CPU and 21GB RAM throughout the cluster. As discussed earlier slot is calculated from the largest reservation of CPU and RAM assigned to the VM’s in this cluster. In our case the slot size would be 2GHZ and 2GB, it can satisfy the requirements for any powered on VM in this cluster.

So HA has 4 slots, Host B has 3 slots and Host C also has 3 slots. A total of 10 slots available in this cluster.

If Host B fails this cluster will have 7 slots available, this is called current capacity. And we have 5 VM’s to power on so all can be powered on without any problem as they are not violating resource constrains.

  • Percentage of resources to be reserved
    • This is the most flexible mechanism to reserve resources for HA failovers.
    • This does not use slots concept to reserve resources, available resources will be calculated based on following formula
      • (Total available resources – Total reserved resources)/(Total available resources)
    • Using this option we can reserve resources at cluster level, not at host level.
    • There are no advanced options needs to be configured.
  • Dedicating a host for fail over
    • A dedicated host will be designated for failover purposes, this wastes resources as we need to dedicate a host and it is sitting idle all the time.
  • Restart priorities are followed in the below order
    • Agent based VM’s
    • FT enabled VM’s
    • VM’s with high priority option defined
    • VM’s with medium priority option defined
    • VM’s with low priority option defined
  • When a host is declared as dead all the VM’s running on it will be powered on on other hosts running in that cluster based on restart priority defined above.

HA in vSphere 4.1

  • It is called as Automated Availability Manager in this version.
  • When we configure HA on vSphere 4.1 cluster, the first 5 hosts will be designated as Primary nodes, out of these 5 one node will act as “Master Primary” and which will handle restarts of VM’s in the event of a host failure.
  • All the remaining hosts will join as Secondary Nodes.
  • Primary nodes maintain information about cluster settings and secondary node states.
  • All these nodes exchange their heartbeat with each other to know the health status of other nodes.
  • Primary nodes sends their heart beats to all other primary and secondary nodes
  • Secondary nodes sends their heart beats to primaries only
  • Heart beats will be exchanged between all nodes every second.
  • In case of a primary failure, other primary node will take the responsibility of restarts.
  • If all primaries goes down at same point, no restarts will be initiated, in other words to initiate reboots at least one primary is required.
  • Election of primary happens only during following scenarios
    • When a host is disconnected
    • When a host is entered into maintenance mode
    • When a host is not responding
    • And when cluster is reconfigured for HA.

 HA in vSphere 5.0

  • It is called as Fault Domain Manager in this version
  • HA is completely re-designed in this version, ha agent directly communicates with hostd instead of using a translator to communicate to vpxa. So every host has information of other host resources, which helps in an event of VC failure.
  • DNS dependency has been completely lifted off
  • When we configure HA on vSphere 5.0 cluster, the first node will be elected as master and all other nodes will be configured slaves.
  • Master node will be elected based on number of data stores it is connected to, and if all the hosts in cluster are connected to same number of data stores, host’s managed id will be taken into consideration. Host with highest managed id will be elected as master.
  • All hosts exchanges their heartbeats with each other to know about their health states.
  • Host Isolation response has been enhanced in this version, by introducing data store heart beating. Every host creates a hostname-hb file on the configured data stores and keeps it updated at specific interval. Two data stores will be selected for this purpose.
  • If we want to know who is master and who slaves are, just need to go to vCenter and click on Cluster Status from homepage in HA area.

VMware App Volumes Deployment Guide

Post Source Virtual langer

Installing and Configuring VMware App Volumes Manager

During VMworld 2014 it was announced that VMware was purchasing Cloud Volumes, an upstart for application virtualization technology. Fast forward to earlier this month and VMware officially released the product under the App Volumes name and made the software available to customers (as a standalone product or part of the Horizon View Enterprise bundle) as well as trial bits to give folks the chance to kick the tires.

App Volumes allows for the creation of containers or “AppStacks” for the delivery of a single application or multiple applications within a single stack. Shown in the picture below you can see with the use of the “CloudVolumes agent” the application stacks are decoupled from the operating system. Additionally, with the use of “Writable Volumes” user settings and changes can be captured and saved.

With all this goodness in hand, this first post in the series will cover the installation and configuration of the brains for App Volumes, the App Volumes Manager. In additional posts I will discuss the creation, presentation, and updating of AppStacks. Let’s get started!

The Prerequisites

  • Created Active Directory Service Account – Used for both LDAP Reads and vCenter Server permissions
  • Created “AppVolumes User” Role in vCenter (see eye chart at the end of the post for role permissions)
  • Windows Server 2012R2 virtual machine configured with 2vCPU’s and 4GB of RAM
  • MS SQL Database configured on remote MS SQL Server 2008R2 server
  • Trial software bits from VMware website -> HERE

The Install

Mount the VMware App Volumes ISO to the “VMware App Volumes Manager” server and execute the installer. Click “Next” on the Installation Wizard dialog:

Accept the the licensing agreement and click “Next”  to continue:

Choose the “Install App Volumes Manager” radial and click “Install” to continue:

The “App Volumes Manager Installation Wizard” dialog will be displayed. Click “Next” to continue:

Choose if you wish to have setup program install/configure a copy of SQL Server Express or utilize an existing SQL Server Database. For this example I am using an existing remote SQL Server Database:

Fill in the needed Database Server connectivity details and click “Next” when ready:

I had no need to change the default HTTP and HTTPS ports. Moving on, click “Next”:

I left the default Destination Location unchanged. Click “Next” to continue:

At this point we are ready for the actual installation. Click ‘”Install “ to proceed:

During my install this screen was accurate, it took about 8 minutes to complete the installation:

Once the installation completes click “Finish” to exit the installer:

The Configuration

With the completion of the installation we have to run through a quick “setup” process to configure VMware App Volumes Manager.

Launch the App Volumes Manager desktop icon:

A “Welcome to App Volumes Manager” webpage will be displayed. It provides an overview of the tasks that can be completed and initiated App Volumes Manager. Click “Get Started”:

The first tab that is displayed is the “License Information”. As I am using the trial version of the software we don’t have much to do here. One thing to point out, when using the trial version of the software only a single Appstack can be attached per user. Click “Next”  to continue:

Second tab displayed is for the configuration Active Directory. Provide the details for your domain, domain controllers (if you want to specify) and user with “Read” access to your directory. I provided a service account that I created for both Active Directory and vCenter permissions. Click “Next” to continue:

Configure the Active Directory group that will be added to the App Volumes Admin Group. For lab use I added my Domain Admins group. Click “Next”:

Next up is the Hypervisor Credentials tab. I am using the “vCenter Server”, as I am assuming most folks will, and the service account I created assigned to the “App Volumes User” vCenter Role I created. I also provided my ESXi hosts root account and password to enable the “Mount on host” setting. Click “Next”:

In the Storage tab we are configuring the location of where the Appstacks and Writeable volumes that are created will be stored. In my lab environment I configured both to use the VSAN Datastore:

Confirm the Storage Settings to proceed. To choose whether to “Import volumes in the background” or “Import volumes immediately”

Provide the vSphere Datastore to upload the volume templates and provide login in credentials for the ESXi host used for the data transfers. Click “Upload” to continue:

Finally, verify the selections/settings made in the previous tabs provide in the Summary. Click “Next” to proceed:

With all the steps completed and configured you are taken to the App Volumes Manager Dashboard:

From here we are good to install our AppVolumes agent and create our first AppStack. Stay tuned!

-Jason

vCenter Server Permissions

  • Datastore
    • Allocate space
    • Browse datastore
    • Low level file operations
    • Remove file
    • Update virtual machine files
  • Folder
    • Create folder
    • Delete folder
  • Global
    • Cancel task
  • Host
    • Local operations
      • Create virtual machine
      • Delete virtual machine
      • Reconfigure virtual machine
  • Resource
    • Assign virtual machine to resource pool
  • Sessions
    • View and stop sessions
  • Tasks
    • Create task
  • Virtual Machine
    • Configuration
      • Add existing disk
      • Add new desk
      • Add or remove device
      • Change resource
      • Remove disk
      • Settings
    • Interaction
      • Power off
      • Power on
      • Suspend
    • Inventory
      • Create from existing
      • Create new
      • Move
      • Register
      • Remove
      • Unregister
    • Provisioning
      • Clone template
      • Clone virtual machine
      • Create template from virtual machine
      • Customize
      • Deploy template
      • Mark as template
      • Mark as virtual machine
      • Modify customization specification
      • Promote disks
      • Read customization specifications

VMware Tools one-liners using PowerCli

VMware Tools one-liners using PowerCli

Post source Pario

This short post is about VMware Tools on VM guests running in a vSphere 5.x cluster/hosts.

“Connect by using integrated authentication. In this case, the credentials you are logged on to your machine must be the same as those for the server”

Connect-VIServer Server

This PowerCli one-liner creates a list of VM guests where the VMware Tools CDROM/ISO is mounted:

(Get-VM | Get-View | Where {$_.Runtime.ToolsInstallerMounted}) | % {$_.Name}

Unmount the VMware Tools installer CDROM on all VM guests. This is useful to run before you try to put at ESXi host in maintenance mode because VM guests thatare installing VMware Tools will not migrate because the VMware Tools is not on a shared storage. You will get the error message “The virtual machine is installing VMware Tools and cannot initiate a migration operation”.

(Get-VM | Get-View | Where {$_.Runtime.ToolsInstallerMounted}) | % {Dismount-Tools }

This last one finds all VM guests running Windows as guest OS and upgrades VMware Tools without a reboot at the end

Get-VM | Where {$_.PowerState -eq “PoweredOn” -and $_.Guest.OSFullName -match “Win*”} | % {Update-Tools -VM $_ -NoReboot}

VM Tools and Virtual Hardware Versions

Get-VM | Select Name, Version

But the returned object doesn’t have a root property for ToolsVersion or ToolsVersionStatus, for this we need to delve into the ExtensionData property and have a look around, once we have found the information it is fairly easy to add these to our object using the New-VIProperty cmdlet as below:

New-VIProperty -Name ToolsVersion -ObjectType VirtualMachine `
-ValueFromExtensionProperty ‘Config.tools.ToolsVersion’ `
-Force

New-VIProperty -Name ToolsVersionStatus -ObjectType VirtualMachine `
-ValueFromExtensionProperty ‘Guest.ToolsVersionStatus’ `
-Force

Now we have added these as a new property to our object (actually they are PowerShell Code Properties), we can use our old friend Get-VM to retrieve the information easily:

Get-VM | Select Name, Version, ToolsVersion, ToolsVersionStatus

Of course we can choose which list of VMs to get this information for:

For a Datacenter: Get-Datacenter London | Get-VM | Select Name, Version, ToolsVersion, ToolsVersionStatus

For a cluster: Get-Cluster Production | Get-VM | Select Name, Version, ToolsVersion, ToolsVersionStatus

For a host: Get-VMHost Host1.mydomain.local | Get-VM | Select Name, Version, ToolsVersion, ToolsVersionStatus

And we can also easily export this information into a csv file:

Get-VM | Select Name, Version, ToolsVersion, ToolsVersionStatus | Export-Csv -NoTypeInformation -UseCulture -Path C:\Temp\VMHWandToolsInfo.csv

Update sequence for vSphere 5.5 and its compatible VMware products

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2057795

 

Update sequence for vSphere 5.5 and its compatible VMware products (2057795)

Purpose

VMware has made available certain releases to address critical issues for several products including:

  • vCloud Director (VCD)
  • vCloud Networking and Security (VCNS) (formerly vShield Manager)
  • Horizon View
  • vCenter Server
  • vSphere Replication (VR)
  • vCenter Site Recovery Manager (SRM)
  • vCenter Operations Manager (vCOPS)
  • vSphere Data Protection (VDP)
  • vSphere Storage Appliance (VSA)
  • ESXi
  • vShield Edge
  • vShield App
  • vShield Endpoint

This article only encompasses environments running vSphere 5.5 and VMware products compatible with vSphere 5.5.

In an environment with vSphere 5.5 and its compatible VMware products, follow the update sequence described in the Supported Update Sequence table.

Note: This article refers only to the latest versions of VMware products that are supported by vSphere 5.5 and its update sequence. For a complete list of VMware products compatible with vSphere 5.5, see VMware Product Interoperability Matrixes.

Resolution

This table describes the sequence in which the vSphere 5.5 and its compatible VMware products must be updated:

Supported Update Sequence

  VCD VCNS View Composer View Connection Server VSA Manager vCenter Server VR / SRM vCOPS VDP ESXi VMware Tools vShield Edge vShield App vShield Endpoint View Agent / Client
Update Sequence Number 1                            
  2                          
    3                        
      4 4**                    
          5*                  
            6 6 6            
                  7*          
                    8        
                      9 9 9 9

* If you are using a Cisco Nexus 1000V, see vSphere 5.5 and its compatible third-party products in the Additional Information section of this article before upgrading vCenter Server (sequence step 5) or the ESXi hosts (sequence step 7).

Note: If you need to update multiple products in your environment, start with updating the product with the lowest sequence number. After you update the product, update the product with the next sequence number. If a product is not present in your environment, update the subsequent product. If you need to update two products with the same sequence number, the order of update does not matter.

Before you update vCenter Server, disable vCenter Server from vCloud Director. Also ensure that you stop or disable other VMware services so that they do not communicate with vCenter Server during the update process. For more information, see the product documentation.

Sample VMware product upgrade scenarios

Example 1

If you have the vCloud Director solution stack installed in your environment, the supported patch update sequence is:

  1. Update vCloud Director (sequence step 1)
  2. Update vCloud Networking and Security (vShield Manager) (sequence step 2)
  3. Update vCenter Server (sequence step 5)
  4. Update ESXi (sequence step 7)
  5. Update vShield Edge (sequence step 9)

Example 2

If you have SRM solution installed in your environment, the supported patch update sequence is:

  1. Update vCenter Server (sequence step 5)
  2. Update vSphere Replication (sequence step 6)
  3. Update ESXi (sequence step 7)

Example 3

If you have the vSphere Data Protection solution installed in your environment, the supported patch update sequence is:

  1. Update vCenter Server (sequence step 5)
  2. Update vSphere Data Protection (sequence step 6)
  3. Update ESXi (sequence step 7)

Example 4

If you have the vSphere Storage Appliance solution installed in your environment, the supported patch update sequence is:

  1. Update vSphere Storage Appliance Manager (sequence step 4)
    1. For a 2 node VSA cluster, see Upgrade a Two-Node VSA Environment in the vSphere Storage Appliance Installation and Administration guide
    2. For a 3 node VSA cluster, see Upgrade a Three-Node VSA Environment in the vSphere Storage Appliance Installation and Administration guide
  2. Update vCenter Server (sequence step 5)
  3. Update vSphere Storage Appliance Cluster (sequence step 6)

Example 5

If you have a Cisco Nexus 1000V virtual distributed switch solution installed in your environment, the supported patch update sequence is:

  1. Verify with Cisco the compatibility for the Nexus upgrade procedure. This will dictate the order for upgrading the vCenter Server and the ESXi hosts.
  2. Update vCenter Server (sequence step 5)
  3. Update ESXi (sequence step 7)

Note: If you are using the vSphere Client in your environment, VMware recommends that you upgrade the vSphere Client to version 5.5.

vSphere 5.5 and its compatible VMware products

This table provides information about the current version of released VMware products and the recommended action required to patch to the next level. The table also provides reference links to release notes and update procedure documents.

vSphere 5.5 and its compatible VMware products

Product Version Recommended Action Important Links
vCloud Director (VCD) 5.5.0 Update to 5.5.0 Release Notes
Update Procedure
vCloud Networking and Security (VCNS) 5.5.0 Update to 5.5.0 Release Notes
Update Procedure
Horizon View (View) 5.2.0 Update to 5.2.0 Release Notes
Update Procedure
vCenter Server 5.5.0 Update to 5.5.0 Release Notes
Update Procedure
vSphere Replication (VR) / vCenter Site Recovery Manager (SRM) 5.5.0 Update to 5.5.0 VR Release Notes
SRM Release Notes
Upgrading VR
Upgrading VR without internet access
Upgrading SRM
vCenter Operations Manager (vCOPS) 5.7.2 Update to 5.7.2 Release Notes
Update Procedure
vSphere Data Protection (VDP) 5.5.1 Update to 5.5.1 Release Notes
Update Procedure
vSphere Storage Appliance (VSA) 5.5 Update to 5.5 Release Notes
Update Procedure
ESXi 5.5.0 Update to 5.5.0 Release Notes
Update Procedure
vShield Edge 5.5.0 Update to 5.5.0 Release Notes
Update Procedure
vShield App 5.5.0 Update to 5.5.0 Release Notes
Update Procedure
vShield Endpoint 5.5.0 Update to 5.5.0 Release Notes
Update Procedure

Troubleshooting VMware snapshots

Post source techtarget

Virtualization administrators can use snapshots in vSphere to travel back in time and figure out what went wrong with their virtual machines (VMs). In part one of this series, I discussed how to use VMware snapshots. In part two, I explained how to delete snapshots without wasting disk space. But what do you do when your snapshots start acting funny? In this tip, we’ll troubleshoot potential problems that may come up when using snapshots in vSphere.

Locating VMs that have snapshots
Finding out which VMs have snapshots can be challenging. In VMware Infrastructure 3, there wasn’t a centralized, built-in way to accomplish this task in the vSphere Client or vCenter Server. You had to use methods, such as scripts and command-line utilities, that made locating snapshots difficult. But there were some enhancements in vSphere that made locating snapshots much easier. Here are a few of the methods that you can use.

Method 1: Find command
Use the find command in the ESX service console or ESXi Tech Support Mode

  1. Log in to the console.
  2. Change to your /vmfs/volumes/ directory.
  3. Type find -iname “*-delta.vmdk” -mtime +7 -ls to find snapshot files that have not been modified in seven days or simply find -iname “*-delta.vmdk” to find all snapshot files.

Method 2: Use the Storage View in vCenter Server
The Storage View, part of a new Storage Monitoring and Reporting plug-in that comes with vCenter Server, shows information related to storage in vSphere. When you select an object in the left pane of the vSphere Client, you can select the Storage View tab in the right pane and view storage information related to that object. One of the columns that you can view is Snapshot Space — which is the total size of all snapshot-related files, including the -delta.vmdk, .vmsd and .vmsn files.

By selecting an object, such as Cluster or Datacenter, and sorting the Snapshot Space field, you can view the size of any VM snapshot that exists under that object. VMs that haven’t had a snapshot will show 0 bytes. Once a snapshot of a VM is taken, it will still show a very small size (around 40 bytes), which is from the residual text left in the .vmsd file.

Method 3: Use alarms in vCenter Server
You can configure a vCenter Server alarm to trigger when a VM snapshot size reaches a predetermined gigabyte threshold. You can also set alarms at any virtualization level — from a single VM to the top vCenter Server level. These alarms will keep you informed of snapshot growth, so you can take action, if needed.

Method 4: Use a PowerShell script
The Get-Snapshot command, part of vSphere PowerCLI, can query VM snapshot information. You can use it in scripts to produce reports on VMs that have active snapshots. There are several, free PowerShell scripts that you can download and run periodically, such asSnapReminderyadr — A vdisk reporter and Snapshot List. You can also set the scripts to run automatically.

Dealing with snapshots that don’t delete properly
Occasionally, a snapshot will not delete properly, leaving an active snapshot for a VM. This situation can happen when using backup applications or deleting snapshots through Snapshot Manager. In most cases, the snapshot will not appear in the Snapshot Manager. The only indication that a snapshot may still exist is the presence of delta files in the VM’s directory.

If you have a snapshot running that is not in Snapshot Manager, you can attempt to delete it in one of two ways. First, create a new snapshot using the vSphere Client and delete all snapshots from the snapshot manager after the new one has been created. Alternatively, use the ESX service console or vSphere CLI. Switch to the VM’s home directory and create a new snapshot by typing vmware-cmd createsnapshot . Wait for the snapshot to be created and type vmware-cmd removesnapshots. When it completes, see if the delta files have been deleted. If they have, then it was successfully completed.

If the delta files weren’t deleted, check the VMX file for the VM and locate the lines starting with scsi. If the VM is configured with only one virtual disk, it is usually scsi0:0. (If .present is false, it is a non-existent drive that you can ignore.) The .fileName should be using the original disk file that was created with the VM and it’s usually the same name as your VM. If this is the case, then your VM is not using the snapshot files. If it has a -00000# in the filename, it is currently using a snapshot file.

To be clear, a VM with no snapshots displays the following: scsi0:0.present = “true” scsi0:0.fileName = “myvmname.vmdk”. And a VM with snapshots will display the following: scsi0:0.present = “true” scsi0:0.fileName = “myvmname-000001.vmdk”

If the above operation failed, your other options are to either clone the VM or clone the VM’s disk file. To clone the VM, you can either use the clone function in vCenter Server or the standalone vCenter Converter application. When it’s completed, shut down and delete the old VM.

Another method is to shut down the VM. Log in to the ESX Service Console or ESXi Tech Support Mode. Then, switch to the VM’s directory and clone the VM’s disk file, using vmkfstools and specifying the snapshot file as the source disk (i.e. “vmkfstools –i myvmname-000001.vmdk myvmnamenew.vmdk”).

Next, go into the settings for the VM. Remove (don’t delete) the hard disk. Then, add a new hard disk and browse to the newly created disk file. Power on the VM and verify everything is working before you delete the old disk and delta files.

Changing snapshot file locations
By default, the snapshots are written to the home directory of each virtual machine. You may want to change this location, as to not take up space on the volume that your VM resides. It is possible to individually specify a new working directory for snapshots on each VM. Both snapshots and .vswp files are written to this directory when you choose this method.

Be warned: If the VM is on shared storage and you specify local storage as a location, you will not be able to use features that move VMs between hosts, such as vMotion, High Availability and Distributed Resource Scheduler. To do this, follow these steps:

  1. Power off your VM and log in to the ESX service console or ESXi Tech Support Mode.
  2. Edit the VMX file of your VM with the nano (ESX only) or vi (ESX/ESXi) editor.
  3. Add a new line, using the following syntax: workingDir=”/vmfs/volumes/SnapVolume/Snapshots/”
  4. If you want your .vswp file to stay in the VM’s directory, add the following line to the VMX file: sched.swap.dir = “/vmfs/volumes/VM-Volume1/MyVM/”. This step is optional. Furthermore, do not worry about updating the existing “sched.swap.derivedName” parameter, because it is generated by the VM and written to the configuration file each time the VM powers on.
  5. Power on your VM, and your .vswp, .vmsn and snapshot (delta-vmdk) files will now be located in this directory.

Using vMotion and Storage vMotion with snapshots
Using vMotion to migrate a VM to a different host is supported and all existed snapshots are retained. If you try to vMotion a VM with running snapshots from one host to another, however, you will receive the following warning: “Reverting to snapshot would generate error (warnings) on the destination host.” In other words, the migration wizard cannot verify the compatibility of the virtual machine state in the snapshot with the destination host.

Because the compatibility cannot be verified, a failure could occur if the VM configuration in the snapshot uses devices or virtual disks that are not accessible on the destination host. A failure can also occur if the snapshot contains an active VM state that was running on virtual hardware and it’s incompatible with the destination host CPU.

Using Storage vMotion to move a VM to another disk location is not supported, initially. To use it, you must first delete all the snapshots on a VM. Alternately, you can power the VM off and perform a cold migration to another disk location.

Using Fault Tolerance with snapshots
VM snapshots are not supported on VMs that use Fault Tolerance (FT). As a result, backing up FT-enabled VMs can be tricky, because many backup applications rely on VM snapshots.

Look at alternative backup methods, such as traditional OS backup agents that run inside the VM, cloning VMs and then backing up the clones, temporarily disabling FT when running backups on the VM or using storage snapshots to backup the VM’s data store.

Fault Tolerance can be controlled via PowerShell scripts, so you can run pre-backup scripts to temporarily disable FT. That way, a backup application can take a snapshot. Then, a post-backup script can re-enable FT.

Understanding the files that make up a VMware virtual machine

Post source techtarget

VMware admins should know the components of virtual machines. Understanding the files that make up a virtual machine can help admins decide what files are unnecessary and clean them out, and ease other management tasks.

Once you understand virtual machines (VMs) from a hardware perspective, you can study the components that make up a VM on an ESX/ESXi host. These are the various VMware file types associated with a VM, located in the VM’s directory on the host (represented in the illustration below).

 

VMware virtual machine files are organized in the Virtual Machine File System (VMFS). If you look at the list of files associated with the VM — use an SCP-based tool or follow VMware’s recommendations — you’ll notice that most of the files start with the actual name of the VM, followed by different file extensions that denote the file type. You may not see all of the possible file types in the VMFS until your VM is in a certain state. For example, the .vswp file is only present when the VM is powered on and the .vmss file is only present when a VM is suspended. Below is a typical VM directory listing (using WinSCP).

So what are all these VMware file types and what are they used for? Here’s each file type in detail.

The .nvram file. This small file contains the BIOS that is used when the VM boots. It is similar to a physical server that has a BIOS chip that lets you set hardware configuration options. A VM also has a virtual BIOS that is contained in the NVRAM file. The BIOS can be accessed when a VM first starts up by pressing the F2 key. Whatever changes are made to the hardware configuration of the VM are then saved in the NVRAM file. This file is in binary format and if deleted it will be automatically re-created when a VM is powered on.

The .vmx file. This file contains all of the configuration information and hardware settings of the virtual machine. Whenever you edit the settings of a virtual machine, all of that information is stored in text format in this file. This file can contain a wide variety of information about the VM, including its specific hardware configuration (i.e., RAM size, network interface card info, hard drive info and serial/parallel port info), advanced power and resource settings, VMware tools options, and power management options. While you can edit this file directly to make changes to a VM’s configuration, don’t do this unless you know what you are doing. If you do make changes directly to this file, make a backup copy first.

VMDK files. All virtual disks are made up of two files, a large data file equal to the size of the virtual disk and a small text disk descriptor file, which describes the size and geometry of the virtual disk file. The descriptor file also contains a pointer to the large data file as well as information on the virtual disks drive sectors, heads, cylinders and disk adapter type. In most cases these files will have the same name as the data file that it is associated with (i.e., myvm_1.vmdk and myvm_1-flat.vmdk). You can match the descriptor file to the data file by checking the Extent Description field in this file to see which -flat, -rdm or -delta file is linked to it. 

The different types of virtual disk data files that can be used with VMware virtual machines are:

  • The -flat.vmdk file
    This is the default large virtual disk data file that is created when you add a virtual hard drive to your VM that is not an RDM. When using thick disks, this file will be approximately the same size as what you specify when you create your virtual hard drive. One of these files is created for each virtual hard drive that a VM has configured, as shown in the examples below.

     

  • The -delta.vmdk file
    These virtual disk data files are only used when making snapshots. When a snapshot is created, all writes to the original -flat.vmdk are halted and it becomes read-only; changes to the virtual disk are then written to these -delta files instead. The initial size of these files is 16 MB and they are grown as needed in 16 MB increments as changes are made to the VM’s virtual hard disk. Because these files are a bitmap of the changes made to a virtual disk, a single -delta.vmdk file cannot exceed the size of the original -flat.vmdk file. A delta file will be created for each snapshot that you create for a VM and their file names will be incremented numerically (i.e., myvm-000001-delta.vmdk, myvm-000002-delta.vmdk). When the snapshot is deleted, these files are automatically deleted after they are merged back into the original flat.vmdk file.

     

  • The -rdm.vmdk file
    This is the mapping file for the raw device mapping (RDM) format that manages mapping data for the RDM device. The mapping file is presented to the ESX host as an ordinary disk file, available for the usual file system operations. However, to the VM, the storage virtualization layer presents the mapped device as a virtual SCSI device. The metadata in the mapping file includes the location of the mapped device (name resolution) and the locking state of the mapped device. If you do a directory listing, you will see that these files will appear to take up the same amount of disk space on the VMFS volume as the actual size of the LUN that it is mapped to, but in reality they just appear that way and their size is very small. One of these files is created for each RDM that is created on a VM.

The .vswp file. When you power on a VM, a memory swap file is created that can be used in lieu of physical host memory if an ESX host exhausts all of its physical memory because it isovercommitted. These files are created equal in size to the amount of memory assigned to a VM, minus any memory reservations (default is 0) that a VM may have set on it (i.e., a 4 GB VM with a 1 GB reservation will have a 3 GB VSWP file created). These files are always created for virtual machines but only used if a host exhausts all of its physical memory. As virtual machine memory that is read/written to disk is not as fast as physical host RAM, your VMs will have degraded performance if they do start using this file. These files can take up quite a large amount of disk space on your VMFS volumes, so ensure that you have adequate space available for them, as a VM will not power on if there is not enough room to create this file. These files are deleted when a VM is powered off or suspended.

Virtual machines will lock the .vswp, -flat.vmdk and -delta.vmdk, .vmx and .log files during runtime.

The .vmss file. This file is used when virtual machines are suspended and is used to preserve the memory contents of the VM so it can start up again where it left off. This file will be approximately the same size as the amount of RAM that is assigned to a VM (even empty memory contents are written). When a VM is brought out of a suspend state, the contents of this file are written back into the physical memory of a host server, however the file is not automatically deleted until a VM is powered off (an OS reboot won’t work). If a previous suspend file exists when a VM is suspended again, this file is re-used instead of deleted and re-created. If this file is deleted while the VM is suspended, then the VM will start normally and not from a suspended state.

The .vmsd file. This file is used with snapshots to store metadata and other information about each snapshot that is active on a VM. This text file is initially 0 bytes in size until a snapshot is created. A VMSD file updates with information every time snapshots are created or deleted. Only one of these files exists regardless of the number of snapshots running, as they all update this single file. The snapshot information in a VMSD file consists of the name of the VMDK and VMSN file used by each snapshot, the display name and description, and the UID of the snapshot. Once your snapshots are all deleted, this file retains old snapshot information but increments the snapshot UID to be used with new snapshots. It also renames the first snapshot to “Consolidate Helper,” presumably to be used with consolidated backups.

The .vmsn file. This file is used with snapshots to store the state of a virtual machine when a snapshot is taken. A separate .vmsn file is created for every snapshot that is created on a VM and is automatically deleted when the snapshot is deleted. The size of this file will vary based on whether or not you choose to include the VM’s memory state with your snapshot. If you do choose to store the memory state, this file will be slightly larger than the amount of RAM that has been assigned to the VM, as the entire memory contents, including empty memory, is copied to this file. If you do not choose to store the memory state of the snapshot then this file will be fairly small (under 32 KB). This file is similar in nature to the .vmss that is used when VMs are suspended.

The .log file. LOG files are created to log information about the virtual machine and are often used for troubleshooting purposes. There will be a number of these files present in a VM’s directory. The current log file is always named vmware.log and up to six older log files will also be retained with a number at the end of their names (i.e., vmware-2.log). A new log file is created either when a VM is powered off and back on or if the log file reaches the maximum defined size limit. The amount of log files that are retained and the maximum size limits are both defined as VM advanced configuration parameters (log.rotateSize and log.keepOld).

The .vmxf file. This file is a supplemental configuration file that is not used with ESX but is retained for compatibility purposes with VMware Workstation. It is in text format and is used by Workstation for VM teaming where multiple VMs can be assigned to a team so they can be powered on or off, or suspended and resumed as a single object.

[Editor’s note: Article update in April 2013]

The .ctk file. VMware CTK files list any changes made to the VM between backups. This file describes the VMDK block and grows in proportion with the number of VMDK blocks. There is one CTK file per VMDK. Change tracking files originated with VMware’s Changed Block Tracking (CBT) technology for incremental backups. The CTK file stores information about what VM information blocks changed, avoiding unnecessary block backups. VMware snapshots also use .ctk files. Like .log and .nvram files, .ctk files are small. 

Other less-frequently seen file types include the .vmem virtual machine paging file and the .vmtm configuration file for team data. Like VMSN files, VMEM files back up a virtual machine’s memory. They exist when the VM is running or in the event of a VM crash. VMTM files support VM teams, a feature in VMware Workstation that allows a group of VMs to work together via a private LAN segment.

That covers all the files that are associated with a VMware VM, and you should have a better understanding of VM anatomy. Check out the VMs on your own VMware hosts to see the various files that make up these virtual machines. You might find a few surprises from old data that has not been properly cleaned up on VMFS volumes. Just be careful before you start deleting any files and make sure that the files you delete are no longer needed and not being used.

 

In Short and Brief

.VMDK — These files are the actual hard disk of the virtual machine itself, and tend to be the largest file within the folder. You can consider the size of this file to be roughly equivalent to the size of either the disk itself (if you’ve chosen to use preallocated disks) or the size of the data currently stored on that disk (if you use growable disks).

.NVRAM — Consider this file the BIOS of the virtual machine.

.VMX — With typically one VMX file per folder, this file holds the configuration information for the virtual machine in a text format. 

Unlike almost all the other files you’ll see, these files can be edited using any text editing program, a process that is actually required for some functionality that is not exposed in the GUI.

.VMXF — This file, in XML format, includes additional information about the virtual machine if it has been added to a team. If a machine has been added to a team and then later removed, this file remains resident. This file can also be opened and read in a text editor.

.VMTM — For virtual machines actively participating in a team, this file stores information about that team membership.

.VMEM — These files, which contain a backup of the VMs paging file, are typically very small or non-existent when the virtual machine is powered off, but grow immediately to the size of configured RAM when the machine is powered on.

.VMSN and .VMSD — When snapshots are created for a virtual machine, these files are created to host the state of the virtual machine. 

The VMSN file stores the running state of the machine, what you could consider the “delta” between the VMDK at the point of the snapshot and what has been processed up until the present time. The VMSD stores information and metadata about the snapshot itself.

.VMSS — If you’ve suspended the state of your machine, this file contains the suspended state of that machine. These files typically only appear when virtual machines have been suspended. 

.HLOG — If you have vMotioned the Virtual Machine, this file is created and can be safely deleted.