vSphere

Understanding VMware Reservations Limits and Shares

September 19, 2013 by FrankBrix 19 Comments

Introduction

A key concept that is very important to understand for the VMware administrator is how Shares, Limits and Reservations work. Why do we have these controls? Well to put it simply if we are over provisioning our ESXi host on memory and CPU we need a tool to make sure that the right machines get the correct amount of resources. An example could be: Make sure the important ERP system always gets 3000MHZ (Reservaton) and make sure that the Test system never gets more than 1000MHZ (Limit). This article will focus on the three concepts on Memory and CPU. You can also apply some of this on storage and network. But that is out of scope for this article.

Rerservations

A reservation is a guarantee on either memory or cpu for a virtual machine. You define the reservation in MB or MHZ. It works kind of different on the two compute resources.

On memory it is a guarantee for access to physical memory for the virtual machine. Remember, every virtual machine has a swap file. The swap file size is defined as (swap file = configured memory – memory reservation). When the virtual machine is running the vmkernel allocates memory to it when the VM requests it. The vmkernel will always try to map the memory to physical memory, but if the ESXi host is running low on memory the memory has to come from the swap file. This has a huge performance penalty. Lets take two examples:

Example 1: You have a virtual machine configured with 2 GB memory and you configure a 1 GB reservation
- When the virtual machine powers on a 1 GB swap file (.vswp) is created on a datastore
- The 1 GB reservation guarantees that the VM will always a least get 1 GB of physical memory. If the ESXi host is running low the remaining 1 GB can come from the swap file on disk!
Example 2: You have a virtual machine configured with 4 GB memory and you configure a 4 GB reservation
- When the virtual machine powers on a swap file with zero size is created
- The 4 GB reservation guarantees that the VM will get ALL its memory from physical memory and it will never do hypervisor swapping or ballooning.

On cpu the reservation is a guarantee for clock cycles. You define the reservation in MHZ. If you give a virtual machine a reservation it means the vmkernel CPU scheduler will give it at least that amount of resources. If a virtual machine is not using its resources the CPU cycles are not wasted on the physical host. Other machines can use it. What you do with CPU reservations is making sure that a VM will always get access to physical cpu in a committed environment.

Limits

A limit is a a limit 🙂 Defined on either Memory or CPU It is defined in MB or MHZ.

On memory the limit defines what is the maximum amount of physical memory the virtual machine can use. This is a very dangerous setting! If you set the limit lower than the configured memory for a VM it will cause swapping and balloon activity for the virtual machine.

Example 1: You have a virtual machine configured with 4 GB memory and you configure a 1 GB limit
- What you have done is BAD, unless you want bad performance. The virtual machine guest operating system sees 4GB of memory (inside windows task manager for instance) but the ESXi is not allowed to give it more than 1 GB of physical memory. The virtual machine will probably request more that 1 GB for its application and when this happens ballooning and hypervisor swapping will start.

Limits on memory can have huge consequences. As a consultant I always look for virtual machines with a limit on memory, more often than you would imagine you find it configured on a few virtual machines!

On CPU the limit defines how much access a virtual cpu can get on a physical cpu (core). This is used to make sure a virtual machine is not using too much resources on a host. By limiting the vCPU you essentially also limits the performance of the virtual machine. Even though capacity is available on the ESXi host the limit will still be enforced.

Shares is a different approach to performance tuning in a virtual environment. Shares define how much access you get to a resource compared to something else. Every virtual machine has 1000 shares configured pr. vCPU as a default. So you are already using them! All virtual machine are equal from a hypervisor perspective unless you change the shares and tell it which machines are really more important. What is important to know about shares is that they only is considered in case of contention! If you have available capacity for all machines it does not help performance to increase the shares on some machines.

Example 1: VM A has 1000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core. In this case the vmkernel CPU scheduler will give each machine 1/2 or 50% access and they will have the same performance
Example 2: VM A has 3000 shares and VM B has 1000 shares and they are both competing for the same physical cpu core the result would be that VM A gets 3/4 or 75% access and VM B gets 1/4 or 25% access.
Example 3: VM A has 3000 shares and VM B has 1000 shares and they are not competing for the same physical cpu core. In this case both machines will get 100% access to physical CPU. Remember shares is only handled when we have contention!

Summary

Reservations, limits and shares are powerful resource controls in a virtual environment. Just make sure to think about how you implement it. Especially watch out for putting a memory limit on your virtual machine.

If you have any questions, please use the comments on this page.

Understanding VMware Ballooning

September 18, 2013 by FrankBrix 24 Comments

VMware ballooning is one of the hard concept to grasp. There are a lot of misunderstanding out there about this feature. I have been discussing this feature with customers and students during the last 5 years. This is my attempt to explain balloning.

VMware ballooning is a memory reclamation technique used when and ESXi host is running low on memory. You should not see balloning if your hosts is performing like it should. To understand ballooning we would have to take a look at the following picture:

This picture shows the three levels of memory in a virtual environment. In a physical world we would only have the two top levels (virtual memory & guest physical memory) but in the virtual world we also have the host physical memory. What is important to know is that the hypervisor (ESXi) has no knowledge of what is happening inside the virtual machine (grey area). The hypervisor maps memory when the virtual machines asks for it. The hypervisor will then give it memory from “host physical memory” but only if memory is available. If memory is not available the memory can med mapped to the .vswp file on a vmfs or nfs datastore. The virtual machine has no knowledge if the memory is mapped to physical memory or to a disk. This is called hypervisor swapping, and this is the last resort for the vmkernel to use this mechanism.

Ballooning in short is a process where the hypervisor reclaims memory back from the virtual machine. Ballooning is an activity that happens when the ESXi host is running out of physical memory. The demand of the virtual machine is too high for the host to handle.

Lets take a high level example:

Inside a virtual machine you start an application. For instance solitaire
solitaire as an application will ask the guest operating system (in this case windows) for memory. Windows will give it memory and map it from the virtual memory -> guest physical memory
what happens next is that the hypervisor sees the request for memory and the hypervisor maps guest physical memory -> host physical memory
Now everything is perfect. You play solataire for a few hours. And then you close it down.
When you close solitaire the guest operating system will mark the memory as “free” and make it available for other applications. BUT since the hypervisor does not have access to Windows’ “free memory” list the memory will still be mapped in “host physical memory” and putting memory load on the ESXi host.
This is where ballooning comes into place. In case of an ESXi host running low on memory the hypervisor will ask the “balloon” driver installed inside the virtual machine (with VMware Tools) to “inflate”
The balloon driver will inflate and because it is “inside” the operating system it will start by getting memory from the “free list”. The hypervisor will detect what memory the balloon driver has reclaimed and will free it up on the “host physical memory” layer!

The balloon driver can inflate up to a maximum of 65%. For instance a VM with 1000MB memory the balloon can inflate to 650MB. The way to avoid ballooning is not to uninstall the balloon driver but to create a “Memory Reservation” for the virtual machine. In case of full inflation for this particular VM the result is the hypervisor gets 650MB memory reclaimed. The downfall of this is that you risk your VM to do Guest OS Swapping to its page file! Just remember page file swapping is better than hypervisor swapping. Hypervisor swapping happens without the guest operating system is aware of it. Page file swapping it is the OS that decides what pages to swap to disk!

To check for ballooning you can either open ESXTOP or the vCenter Performance Graphs.

High Availability in a vCloud Director Environment

May 22, 2013 by FrankBrix 2 Comments

High Availability in a vCloud Director Environment

Making sure your cloud is accessible, at all time, to end users and administrators is an important task. When looking into a cloud based on VMware vCloud Director there are several things to take into consideration. A VMware vCloud can consist of a lot of components. The mandatory components are the following

VMware vCenter Server (Typically Windows 2008/2012 or Linux Appliance)
VMware ESXi hosts
VMware vShield Manager (Linux Appliance)
VMware Director cell (Red Hat Enterprise Linux)
Database Server for vCenter and vCloud Director

Optional Components

vCenter Chargeback
vCenter Operations Manager
vCloud Automation Center
VMware Orchestrator
…

You have to think about how you want to protect the mandatory and optional components. I will focus on the mandatory now.

VMware vCenter Server

There are different ways to protect the vCenter Server. But before you think about protection you need to think about how you are going to deploy it. Your first option is am I going with the Windows version or going with the Linux appliance. In most cases people go for the Windows version, and that is definitely the safe bet. One of the things the Linux appliance has against it is only support for external Oracle databases. No MS SQL support.

If you choose the Windows version you have to consider if you are a splitting the vCenter roles up on multiple virtual machines. The roles you are considering are the following: SSO, Inventory Service, vCenter Service and Update Manager. For best performance and scalability you would split it up. But that also means you have to consider how to protect each server.

SSO has its own “clustering” functionality. But Inventory Service and vCenter does not. In most situations the software is installed inside a virtual machine running on a ESXi host with VMware HA enabled. This means you don’t need to worry about physical server failure. But it does not protect you against a failure on the software side that could be related to the Operating System or vCenter itself. If you figure out you want to go beyond hardware failure your only option is to deploy VMware vCenter Heartbeat. Heartbeat is able to protect all servers running SSO, Inventory Service and vCenter Service. It does it by replicating the server to another virtual or physical machine. Heartbeat is the tool if you have very strict SLA agreements. It will make your system more secure, but it will be on the cost of losing simplicity.

ESXi hosts

The ESXi hosts are the easiest to protect, and you don’t have to do a lot of planning in this area. Just make sure you have multiple ESXi hosts. All connected to the same network and access to the same shared storage. This way the virtual machines can gets it compute resources from any host in the cluster. On that cluster you will enable the DRS and HA functionality. I would not even worry about backing up the ESXi hosts. If you lose a host just reinstall it.

VMware vShield Manager

The vShield Manager comes as a virtual appliance from VMware. You simply download it and import into your management cluster. There is no native clustering functionality so you have to rely on VMware HA or VMware Fault Tolerance.

VMware Director Cell

The vCloud Director software has to be installed on a Linux machine running Red Hat Enterprise. Luckily for us the software has its own clustering functionality. What you basically do is install multiple Red Hat machines. All of these machine will share the same database on MS SQL or Oracle and the will also have access to the same NFS share. When implementing multiple vCloud Director Cells you would deploy the behind a load balancer. That load balancer could be VMware vShield Edge or you could use something else like Netscaler, F5 etc.

Database Server

The database server is extremely critical in the vCloud Environments. If we are only looking at the mandatory components you will have created three databases on your server. One for the SSO service, One for the vCenter Service and One for the vCloud Director Cell.

What you would normally do with your database is to look into clustering the MS SQL or Oracle with its own tools. If you don’t want to cluster the database. At least make sure that it is running as a virtual machine so you get the benefit of VMware HA. And of course remember to backup all databases at a regular interval.

If you are using a MS SQL cluster you have to know that it is not “officially” supported by VMware for the SSO service. Although they will do their best to help you if you have problems. I have installed all of the components against a MS SQL Cluster and haven’t experienced any problems yet. But we aware of this and set it as a risk.

vCenter Performance Graph Versus vCenter Operations Manager Metric Chart

March 11, 2013 by FrankBrix Leave a Comment

Most people who works with vSphere already knows the vCenter Performance Graphs. What most people don’t know is the “Metric Chart” in vCenter Operations Manager. In this article I want to discuss pros and cons of vCenter Performance Graphs vs. vCenter Operations Manager Metric Charts.

The Performance Graphs in vCenter can show information about the following CPU, Datastore, Disk, Memory, Network, Power, System, Virtual disk. You can the select if you want the graph to show: Real-time, Past day, Past week, Past month, past year or Custom.

When looking at these graphs the following table sums up what you are seeing.

vcenter_data

So by looking at the Real-Time graph you actually looks at a graph that is updated by a frequency of 20 seconds. So 180 datapoints in just one hour. But when you look at Past Day your graph is updated every 5 minutes. This is where rollup occurs. Without rollup you vCenter database would grow a lot in size. Whatever data you look at in the Past Day the graph is updated in 5 minutes interval, but what happens if you look at the graph more than 24 hours ago? Well if it is in the past week your graph will be updated every 30 minutes, but if you are further back your graph is updated every 2 hours! This basicly tells us that vCenter Performance Charts is BEST in Real-Time, it is alright in Past Day but moving further back than 24 hour data is being rolled up and we loose valuable insight. Look at the following screenshots to see what happens:

You should have noticed how we lose information when we try to investigate performance back in time. The further back we go the more we lose. Not only does data points get rolled up. But try and go back to the first graph. On the first graph we had the counters “Ready” and “Used” but on the other two we only have “Ready”. No I did not remove it. This is another feature of vCenter. Not only does it roll up. But it also chooses to remove counters as soon as we are looking at data more than one hour ago.

Now you should know some of the limitation with vCenter. So what is the solution if you want to look at data that is 1) More than one hour ago or 2) more than one day ago. The answer is of course vCenter Operations Managers Metric Charts. vCenter Operations Manager collects every counter and does not remove them when looking back. And vCenter Operations Manager have data points every 5 minutes. So yes you can argue that the past hour graph in vCenter is better than vCenter Operations Manager! That is true. The only time you actually should use vCenter Performance Graphs is when looking at the “Real-Time” graph.

Lets take a look in vCenter Operations Manager:

The graph is for the “Last Hour” but I can dive into any period and see all counters updated every 5 minutes. Maybe it is monday morning now and we have a reported performance problem that happened this saturday between 2am and 3am. Well just change the graph to that interval. Data is there, and you have EVERY counter.