www.vExperienced.co.uk

For the last few weeks we’ve been struggling with our vSphere5 upgrade. What I assumed would be a simple VUM orchestrated upgrade turned into a major pain, but I guess that’s why they say ‘never assume’!

Summary: there’s a bug in the upgrade process whereby NFS mounts are lost during the upgrade from vSphere4 to vSphere5;

• if you have NFS datastores with a space in the name
• and you’re using ESX classic (ESXi is not affected)

Our issue was that after the upgrade completed, the host would start back up but the NFS mounts would be missing. As we use NFS almost exclusively for our storage this was a showstopper. We quickly found that we could simply remount the NFS with no changes or reboots required so there was no obvious reason why the upgrade process didn’t remount them. With over fifty hosts to upgrade however the required manual intervention meant we couldn’ t automate the whole process (OK, PowerCLI would have done the trick but I didn’t feel inspired to code a solution) and we aren’t licenced for Host Profiles which would also have made life easier. Thus started the process of reproducing and narrowing down the problem.

• We tried both G6 and G7 blades as well as G6 rack mount servers (DL380s)
• We used interactive installs using a DVD of the VMware ESXi v5 image
• We used VUM to upgrade hosts using both the VMware ESXi v5 image and the HP ESXi v5 image
• We upgraded from ESXv4.0u1 to ESX 4.1 and then onto ESXiv5
• We used storage arrays with both Netapp ONTAP v7 and ONTAP v8 (to minimise the possibility of the storage array firmware being at fault)
• We upgraded hosts both joined to and isolated from from vCentre

Every scenario we tried produced the same issue. We also logged a call with VMware (SR 11130325012) and yesterday they finally reproduced and identified the issue as a space in the datastore name. As a workaround you can simply rename your datastores to remove the spaces, perform the upgrade, and then rename them back. Not ideal for us (we have over fifty NFS datastores on each host) but better than a kick in the teeth!

There will be a KB article released shortly so until then treat the above information with caution – no doubt VMware will confirm the technical details more accurately than I have done here. I’m amazed that no-one else has run into this six months after the general availability of vSphere5 – maybe NFS isn’t taking over the world as much as I’d hoped!  I’ll update this article when the KB is posted but in the meantime NFS users beware.

Sad I know, but it’s kinda nice to have discovered my own KB article. Who’d have thought that having too much space in my datastores would ever cause a problem?

While undertaking some scheduled maintenance on our Netapp shared storage (due to an NVRAM issue) we discovered that some of our Oracle applications didn’t handle the controller outage as gracefully as we expected. In particular several Oracle RAC nodes in our dev and test environments rebooted during the Netapp downtime. Strangely this only affected our virtual Oracle RAC nodes so our initial diagnosis focused on the virtual infrastructure.

Upon further investigation however we discovered that there’s timeouts present in the Oracle RAC clusterware settings which can result in node reboots (referred to as evictions) to preserve data integrity. This affects both Oracle 10g and 11g RAC database servers although the fix for both is similar. NOTE: We’ve been running Oracle 10g for a few years but hadn’t had similar problems previously as the default timeout value of 60 seconds is higher than the 30 second default for 11g.

Both Netapp and Oracle publish guidance on this issue;

• CSS Timeout Computation in Oracle Clusterware (login required – Oracle Metalink Note 294430.1)
• What are the Oracle Clusterware/RAC CSS Timeout Settings with NetApp Clustered Failover? (Netapp KB3011276). This article covers Oracle 10g through to Oracle 11gR1 and is intended to complement the above Oracle guidance.

The above guidance focuses on the DiskTimeOut parameter (known as the voting disk timeout) as this is impacted if the voting disk resides on a Netapp. What it doesn’t cover is when the underlying Linux OS also resides on the affected Netapp, as it can with a virtual Oracle server (assuming you want HA/DRS). In this case there is a second timeout value, misscount, which is a shorter value than the disk timeout (typically 30 seconds instead of 200). If a node can’t reach any of the other RAC nodes within misscount seconds timeframe it will start split-brain resolution and probably evict itself from the cluster by doing a reboot. When the Netapp failed over our VMs were freezing for longer than 30 seconds, causing the reboots. After we increased the network timeout we were able to successfully failover our Netapp’s with no impact on the virtual RAC servers.

NOTE: A cluster failover (CFO) is not the only event which can trigger this behaviour. Anything which impacts the availability of the filesystem such as I/O failures (faulty cables, failed FC switches etc) or delays (multipathing changes) can have a similar impact. Changing the timeout parameters can impact the availability of your RAC cluster as increasing the value results in a longer period before the other RAC cluster nodes react to a node failure.

Configuring the clusterware network timeouts

The changes need to be applied within the Oracle application stack rather than at the Netapp or VMware layer. On the RAC database server check the cssd.log logfile to understand the cause of the node eviction. If you think it’s due to a timeout you can change it using the below command;

# $GRID_HOME/bin/crsctl set css misscount 180 

To check the new settings has been applied;

# $GRID_HOME/bin/crsctl get css misscount

The clusterware needs a restart for these new values to take affect, so bounce the cluster;

# $GRID_HOME/bin/crs_stop -all
# $GRID_HOME/bin/crs_start –all

Further Reading

Netapp Best Practice Guidelines for Oracle Database 11g (Netapp TR3633). Section 4.7 in particular is relevant.

Netapp for Oracle database (Netapp Verified  Architecture)

Oracle 10gR2 RAC: Setting up Oracle Cluster Synchronization Services with NetApp Storage for High Availability (Netapp TR3555).

How long it takes for Standard active/active cluster to failover

Node evictions in RAC environment

Troubleshooting broken clusterware

Oracle support docs (login required);

• NOTE:284752.1 – 10g RAC: Steps To Increase CSS Misscount, Reboottime and Disktimeout
• NOTE:559365.1 – Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions
• Note: 265769.1 – Troubleshooting 10g and 11.1 Clusterware Reboots
• NOTE: 783456.1 – CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage

In last night’s VMware Community podcast John Hall, VMware’s lead technical certification developer gave some tidbits of information about the upcoming VCAP5 exams;

• There will be an expedited path for those with VCAP4 certifications BUT they will be similar to the VCP upgrade in that it’ll be a time limited offer. He didn’t specify exactly what form this would take but with the VCP upgrade you have roughly six months to take the new exam with no course prerequisites.  I’m guessing you’ll have a similar period where the VCP5 prerequisite doesn’t apply.

• While not committing to dates he did state that exams might be available at the upcoming partner exchange which starts on Feb 13th 2012. Even if those dates slip expect them soon!

With the upcoming Feb 29th deadline for the VCP5 exam you’d better get your study skates on. If you don’t take the VCP5 before the 29th and you’re not in a position to take the the new VCAP5 exams in the ‘discount’ period (however long that turns out to be) you might find yourself needing to sit a What’s New course and passing the VCP5 exam before you’re even eligible for the VCAP5 exams. Not a pleasant thought!

UPDATE FEB 2012 – After some further testing I’ve concluded that this is a bigger pain than I previously thought. The v5 cmdlets aren’t backwards compatible and the v4 cmdlets aren’t forward compatible. This means that while you’re running a mixed environment with VMs on v4/v5 VMtools a single script can’t run against them all. Think audit scripts, AV update scripts etc. You’ll have to run the script twice, from two different workstations, one running PowerCLI v4 (against the v4 VMs) and one running PowerCLI v5 (against the v5 VMs). And I thought this was meant to be an improvement??

———- original article ————–

There are quite a few enhancements in PowerCLI v5 (there’s a good summary at Julian Wood’s site) but if you make use of the guest OS cmdlets proceed with caution!

We have an automated provisioning script which we use to build new virtual servers. This does everything from provisioning storage on our backend Netapps to creating the VM and customising configuration inside the guest OS. The guest OS configuration makes use of the ‘VMGuest’ family  of cmdlets;

• Invoke-VMScript
• Copy-VMGuestFile
• Get-VMGuest, Restart-VMGuest etc

Unfortunately since upgrading to vSphere5 and PowerCLI v5 we’ve discovered that the guest OS cmdlets are NOT backwards compatible! This means if you upgrade to PowerCLI v5 but your hosts aren’t running ESXiv5 and more importantly the VMTools aren’t the most up to date version any calls using the v5 cmdlets (such as Invoke-VMGuest) will no longer work. Presumably this is due to the integration of the VIX API into the base vSphere API – I’m guessing the new cmdlets (via the VMTools interface) now require the built-in API as a prerequisite.

As PowerCLI is a client side install the workaround is to have a separate install (on another PC for example) which still runs PowerCLI v4, but we have our vCenter server setup as a central scripting station (it’s simpler than every member of the team keeping up with releases, plugins etc) so this is definitely not ideal.

This is covered in VMware KB2010065.The PowerCLI v5 release notes are also worth a read.

Further Reading

Will Invoke-VMGuest work? (LucD)

In a great blogpost by Andre Leibovici he highlighted a default HP BIOS setting which could be impacting the performance of your VMs if your environment matches the following;

• low physical CPU utilisation
• higher than expected CPU %Ready times

Julian Wood has also blogged about this issue (Your HP blades may be underperforming) but neither go into too much detail about the fix. Having investigated I thought I’d record it here for others convenience.

To check for these symptoms you could use the VI client, ESXTOP in batch mode combined with the batch processing scripts in the vMA to capture pCPU statistics from a group of servers, or PowerCLI -whichever suits your skillset.

We run HP C-class blades and after checking the VMware knowledgebase article KB1018206 and a sample of our BIOS settings we found that it applied to us too – not surprising as we don’t modify the BIOS defaults during provisioning.

Using a mixture of ESXTOP and vCenter’s performance charts I was able to confirm that the %CPU Ready was hovering around the 4% mark even when the physical host was using less than 15% pCPU. After changing the power setting the same VMs (under a similar load) dropped to under 1% CPU Ready (the change was made at 17:00 if you look at the graph).
Not necessarily a show stopper but definitely an improvement.

For my infrastructure (with around 160 physical blades) changing them all was a time consuming process (and could potentially be disruptive depending on whether your ESX/i hosts are all clustered).

You can check the current power management setting in various ways;

• in the BIOS settings (slow and potentially disruptive)
• via the ILO (under Power Management, Power settings) or via the ILO CLI
• in the VI client. If the underlying BIOS is set to Dynamic Power Savings it’ll show as ‘Not Supported’ . ie the hardware is controlling power management. Where to check depends on your version of ESX (or ESXi);
  • For a 40 host go to Configuration -> Processors and look at the Power Management settings.
  • For a 4.1 host go to Configuration -> Power Management and look at the Active Policy. You can also configure it using the Properties button.
• You can also use PowerCLI (ESX4 only) by querying the host’s Advanced setting ‘Power.cpupolicy’

  get-vmhost myhost | get-vmhostAdvancedConfiguration -name Power.cpupolicy

Changing power saving via the ILO

Read more…

Over the Christmas break I finally got some time to upgrade my home lab (link). One of my tasks was to build a new shared storage server and it was while installing the base ESXi (v5, build 469512) that I ran into an issue. I was unable to add any of the local disks to my ESXi host as VMFS datastores as I got the error “HostDatastoreSystem.QueryVmfsDatastoreCreateOptions” for object ‘ha-datastoresystem’ on ESXi….” as shown below;

The VI client error when adding a new datastore

I’d used this host and the same disks previously as an ESX4 host so I knew hardware incompatibility wasn’t an issue. Just in case I tried VMFS3 (instead of VMFS5) with the same result. I’ve run into a similar issue before with HP DL380G5′s where the workaround is to use the VI client connected directly to the host rather than vCentre. I connected directly to the host but got the same result. At this point I resorted to Google as I had a pretty specific error message. One of the first pages was this helpful blogpost at Eversity.nl (it’s always the Dutch isn’t it?) which confirmed it was an issue with pre-existing or incompatible information on the hard disks. There are various situations which might lead to pre-existing info on the disk;

• Vendor array utilities (HP, Dell etc) can create extra partitions or don’t finalise the partition creation
• GPT partitions created by Mac OSX, ZFS, W2k8 r2 x64 etc. Microsoft have a good explanation of GPT.

This made a lot of sense as I’d previously been trialling this host (with ZFS pools) as a NexentaStor CE storage server

Read more…

Prior to buying Netapp Operations Manager we used to run lots of daily checks to ensure the uptime and health of our Netapp controllers. Many of these checks were written using the Data ONTAP Powershell Toolkit so I thought I’d post them up in case they’re of use to anyone else.

First up is a function to check for the ‘maxfiles‘ value (the number of inodes consumed in a volume). This is typically a large number (often in the millions) and is based on the volume size, but we had an Oracle process which dumped huge numbers of tiny files on a regular basis, consuming all the available inodes. This article only covers checking for these occurrences – if you need a fix I’d suggest checking out Netapp’s advice or this discussion for possible solutions.

Simply add the function (below) to your Powershell profile (or maybe build a module) and then a Powershell one-liner can be used to check;

connect-NaController yourcontroller | get-NaMaxfiles -Percent 30

This will give you output like this;

Controller : Netapp01
Name       : test_vol01
FilesUsed  : 268947
FilesTotal : 778230
%FilesUsed : 35

Controller : Netapp01
Name       : test_vol02
FilesUsed  : 678111
FilesTotal : 1369688
%FilesUsed : 50

And here’s the function;

function Get-NaMaxfiles {
<#
.SYNOPSIS
 Find volumes where the maxfiles values is greater than a specified threshold (default 50%).
.DESCRIPTION
 Find volumes where the maxfiles values is greater than a specified threshold (default 50%).
.PARAMETER Controller
 NetApp Controller to query (defaults to current controller if not specified).
.PARAMETER Percent
 Filters the results to volumes when the %used files is greater than the number specified. Defaults to 50% if not specified.
.EXAMPLE
 connect-NaController zcgprsan1n1 | get-NaMaxfiles -Percent 30

 Get all volumes on filer zcgprsan1n1 where the number of files used is greater than 30% of the max available
#>
 [cmdletBinding()]
 Param(
 [Parameter(Mandatory=$false,
 ValueFromPipeLine=$true
 )]
 [NetApp.Ontapi.Filer.NaController]
 $Controller=($CurrentNaController)
 ,
 [Parameter(Mandatory=$false)]
 [int]
 $Percent=50
 )
 Begin {
 #check that a controller has been specified
 }
 Process {
 $exception = $null
 try {
 # create a null valued instance of $vol within the local scope
 $vols = $null
 $vols = Get-NaVol -controller $Controller -ErrorAction "Stop" | where {$_.FilesTotal -gt 0 -and ($_.FilesUsed/$_.FilesTotal)*100 -gt $Percent}
 #check that at least one volume exists on this controller
 if ($vols -ne $null) {
 foreach ($vol in $vols) {
 #calculate the percentage of files used and add a field to the Volume object with the value
 $filesPercent = [int](($vol.FilesUsed/$vol.FilesTotal)*100)
 add-member -inputobject $vol -membertype noteproperty -name Controller -value $Controller.Name
 add-member -inputobject $vol -membertype noteproperty -name %FilesUsed -value $filesPercent
 }
 }
 }
 catch {
 $exception = $_
 }
 if ($exception -eq $null) {
 $returnValue = ($vols | Sort-Object -Property "Used" -Descending | Select-Object -Property "Controller","Name","FilesUsed","FilesTotal","%FilesUsed")
 }
 else {
 $returnValue = $exception
 }
 return $returnValue
 }
}

———————————————–

UPDATE FEB 2012 – Netapp have just released a firmware update for the battery and confirmed that all 32xx series controllers shipped before Feb 2012 are susceptible to this fault. You can read more (including instructions for applying the update – it’s NOT click, click, next) via the official Netapp KB article. I’ll be applying this to my production controllers soon so I’ll let you know if I encounter any problems.

———————————————–

Recently (Dec 2011) I’ve been experiencing a few issues with the newer Netapp filers at my work, specifically the 3240 controllers. There is currently a known issue with NVRAM battery charging which if you’re not aware of can result in unplanned failovers of your Netapp controllers. This applies to the 3200 series (including the v3200 and SA320).

We have six of these controllers and my first warning (back at the beginning of November) was an autosupport email notification;

This message indicates that the NVRAM or NVMEM battery is below the minimum voltage required to safeguard data in the event of an unexpected disruption.

If the system has been halted and powered off for some time, this message is expected.This message repeats HOURLY as long as NVRAM or NVMEM battery is below the minimum voltage, if you are using ONTAP version 7.5, 8.1, or greater with an appliance that uses an NVMEM battery, the error will repeat WEEKLY.

When the storage controller is up and running, the battery will be charged to its normal operating capacity and this message should stop. However, if this message persists, there may be a problem with the NVRAM or NVMEM battery.

This was unexpected but a faulty backup battery wasn’t an immediate priority – after all it’s only required to protect against power failures or controller crashes which are pretty rare. A few days later it became a high priority after the controller failed over unexpectedly. This failure was actually triggered by the low battery level and is expected behaviour as documented in Netapp KB2011413 though it’s not made overly clear that a controller shutdown is the default action if the battery issue persists for 24 hours. I logged a call with Netapp but they were unaware of any systemic issues and despite pointing out that this was affecting all six of our controllers they simply sent replacement NVRAM batteries and suggested we swap them all out. I posted a question on the Netapp forums but at the time no-one else seemed to be having the same issue. The new batteries were duly fitted and the problem seemed to be resolved – I’ve since rechecked our battery charges and they’re stable at around 150 hours.

An update in an email we received from Netapp on the 22nd December now states that it’s a known firmware issue with a permanent fix currently expected in Feb 2012. Netapp advise that further downtime will be required to implement the fix when it’s made available.

Don’t ignore low battery alerts!

Read more…