Finding Memory Leaks in Azure
Tracking down memory leaks isn’t anyone’s idea of a good time. There are a lot of great articles out there, but they can be hard to find, and some of them are getting a little long in the tooth. Things have changed since 2004! It turns out that you can find memory leaks in your Azure apps without too much effort, and without buying any tools. Here’s how.
Setup
You’ll need a couple of things to get started. First, make sure you’ve enabled RDP access to your Azure roles. It’s pretty easy to set up through Visual Studio, and there are plenty of walkthroughs already, so I’m not going to cover it. If you didn’t have RDP set up already, then re-deploy your application with RDP enabled.
Second, you need WinDbg which is part of the Debugging Tools for Windows. WinDbg is not as easy to use as some of the commercial products, but if you’ve seen a command line before then you’ll be fine. It also has the notable advantage of being free. You can get the installer here. It does take a little one-time configuration – I recommend the Basic Configuration section of this post.
Collecting a Memory Dump
Once the setup is out of the way, you need to collect a memory dump. To do that, you need to RDP into your machine, but with a slight settings change that’ll help you download the dump file.
Go to the Azure portal, select an instance of your role, and hit the Connect button. That should download a .RDP file to your machine. Don’t open it immediately, we want to change settings to share your desktop’s drive to the Azure VM. Go find the .RDP file on your disk, right click it and select “Edit”. Go to the “Local Resources” tab, click the “More” button, and share one of your drives. It’ll look something like this:
Changing settings for RDP
Now connect to the VM as normal. Once you’re logged in, open Windows Explorer and you verify that your desktop drive shows up in the list of drives. Then open Task Manager and find the leaking process. If you’re debugging a worker role, it’ll probably be WaWorkerHost.exe. For web roles, it will probably be w3wp.exe. (Or you could just look for the one that’s using a whole lot of memory!) When you’ve found the misbehaving process right click on it and select “Create Dump File”.
This will take a while. When it’s done you’ll see a dialog telling you where to find the dump file. Locate the dump file, then copy it to the drive you shared from your desktop. Dump files tend to be very large, so this will probably take a few minutes.
Analyzing the Dump File
Now you need crack open the dump file and see what’s going on. The dump file contains a snapshot of all the memory in your program. We’re going to look through it to figure out what’s taking up all that memory. You’re basically looking for anything suspicious, such as a class that has thousands more instances than it should. Open WinDbg and use File -> Open Crash Dump to open the dump file you just collected. Then type “.load sos” to load the .NET debugger module.
Now you need to get out your magnifying glass and detective hat, and try to figure out what’s going on in your program. I’m certainly no expert on this topic, but this approach worked for me.
- Run !dumpheap -stat. This will dump out a summary of what classes are using the most memory, with the biggest ones at the bottom. Look for anything that seems suspicious.
- When you find something suspicious, copy the MT value (the first column), then run !dumpheap -mt <copied mt value>. This will dump out a list of all the instances of that object.
- Copy the Address (first column, again) for an object, and run !gcroot <copied address>. This should show you what objects are referring to that object instance. It should give you an idea of who created it, and why it hasn’t been reclaimed. Repeat the process for several objects, and look for patterns.
Hopefully the above gives you some ideas. If not, try taking two or three dumps and look at what’s been allocated in the time between them. If you’re still stuck, Tess Ferrandez wrote a very good and still relevant blog post series about this topic. I particularly recommend part 6 and this followup post for ASP.NET problems.
Written by Brian Reischl