Rob Garfoot's Blog

When I consult with development teams, at some point performance always rears its ugly head. When I code review something I often find it overly complex with lots of caching, “clever” code etc. and when I ask why the code is this complex the answer is usually for performance, E.g. “we need to cache the results as that’s an expensive operation”.

At this point I usually ask if they have any evidence to show how expensive the operation is, all too often the answer is “no”. Once again another developer is the victim of premature optimisation.

Now don’t get me wrong, optimising code is important, things like caching can indeed help reduce the cost of expensive operations. However, if that operation is only called once is caching really going to help? If the operation is “expensive” but only takes 2ms to execute and is called next to an operation that takes 2s is optimising it really going to help? I hope you all answered no there!

So how do we find out if we need to optimise, where do we focus our effort with limited time and resources to get the best return? That’s where profilers come in. If you’ve not used one before, they are applications that monitor how an application runs, usually capturing data such as number of calls, time taken, memory allocated etc.

Visual Studio comes with some pretty good profiling tools in the Team System version that can really help. There is a nice Performance Wizard on the Analyze menu that does much of the work for you. But as nice as the wizard is, it doesn’t fit all situations. If you need to profile services, code running as different users, on servers without VS installed or just need more control then the command line tools is where it’s at. You can download a standalone version from http://www.microsoft.com/downloads/details.aspx?familyid=fd02c7d6-5306-41f2-a1be-b7dcb74c9c0b&displaylang=en for installation on servers or machines without VS.

In this post I’m going to go through the (simple) steps needed to profile an application, I’ll leave interpreting the reports to another post.

Types of profiling

There are two types of profiling you can do, sampling and instrumentation. I’m going to mainly cover instrumentation as that’s the one I usually use as it’s the most accurate, it is however also the most invasive and requires changing assemblies so isn’t always suitable.

Setting up the profiler

If you have Visual Studio Team System for Developers then you have the profiler installed. The first thing to do is configure the command prompt to let you use the profiling tools. You’ll need to add them into your path. They are located by default in “C:\Program Files\Microsoft Visual Studio 9.0\Team Tools\Performance Tools” but it may be a different location on 64bit (Program Files (x86)) or if you' installed to a none standard location.

Once that’s done you need to enable the environment variables the profiler needs. You can do this with the VSPerfCLREnv command. You want to use /traceon if you are able to trace the application from the command console window by launching it but if you are tracing a service then use /globaltraceon. The global one will most likely need a reboot for the service to pick up the new settings so I usually do that afterwards.

Note if you are using sampling then you use /sampleon and /globalsampleon instead.

Preparing the assemblies

Now that the profiler is configured you need to prep the assemblies as we are instrumenting and not sampling. If you are sampling you can miss this step out.

Build your assemblies as normal, ensure you have pdb files generated though as you’ll need those later to view the source code from the profiling reports.

Now instrument all of the assemblies you want performance data from, you use VSInstr to do this. Simply specify the assembly name and you’ll get a new assembly and pdb file, the original ones get renamed to .orig.

Note that if you do this to a strong named assembly it’ll break the signing, you’ll need to use sn with the –Vr option to skip strong name verification on the assemblies you have instrumented. Don’t forgot to switch it back once you are done.

Profiling

Now you’re all ready to profile. To control the profiler you simply start it using the VSPerfCmd command. You specify the /start:trace option to start instrumenting or /start:sample to start sampling. You will also need /output to set the output filename for the trace results.

E.g. VSPerfCmd /start:trace /output:traceoutput.vsp

If you are tracing a service you’ll also need to add /user: and probably /crosssession as the service will be running as a different user and in a different session from the profiler. You’ll need to have the permissions to do this.

Once the profiler is started simply now run up your application and use it. It's important to make sure that the application is not running until after the profiler is started, this is often missed when profiling IIS, stop the app pool or do an IISReset before starting the profiler.

When you’ve done you need to shutdown the profiler, simply issue VSPerfCmd /shutdown to do that.

You should now have a shiny new .vsp file that contains the trace data. You can open this in Visual Studio, however the chances are you’ll just see stream of hex addresses and no source information.

I’m packing symbols

To make a .vsp file that can link back to source you need symbols, remember I told you that you’ll need these so if you don’t have them go back and start again and tell yourself that you’re a very naughty boy/girl (delete as appropriate).

How do we get the symbols in there or what if I don’t have Visual Studio? You can use the VSPerfReport command to pack the symbols into the .vsp file and to generate a text report of the trace.

You want to ensure you have your symbols set up to pull down from a symbol server to get all of them for the core .NET bits. To do that set the environment variable _NT_SYMBOL_PATH to point to the symbol server and your local cache.

E.g. set _NT_SYMBOL_PATH=symsrv*symsrv.dll*c:\mylocalcache*http://msdl.microsoft.com/download/symbols

See http://support.microsoft.com/kb/311503 for more details on setting the symbol path.

I usually just do a VSPerfReport /packsymbols /summary:function /symbolpath=<path to .instr.pdb files> traceoutput.vsp to get my symbols packed. This will also generate a summary text file, you need to do that even if you don’t need the summary or the symbols will not get packed. If you want a more detailed report then you can use the /summary option to specify what information you want in CSV format.

Comparing runs

Another nice feature of VSPerfReport is the /diff option. This lets you compare two .vsp files and shows the differences, this can be useful for checking against a baselined performance session to see if you’ve made things better or worse.

And finally…

Don’t forget that you want to turn instrumentation off if you are running it on a production server, replace the assemblies with the uninstrumented versions, revert the sn commands if you used them, run VSPerfCLREnv /off to clean up environement variables.

I’ll try and post something about interpreting the reports that the profiler gives you soon, after all doing all of this doesn’t help if you can’t read the data!

I recently just rebuilt my home PC and installed Winodws 7 on it. It was a nice fast, smooth install and generally went without a hitch.

That is, without a hitch until I discovered I'd installed Windows whilst the motherboard's SATA controller was in IDE mode and not AHCI mode. Why is this important? Well without AHCI you don't get nice things like power saving, native command queuing etc and that impacts the power consumption, speed and noise of your drives.

Of course that does rely on having a drive that supports those features, since like most newish drives mine do, I wanted to benefit.

Unfortunately just changing the setting in the BIOS causes Windows 7 and Windows Vista to both blue screen (BSOD) at boot up with the error STOP 0x0000007B INACCESSABLE_BOOT_DEVICE. That's because Windows doesn't have the drivers for AHCI since I installed it with IDE drivers.

How to get around this? Well it's surprisingly simple. After reading lots of hairy articles about hacking in drivers etc I found this knowledge base article http://support.microsoft.com/kb/922976. One simple registry setting and Windows enables it's default AHCI driver. You can then reboot, change your BIOS settings from IDE to AHCI and Windows will boot and redetect your controller and drives.

Once that's done, I rebooted to finish the install, then I installed the Intel Matrix Raid drivers so I had the actual manufacturers drivers on rather than the generic Windows ones.

All sorted now and it saved me a reinstall. I've just done a copy of 20GB of data from partition to partition on the same drive, much faster and quieter with AHCI than without. I guess that's down to NCQ being able to reorder the reads and writes into something a bit more effiicient.

It’s been a little while since I last blogged. Well I’ve decided to change the blogging engine that I use from Graffiti to Subtext. Why? Well no real reason other than I’m not great at web design and there are more themes available for Subtext that I like. I reserve the right to move back though!

I did have some issues moving posts since Graffiti doesn’t have a BlogML export function, only import. Fortunately I was able to find a little utility on Curt’s blog at http://darkfalz.com/post/2009/04/14/Graffiti-To-BlogML-Exporter.aspx that allowed me to export by blog and comments although it did lose some of the names against the comments, sorry about that those that posted.

In my previous post "Installing CrashPlan on WHS" I looked at installing CrashPlan on my home server and making it use UNC paths. In this post I was originally intending to look at what was needed to perform a backup between two home servers using a backup seeded from a portable hard disk.

I did manage to get all of that set up and working and was about to post when my friend noticed a problem. For some reason his backup files got erased and CrashPlan started a full backup over the net, this was going to take nearly 2 months.

After restoring the backup from portable hard disk again and starting things off it did the same thing again. So for now we've decided to uninstall CrashPlan until we have time to look into what's going on.

I've been looking at a backup solution for my home server for a bit. Now I'll forgive you at this point if you are thinking why do you need a backup solution for WHS when it does duplication.

Well several reasons but the main one being I want an offsite copy of most of the server contents in case of theft or fire. There are plenty of solutions out there for this but most of them require you to upload your files to the cloud and that is rather slow when trying to send a TB or so over a 800kbps uplink.

CrashPlan has the nice feature that you can back up to a friend's machine and you can seed the backup locally. So the plan is to get a new removable hard disk, backup, take to friends then update over the net.

Installing CrashPlan

I downloaded CrashPlan (www.crashplan.com) and installed it on my home server. I went to add my folders to the sources for backup and immediately hit a problem. CrashPlan doesn't let you specify a UNC path for the source (or the backup archive folder for that matter). Now I could just use D: but recommended practice for WHS is to always access data through the UNC paths so that's what I'd like to do.

The reason for this is that CrashPlan runs as SYSTEM and this account doesn't have network privileges and thus cannot access UNC paths.

To get around this I just changed the service to run as administrator and manually edited the %programfiles%/crashplan/conf/my.service.xml file to use UNC paths. The easiest way to do this is to add a dummy directory using the UI then find it in the config and change it. Use / instead of \ in the config file as it's a Java app and needs that.

I've only tested scanning to see if it can read the files and it seems to work, getting a removable drive and doing a backup is the next step.

Issues

Doing the above has 2 main issues, first you are running a service that provides a remote access interface on the internet as administrator, the second is that administrator may not have permissions to read the files being backed up.

To be honest both of these issues are present with CrashPlan anyway, it runs as SYSTEM and exposes the remote interface and SYSTEM can be denied permission on ACLs and so prevent backups from working. Fortunately WHS seems to create directories with ACLs that have both SYSTEM and Administrator in them with Full Control so it should work, I'll just have to remember not to change the ACLs on any files to remove those permissions.

Ideally the service interface should be running in a separate service with limited permissions and the backup engine service should be running as a user in the backup operators group.

I did initially try creating a new user in the backup operators group and run CrashPlan as that user, unfortunately CrashPlan doesn't use backup semantics when opening the files for backup and as such the ACL bypassing of the backup operators group doesn't kick in and it still can't access protected files.

I have suggested to the developers that the split the remote interface from the backup engine and use backup semantics when calling CreateFile() to access the files during backup. I'll let you know if I hear anything from them.

In the meantime I'm hoping the configuration I've built works ok, I'll follow this post up once I've got my external drive and have a backup done.

A useful little snippet when using WCF with web services that have test SSL certificates.

When you call a service in WCF it's still using the underlying .NET classes in System.Net so it's actually here that you need to tell to ignore the invalid SSL certificate.

As it turns out this is very easy, all you need is the snippet below and it's good. Obviously you shouldn't deploy to a live environment with this code since it essentially says all certificates are valid but it's a handy trick to get a service working with any SSL certificate for testing.

ServicePointManager.ServerCertificateValidationCallback = (sender, certificate, chain, sslPolicyErrors) => { return true; };

I just tried installing SQL Server 2008 and kept getting an error during the setup support rules phase of the install. It was insistent that my machine needed a reboot, after rebooting a few times I was pretty sure it didn't really need a reboot.

After a little bit of trawling around I found this thread on the MSDN forums. Basically navigate to "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager" and delete any values in "PendingFileRenameOperations".

Thanks to the post from Tomas Leung for that, I can now install SQL Server 2008.

Recently I wanted to send large files (several GB) from a rich desktop client to a web service. The client and service communicate using WCF and I thought that this would be quite easy. As it turns out it is but there are a few gotchas on the way.

Streaming

The first gotcha is that you do not want to buffer the message as you send it. Depending on the amount of RAM and the size of the file this may or may not be a problem on the client but it's almost certainly going to be a problem on the server with multiple clients sending files at the same time.

To solve this you can either write a service contract that accepts the file in chunks or use WCF streaming. In this instance I decided to use streaming.

Streaming in WCF does have some drawbacks which I'm not going to go into here so it's not suitable for everything but it was fine in this situation.

To use streaming you simply pass parameters of type 'Stream' and change the config to use streaming. This is a per endpoint setting so I set up a separate endpoint for the upload service.

Passing Metadata

The next problem I found was how to pass metadata with streams. Since I'm uploading via HTTP I want to make sure the file gets there intact so I'd like to transfer a checksum with the file (along with other details). WCF will not let you have any other parameters if using streams so how do you pass it.

Since using sessions with streaming can result in unpredictable behaviour there are a couple of options open. The first is to return a GUID (or other unique ID) after uploading the file and then allow a separate operation to set the metadata for that GUID and then link it to the previous file upload.The other option, and the one I chose, is to explicitly specify the message and send the metadata in the message header.

Max Message Sizes

The next issue I faced was the size of the messages, you need to ensure that the maxReceivedMessageSize is set large enough for your largest file. Since headers are always buffered, even when streaming, you want to ensure that this doesn't result in DOS attacks by settings maxBufferSize to something reasonable such as 64K. This will allow large streamed bodies but limits the size of the headers.

Gotchas

So that's it, all set up and working now. Well not quite. As I mentioned at the start there are a few gotchas with this.

First, the VS webdev server (Cassini) cannot handle streaming over HTTP. This is simple to fix, use IIS or self host in a console app or windows service.

Second, IIS uses the ASP.NET maxRequestLength setting for the max length not the WCF setting. You need to add this to your web config.

Third, IIS cannot transfer more than 2GB of data. You'll need to self host to get around this if you need to send more data than that.

Fourth, timeouts can occur so you need to increase the send/receive timeouts in WCF.

Code

So here are samples of the code bits I used.

web.config - server

app.config - client

Contracts

The following blogs / articles were useful in sorting this out.

• WCF Streaming: Upload files over HTTP
• Large Data and Streaming
• How to: Enable Streaming

Visual Studio Team System has some nice testing feature, amongst which is the ability to record and playback web browsing as part of a web test. The recorder makes building web tests much easier that writing the test scripts by hand. I went to record a new web test today but the test recorder didn't appear.

I've recently installed IE8, I don't know if this was the culprit for breaking the recorder or not but fortunately the fix was quite easy. After a bit of searching I came across an entry on Michael Taute's blog at blogs.msdn.com/mtaute/archive/2007/11/09/diagnosing-and-fixing-web-test-recorder-bar-issues.aspx.

This article contains several possible fixes for these kinds of problems, the one that worked for me was regarding VSTS 2008 and Vista 64 (I'm running on Server 2008 64 bit). I've included that particular fix below but have a look at the original article for others.

VSTS 2008 : Vista (64 bit) : Recorder bar does not appear when recording a new webtest

Fix:  Vista caches the list of explorer bars you have available and the recorder bar was not included in your list.  The fix is to force Windows to rebuild that cache.  To do this, first make sure you have all Internet Explorer instances shut down, then open the 32 bit registry editor and delete the following keys:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Discardable\PostSetup\Component Categories\{00021493-0000-0000-C000-000000000046}
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Discardable\PostSetup\Component Categories\{00021494-0000-0000-C000-000000000046}

[Note: by default, the 32 bit registry editor is located in %WINDIR%\SysWow64\RegEdt32.exe]

The next time you boot Internet Explorer, your explorer bar cache will be rebuilt and the recorder bar should be available.
 

I use a Pinnacle ShowCenter for as my network media player for playing back music and movies on my A/V equipment. It's a nice bit of hardware but the software it comes with doesn't fit my needs. Fortunately the player uses a simple web browser for it's UI and this is served from the Pinnacle server so it's possible to replace the server component with a new one that provides whatever functionality you like.

There are a few open source server packages out there for the ShowCenter (and other Syabas based media devices) but the one I use, and in my opinion the best of them, is the SwissCenter (available from www.swisscenter.co.uk).

This software is written using PHP, Apache and MySQL and runs on either Linux or Windows. However, on Windows there is a nice simple install option using another project called Simese (Simple Media Server). This provides a single installer for the Simese media server, PHP, MySQL and SwissCenter.

You can install this on a WHS quite easily, I just downloaded the latest build (1.45 Simese and 1.20.1 SwissCenter at the time of writing), logged on via terminal services and installed it. I made sure that I entered the media locations as UNC paths and that the user Simese was running as had access to the media locations and it all works fine. Upgrading the box to 2GB helped though since running this stuff on a standard HP MediaSmart with 512MB is a little slow.

In addition to installing Simese and SwissCenter I also installed MusicIP. This is an optional component that SwissCenter can use in order to create custom playlists based on similar music, you simply select a track in SwissCenter and then get a link to 'play similar', this functionality uses MusicIP to generate the playlist which the SwissCenter then plays.

Installing MusicIP on the WHS was also pretty easy, I just followed the instructions for a standard install on Windows. The only thing I needed to change was to change the user the service runs as to ensure it had access to my media.

Overall a nice little setup and I'm quite happy I can run it all from my home server now.