UK Test Managers’ Forum – 28 April 2010

My first visit to the Test Managers’ Forum on 28 April was a positive experience for me.

There were six sessions on the programme with three running concurrently either side of a break.  The project work I am involved with at the moment involves a requirement to measure the performance of our systems under load so I went to the two load testing talks.  The first of these was entitled “Effective Load Testers. Who are they? How are they created?” and was presented by Gordon McKeown from Facilita.

We were a small group and the session quickly became a discussion with lots of viewpoints put forward.  The main points from my perspective were:

  • Load testing is a highly skilled, but wide ranging, aspect of testing.  The wider environment in which the system operates must be understood and its impact needs to be considered.  Examples of the things to consider included:
    • the effect of changes to network infrastructure;
    • the physical processing power of the computers running tests (CPU, memory, hard disk speed, etc.); and
    • the operating system and software configuration on the system under test.
  • There are essentially four parts to load testing and the skills needed are rarely found in one person:
    • planning the test strategy;
    • writing and executing the test scripts;
    • analysing the data coming out of the test; and
    • closure activities such as reporting the results.
  • At the analysis stage it is very easy to become bogged down by too many numbers being thrown at you.  A wealth of experience in interpreting the numbers is crucial.
  • Good load testers often start from a development background or some other specialism and then move into testing once they have acquired a great deal of experience.  Others’ experience suggested that it is impractical for a new starter straight from university to go straight into load testing because they do not have the wide experience necessary to understand what they are seeing.
  • People moving into load testing need to have the right mindset to be successful.  They need to have an enquiring mind and know how to get at and ask the right questions to get a feel for the risks they are trying to address.
  • A good way of getting into load testing is to get experience with performance monitoring tools built into Windows – for example Perfmon – and get an understanding of what the different counters mean and what affect different applications and/or processes have on those counters.  They should then seek to get experience in a company with a good training philosophy to build up the requisite knowledge of their chosen subject area – be it hardware diagnostics, network infrastructure or plain old development.  From there they will build up the experience needed to get into effective load testing.

After the break we met up, again a small group, for discussions under the heading of “Performance management: from black art to process”.  This discussion was facilitated by Peter Holditch from Dynatrace Software.  Dynatrace produce an application called Dynatrace which allows load testers to carry out end-to-end analysis of the path transactions take through their systems.

One of the main benefits of software like this is that it enables testers to see where the bottlenecks are in fine granularity – for example they can see whether there is a hold up in one particular server – and can drill down to see the individual services affected.

As before, there was a lot of debate about the process supported by Dynatrace’s software.  It was emphasised that care would need to be taken to avoid information overload.  This is, after all, a tool to help the load tester do his/her job.  The skill is in knowing which counters to start with and how to interpret them.  A starter for ten was suggested:

  • Memory usage
  • CPU utilisation
  • Network traffic
  • Disk queue

From here, testers could identify further avenues of exploration to home in on discrepancies.

I could see a great deal of benefit from having such software and I could see the benefits that it could bring as part of a wider strategy to understand the performance quality attribute of our software.  It would be of limited use for me personally because I simply do not have the detailed knowledge of all the inwards of our network infrastructure and computers to make good use of it.  It is something that I am slowly but surely learning though and I hope to become proficient enough to make use of such a tool.

After the talks we adjourned to the bar area upstairs where a tab had been set up and the networking continued.

I thoroughly enjoyed the afternoon and feel that I benefitted greatly from going.  I hope to make this a regular fixture in my calendar along with the SIGiST.  The TMF is a different type of event all together to SIGiST and is aimed primarily at testing managers and experienced practitioners.  The format is much more geared towards networking and discussions whereas the SIGiST tends to be that bit more formal – at least that is the impression I got from this time round!

For any test managers that have not been, I highly recommend attendance.  More information about the forum is available at and I understand the slides will be made available from all the talks later on.

Thank you very much to all those who facilitated and especially to Paul Gerrard from Gerrard Consulting who organised the whole thing.

5 Responses to “UK Test Managers’ Forum – 28 April 2010”

  1. Thomas Ponnet Says:

    Nice writeup, thanks for that. Makes me even more sad that I missed the event this year again – I couldn’t go last year either. Oh well, next time.
    Are you testing your application with hardware on site or are you using the cloud for that? I’m asking as I’m looking into the latter at the moment.

    • pedantictester Says:

      Wow! That was quick! Thanks for the comment.

      Both – primarily testing on the cloud now but that brings its own challenges such as not knowing whether your elastic storage volumes are in the same bit of kit when they ‘expand’. Ultimately I am finding that I have to ‘go with the flow’ a bit with it all at the moment. This will not be the case for long though!

  2. James Christie Says:

    Thanks for this update Stephen.

    It’s interesting what you said about the differing skills rarely being found in one person. My first involvement in load testing was being asked to write a test strategy for a load testing exercise. I was a bit nervous and baffled. The actual load testing was a mystery to me, and the strategy seemed the easy part. I couldn’t see why the load testers couldn’t do the easy part, when they were so skilled at the hard part.

    However, they seemed grateful for someone coming in to speak to the various stakeholders and help shape their thoughts along with the various demands and business constraints into a coherent strategy that allowed them to go off and do their thing.

    • pedantictester Says:

      I have often found that people are grateful for someone to come on board and deal with speaking to stakeholders. This is especially so when you get technical and non-technical people in a meeting neither of whom really understand the other side’s viewpoint.

      Thanks for the comment.

  3. Paul Gerrard Says:

    Hi Stephen,

    Very belated “thanks very much” for the positive comments on TMF. Hope to see you at the next one 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: