Archive for April, 2010

UK Test Managers’ Forum – 28 April 2010

29 April, 2010

My first visit to the Test Managers’ Forum on 28 April was a positive experience for me.

There were six sessions on the programme with three running concurrently either side of a break.  The project work I am involved with at the moment involves a requirement to measure the performance of our systems under load so I went to the two load testing talks.  The first of these was entitled “Effective Load Testers. Who are they? How are they created?” and was presented by Gordon McKeown from Facilita.

We were a small group and the session quickly became a discussion with lots of viewpoints put forward.  The main points from my perspective were:

  • Load testing is a highly skilled, but wide ranging, aspect of testing.  The wider environment in which the system operates must be understood and its impact needs to be considered.  Examples of the things to consider included:
    • the effect of changes to network infrastructure;
    • the physical processing power of the computers running tests (CPU, memory, hard disk speed, etc.); and
    • the operating system and software configuration on the system under test.
  • There are essentially four parts to load testing and the skills needed are rarely found in one person:
    • planning the test strategy;
    • writing and executing the test scripts;
    • analysing the data coming out of the test; and
    • closure activities such as reporting the results.
  • At the analysis stage it is very easy to become bogged down by too many numbers being thrown at you.  A wealth of experience in interpreting the numbers is crucial.
  • Good load testers often start from a development background or some other specialism and then move into testing once they have acquired a great deal of experience.  Others’ experience suggested that it is impractical for a new starter straight from university to go straight into load testing because they do not have the wide experience necessary to understand what they are seeing.
  • People moving into load testing need to have the right mindset to be successful.  They need to have an enquiring mind and know how to get at and ask the right questions to get a feel for the risks they are trying to address.
  • A good way of getting into load testing is to get experience with performance monitoring tools built into Windows – for example Perfmon – and get an understanding of what the different counters mean and what affect different applications and/or processes have on those counters.  They should then seek to get experience in a company with a good training philosophy to build up the requisite knowledge of their chosen subject area – be it hardware diagnostics, network infrastructure or plain old development.  From there they will build up the experience needed to get into effective load testing.

After the break we met up, again a small group, for discussions under the heading of “Performance management: from black art to process”.  This discussion was facilitated by Peter Holditch from Dynatrace Software.  Dynatrace produce an application called Dynatrace which allows load testers to carry out end-to-end analysis of the path transactions take through their systems.

One of the main benefits of software like this is that it enables testers to see where the bottlenecks are in fine granularity – for example they can see whether there is a hold up in one particular server – and can drill down to see the individual services affected.

As before, there was a lot of debate about the process supported by Dynatrace’s software.  It was emphasised that care would need to be taken to avoid information overload.  This is, after all, a tool to help the load tester do his/her job.  The skill is in knowing which counters to start with and how to interpret them.  A starter for ten was suggested:

  • Memory usage
  • CPU utilisation
  • Network traffic
  • Disk queue

From here, testers could identify further avenues of exploration to home in on discrepancies.

I could see a great deal of benefit from having such software and I could see the benefits that it could bring as part of a wider strategy to understand the performance quality attribute of our software.  It would be of limited use for me personally because I simply do not have the detailed knowledge of all the inwards of our network infrastructure and computers to make good use of it.  It is something that I am slowly but surely learning though and I hope to become proficient enough to make use of such a tool.

After the talks we adjourned to the bar area upstairs where a tab had been set up and the networking continued.

I thoroughly enjoyed the afternoon and feel that I benefitted greatly from going.  I hope to make this a regular fixture in my calendar along with the SIGiST.  The TMF is a different type of event all together to SIGiST and is aimed primarily at testing managers and experienced practitioners.  The format is much more geared towards networking and discussions whereas the SIGiST tends to be that bit more formal – at least that is the impression I got from this time round!

For any test managers that have not been, I highly recommend attendance.  More information about the forum is available at and I understand the slides will be made available from all the talks later on.

Thank you very much to all those who facilitated and especially to Paul Gerrard from Gerrard Consulting who organised the whole thing.


Usability and Process Control

23 April, 2010

A blog post by James Christie ( has prompted me to post a few thoughts about recent experiences I have had where illogical (at least to the user) processes have apparently gone unchallenged.

The Utility Company

I received a letter one morning from my electricity and gas supplier to express their sorrow that I was leaving them – but I was not intending to leave them at all!  I contacted the company and explained the situation.  The operator looked up my account details and said that there was no record of me leaving them.

A couple of months later I attempted to log on to my online account management system but I was unable to do so because I no longer had an account with the company.  I contacted the helpdesk again and was told that another utility company was supplying my electricity (not my gas – that was still with them).

What followed was a lengthy process to get my account reinstated.  I became very interested in establishing what went wrong when the same thing happened a second time.  This is what I found out:

  • A representative of another company had mistyped the meter number of a property.
  • At no point was a check carried out to make sure the installation and billing addresses matched.
  • My energy company had no means of verifying that the details matched because insufficient information was sent to enable them to do so.
  • Energy companies are reliant on people noticing that they have been erroneously transferred and informing them.
  • Having erroneously transferred my account to another supplier it was impossible for my original dual fuel account number and information to be reinstated because only the one fuel type had been switched.  This is despite a regulatory requirement for them to do so.

I would be very interested to hear which testing techniques were employed by the suppliers and which test types they used.  I suspect there were a lot of automation and manual scripts with very little in the way of exploratory testing carried out.

To me this emphasises the importance of elements of process auditing to testing.  As an ISO 9001 auditor I have to examine the business processes my employers have in place.  Testers need to think about the wider processes of which the software they are testing is just a part.

The Mobile Phone Company

I have experienced the opposite effect to James with my present mobile phone operator.  Far from being able to have an account with multiple telephone numbers and/or services, I have to set up a separate on-line account for each number/service.

Worse still the website for the company concerned is only fully functional in Internet Explorer.  There is a help forum which does not work at all in Firefox; one of the most commonly used web browsers.

Registering numbers/services on your account involves:

  • Entering your details on the website
  • Submitting the form
  • Waiting for a text message to come through to the telephone
  • Entering the code into the website – whilst hoping the session and/or code has not timed out in the meantime.

Once this has been done, the number/service can be administered online.

The other day my father purchased a new telephone from this company and tried to go through the registration process.  He was unfamiliar with the handset and is not adept at text messaging.  This meant he found the whole process time-consuming and frustrating and in the end called on me for assistance.

I can understand that the company wants to make sure the telephone really does belong to the account holder but there must be better ways of achieving the same effect.  Also, does it really matter that much to them?

I am sure this whole process works as designed – it sounds like a wonderfully technical solution to the desire to have users register their telephones but, again, a bit of process auditing logic could have been applied to say “this is giving a bad user experience”.