Quality, Not Testing
May 24, 2010 When I am asked to review projects, one of the things
that I ask to see is the quality plan. Almost always I am handed a test
plan, and usually the project manager is very proud of the fact that
they have a well-documented test plan. There seems to be a common belief
that you can test quality into a project--or perhaps worse, that
testing is the same as quality.
So I’d like to make this a bit of
a “back to basics” article and just explore quality and testing--and
perhaps help you identify some ways that you can improve the way that
you manage quality on your own projects.
Quality and Testing
Let’s start off by defining what we are talking about here, at least
for the purposes of this article. Google will of course give you more
definitions than you could ever want to deal with, but from my
perspective we are dealing with the following:
Quality – the way that we ensure that standards are achieved
Testing – the measurement of performance against standards
Both
of those use the word “standards”; that’s simply the level that we want
to reach. Consider an exam: If the pass mark is 50 out of 100, then
that’s the standard. If, on the other hand, the pass mark is 75 out of
100, then that is a higher standard. You have delivered quality if you
achieve that standard--in one case, 60 out of 100 is a quality outcome;
in the other it isn’t.
You could argue that you should strive for
100. In fact, the parents of school-age children reading this tell
their kids that regularly. I won’t argue with that, but in our projects
it’s different. Assuming that the standard is appropriate, if we are
significantly exceeding the standard then we are unnecessarily spending
resources (effort, time, money) that could be diverted elsewhere.
But
back to our definitions: Quality is all about the way that we do our
work. It’s is about ensuring that the way we do our projects is likely
to allow us to achieve the standard. Consider another example--we are
driving 100 miles and wish to use no more than 3 gallons of gas for the
journey. Think about how you would achieve that--timing your journey to
avoid rush hour, avoiding heavy acceleration and braking, etc. These are
actions that we plan and then execute in order to help us achieve our
goal (the standard). Testing only comes in at the end--how much gas did
we actually use and did we achieve the standard?
In projects, the
measures are more complex but the principles remain the same. There
will be more criteria that we have standards for--performance, accuracy,
speed, number of users, uptime, etc.--and as a result there will need
to be many more measures taken or tests carried out.
Planning for Quality
Quality standards can come from a number of places--they may be
defined organization wide, they may be part of the contract with the
customer or they may simply be a series of goals that the project team
sets for themselves. Regardless of the source of the standards, you need
to plan how to achieve that quality. Your organization should have done
a lot of the work for you by providing consistent processes through
your PMO that are designed to ensure that the organizational standards
are met. You may need to enhance these for your own project if there are
additional standards that need to be achieved, and there will likely be
project or product-specific standards that need to be achieved,
especially when we deal with IT systems. These project specific
requirements should be readily available to you from your requirements
documentation.
To be successful, these standards need to be
understood and planned for right at the start of the project. You can’t
achieve a quality target of 100,000 concurrent users if your system
design assumed a user base of 1,000. Similarly, you can’t expect to
redesign a system to move from 1,000 users to 100,000 if you only
provide one day for systems redesign.
This is something that we
readily accept in our personal lives: We don’t expect to be able to buy a
custom kitchen for $200 and have it ready in half a day, and yet we
tend to forget it in our work environment. We want high quality but then
don’t budget time or money to achieve it, instead relying on testing to
save us (do you really need to “test” the $200 half-day custom kitchen
to know that it isn’t what you want?).
The Role of Testing
Testing really is nothing more than determining whether the quality
standard has been achieved. In some cases this can be a very
black-and-white quantitative measure--is it fast enough, does it support
sufficient concurrent users, etc. At other times we are dealing more
with shades of gray, like qualitative measures--is the UI intuitive
enough, is the documentation clear enough, etc.
When we test we
are not looking to measure the quality processes that were put in place
by the organization and the project team (that’s a quality audit, and
quite different). Instead we are looking to measure the project output
to see whether the planning that was done around quality--and the
execution of that plan--resulted in a quality product.
Testing needs
to be planned for--we need to establish what it is that’s going to be
tested, how it is going to be tested and what it is going to be tested
against (it’s measured against requirements, but qualitative measures
need some kind of measurement scale).
Similarly, testing should
only focus where there are quality standards. If there is no requirement
for the quality of documentation or training materials, then they
shouldn’t be tested. Can that result in problems reaching customers?
Yes, but if the project hasn’t identified that as a quality target then
it doesn’t matter--we have to assume that the requirements are both
accurate and complete.
I often hear of a product not being
released because the testing didn’t “pass it”, but to me that’s not a
testing decision. Testing merely establishes whether the standard has
been achieved; the decision as to whether a product that doesn’t meet
the standard should be released is a management one. The answer may
sound obvious, but consider the recent Winter Olympics--the weather
meant that some of the courses didn’t meet the standard, but the event
still went ahead because time was the critical factor.
Testing can Improve Quality
Although the two disciplines of quality and testing are very
different, they are also very closely tied together. Testing can
identify a number of trends that may help identify weaknesses in the way
that we try to achieve quality. In a simple example, if we find that
one developer consistently has more bugs in their code than the other
developers, then it may identify a training need for that developer
(although it may simply be a symptom of them producing more code than
anyone else, and the bugs per lines of code may be lower).
Usually
things are much more complex than that, but test results are a vital
input to root-cause analysis that helps identify the cause of failures
to achieve quality (and by extension helps to identify the corrective
action that needs to be taken). These failures are usually related to
processes or training so we can see that testing becomes a vital input
to the continuous improvement process for quality processes.
Conclusion
There’s nothing profound in this article, but I have tried to provide
a reminder to one of the cornerstones of project management. In my
experience it’s something that is commonly forgotten, causing huge
problems on projects. I have seen projects where the bug tracking
database recorded thousands of defects and delayed the project by
months. The simple fact was that quality had been ignored in the
planning and execution of the project and testing had been relied upon
to catch the problems. Testing did catch the issues, but at what cost?
Plan for quality in the first place and you won’t need to spend so much time fixing the things that have gone wrong.
