Introduce new software testers, reveal Goldovsky errors

Its amazing how often new software bugs are reported when you introduce new testers to the testing process. Its quite likely that some of these are not new breakages in the latest software release. What is happening is usually that the new guys:

• read the test script differently. It is very difficult to write unambiguous instructions, and it would be unconventional to test the test script on a lot of testers until any editorial problems were fixed - usually that would be a poor use of resources. Or;
• they read it carefully and take it literally. By contrast, your established testers have long got used to skimming the instructions to remind themselves what to do, but rely mostly on their experience and memory of previous tests. It's the trade-off you make for their faster progress through the tests. It's very difficult to make yourself read a long, boring test script for the umpteenth time as if you'd never seen it before (at least, I find this so). In this case the testers, new and old, are making an interesting kind of error (if it is an error)- a Goldovsky error

This phenomenon of new tester-related bug reports can go on for a long time if the test script is long and complicated or gets updated over time (or both).

Usually this is seen as a nuisance - after some effort, it's discovered that the bugs aren't real things going wrong with the software, they are artifacts of the test case and tester. It's easy to feel that the tester (or the test script author) "should have got it right in the first place". That would be a fair criticism if lots of time and resources were allowed to get the test script exactly right and train up the testers. But how often do you see that? More usually the team  decided on a trade-off (faster preparation for testing in return for lower-quality script) and now someone is moaning about the downside of it. Or, we didn't decide on a trade-off as such, we just didn't allow enough time to prepare the script, and the test-script errors coming through are how we are learning this uncomfortable truth.

You could see Goldovsky errors as a blessing. Just occasionally the new guy discovers something that is important and was overlooked by the specific way things were done before. You could argue that, for best bug discovery, you ought to rotate people on and off the team of testers to take advantage of this. Hmm, I'm not sure. On long projects, or systems being regression tested each release, staff turnover or other change tends to make this happen anyway without having to do it as a matter of policy.

Conversely, training your testers extensively teaches them to use the system just like you do and risks missing out on all the unexpected and creative things people do when they work from instructions, and the discoveries that might come from that.

Anyway, your customers don't have the benefit of a script, probably won't read the instructions and will certainly do a whole pile of things you won't ever think of until you either do usability tests or launch the product and get customer feedback.

Comments

[Jim]

As someone with over 20 years in software testing, both manual and automated, your next to last paragraph is (in my experience) the most valuable part of this post.

I despise prescriptive test scripts. First, they take the creativity out of things as a tested. And second, they take simple randomness out of the picture. (Just as an example: if your test script is followed to the letter, you always go to http://www.google.com and never to http://google.com or to www.google.com. Both are perfectly legitimate -- probably -- but if you miss testing them because they aren't in the script…)

Testing scripts should be guidelines -- go to a location like X, not go to location X.

And the value of new testers on a project cannot be overstated. A new set of eyes, a new set of hands — they are not to be feared, they are to be rejoiced in because everything is fresh to them. They see the behaviors other have become used to. They can question things that others have become inured to. They see connections present in the whole forest that those who saw the trees grow up cannot.

[Chris Baker]

Thanks for this great comment, Jim. You reminded me that recently I saw this article http://econsultancy.com/uk/blog/10959-are-online-retailers-being-tripped-up-by-postcode-entry about e-commerce sites that could not handle the various ways that customers enter postcodes (postcodes are the UK equivalent to zip codes). When I saw the kinds of problems being found, I immediately suspected that they could have been pre-empted by less prescriptive testing.

The problem with the post codes is exactly the one you've highlighted , Jim. If your test script says "enter postcode EC1M 3JF", that is what the testers do, every time. They always do it in capitals, with a space. You never uncover the problems that ensue when someone enters a capital I instead of that 1, or omits the space, or uses lower case, or tries a different postcode. Or cannot enter a postcode (overseas customer, member of HM armed forces etc.).

By working from the test script you've restricted yourself to a very different test from a usability test, in which you'd get a bunch of folks and instruct them something like "use this website to order some items for yourself."

I believe that tightly written test scripts do have an important place. That's probably uncontroversial, but probably worth elaborating "How do I love test scripts? Let me count the ways")

(1)If the system cannot pass even that test then clearly there is a problem.
(2)Writing the test script is often a good opportunity to clarify aspects of what we agreed about the system, or what we need to do, or whether there's a factor which we overlooked.

(3) knowing that a release is promptly going to go through a well-thought-out test & be thrown back if it fails is a good moral stiffener. It helps promote the right discussions if the deadline is approaching and development is behind and it might be tempting to release "what we could do by the deadline" not "what we undertook to do by this point".

(4)The test script acts as a good reminder to cover the important areas (with the proviso that its limited to those areas which have been recognized as important).

(5) you get some consistency when testers change over, and that the onus is not on the tester to document every aspect of what he or she did.

But I do agree that the error is to make formal, very prescriptive testing the entirety of the testing. That's a temptation because projects like predictability and accountability, and a thorough test script gives you a measure of both, and the appearance of more.

For a long time I've been in favour of additionally allowing some testers or test-users to wander round the system and see what (within limits of reasonable behaviour) they can break. Of course you can then have some interesting conversations with the developers if they then take the position that whatever you've found is "not in scope". As I say, projects like certainty, and realistic "try to use the system and see what happens" testing breaks that down. So some mutual patience and understanding is needed, rather than a relationship based on tight, legalistic reading of the specification. It's also important to have a way to stay sensible about scope - every new finding is not necessarily an important new requirement.

These tensions are worst when the project is a once-and-for-all opportunity to get something done (e.g. very much a waterfall style project and perhaps done to fixed price). People developing in increments (Agile methodology, perhaps, or anything which gives them an ongoing sequence of releases)at least have a built in opportunity to fix the things which PRETTY INEVITABLY no-one will have thought of first time around.

There is a further hazard of formal scripts - in a project I was once involved in, the developer discovered that their subcontractor had written code so tightly to the test script that passing the test script was about the only thing the system could do. The idea that the test script was a set of examples of what the system should do, rather than a specification had been lost somehow (by mis-communication, or perhaps because the subcontractor was unprofessional and was doing the worst job they thought they could get away with - I don't know). The developer re-factored the code in question at their own cost, which was of course the honourable thing to do. But it did naturally cause a delay.

[Best Project Management in the Philippines]

Good article. You have laid your points really well. I agree with you on that.