Usability Notes - by Chris Baker

Introduce new software testers, reveal Goldovsky errors

Its amazing how often new software bugs are reported when you introduce new testers to the testing process. Its quite likely that some of these are not new breakages in the latest software release. What is happening is usually that the new guys:

• read the test script differently. It is very difficult to write unambiguous instructions, and it would be unconventional to test the test script on a lot of testers until any editorial problems were fixed - usually that would be a poor use of resources. Or;
• they read it carefully and take it literally. By contrast, your established testers have long got used to skimming the instructions to remind themselves what to do, but rely mostly on their experience and memory of previous tests. It's the trade-off you make for their faster progress through the tests. It's very difficult to make yourself read a long, boring test script for the umpteenth time as if you'd never seen it before (at least, I find this so). In this case the testers, new and old, are making an interesting kind of error (if it is an error)- a Goldovsky error

This phenomenon of new tester-related bug reports can go on for a long time if the test script is long and complicated or gets updated over time (or both).

Usually this is seen as a nuisance - after some effort, it's discovered that the bugs aren't real things going wrong with the software, they are artifacts of the test case and tester. It's easy to feel that the tester (or the test script author) "should have got it right in the first place". That would be a fair criticism if lots of time and resources were allowed to get the test script exactly right and train up the testers. But how often do you see that? More usually the team  decided on a trade-off (faster preparation for testing in return for lower-quality script) and now someone is moaning about the downside of it. Or, we didn't decide on a trade-off as such, we just didn't allow enough time to prepare the script, and the test-script errors coming through are how we are learning this uncomfortable truth.

You could see Goldovsky errors as a blessing. Just occasionally the new guy discovers something that is important and was overlooked by the specific way things were done before. You could argue that, for best bug discovery, you ought to rotate people on and off the team of testers to take advantage of this. Hmm, I'm not sure. On long projects, or systems being regression tested each release, staff turnover or other change tends to make this happen anyway without having to do it as a matter of policy.

Conversely, training your testers extensively teaches them to use the system just like you do and risks missing out on all the unexpected and creative things people do when they work from instructions, and the discoveries that might come from that.

Anyway, your customers don't have the benefit of a script, probably won't read the instructions and will certainly do a whole pile of things you won't ever think of until you either do usability tests or launch the product and get customer feedback.

The Prioritizing Grid - as a project tool

When deciding requirements for a project, we're often trying to decide priorities between many items. This rapidly becomes difficult to do in your head ("or in your heads" in a meeting). Here's a useful tool, the prioritizing grid.

It originally comes, I think, from Richard N Bolles, author of "What Color is your Parachute" (a manual for job-hunters and career changers). Career counsellor and consultant Beverly Ryle has an interactive version on her website. The original purpose is to rank skills that a person has (e.g. "I am good at writing and also at cooking, particle physics and  kung-fu, but which of these do I like doing best?"). In its original context it is part of an exercise to help identify careers that a person is likely to find they'd enjoy. But the system could be used for prioritizing any kind of things - here is a screenshot of it in use to prioritize features of interest when choosing a car.

The way it works is that you write out the things you want to prioritize in any order, and number them 1-10 (if there should happen to be 10). In the screenshot, the un-prioritized items are listed on the left. Then you work through them in pairs like this:

• Which do I prefer, 1 or 2?
• Which do I prefer, 1 or 3?
• Which do I prefer, 1 or 4?

Once done with number 1, you do the same  with 2:

• Which do I prefer, 2 or 3?
• Which do I prefer , 2 or 4?

and then you move onto number 3:

• Which do I prefer, 3 or 4?
• Which do I prefer, 3 or 5?

...and then with each of the other factors until you reach the last pair.

It sounds cumbersome, but actually does not take at all long, unless you are over-thinking it, or need to define the factors better, or have a lot of items that you don't care about much.

Once you're done, you go through each  factor and you count up the number of times you preferred it to something else. This gives you the rank order.

In the screenshot example I've done that (the website has pairs of radio buttons to indicate your choice, but you could of course do this on a piece of paper....). On the right of the screenshot you see the priority order that this generates. It looks like I want a car that seats 4 passengers, has plenty of boot ("trunk" to US readers) space, is cheap on fuel and hard to steal (family life in Oxford...).  

As with anything else, I'd say "don't let the tool take over". Perhaps you don't really have a preference between a pair, or can't answer until  the items are better defined. Either of those discoveries could be useful - probably more useful than forcing everything artificially into neat priorities in order just to get the worksheet done.

Reviewing your unwritten plan

How often do you join a project and find that the documentation tells you all you want to know? Yep - never works completely for me, either. In "Your Crucial - and Unwritten - Plan"  (Linda Hill & Kent Lineback - Harvard Business Review) the authors give one important reason for this - our written plans are rapidly supplemented or superceded by a  constantly evolving unwritten plan. It' not just that most of us don't make the time to update our written plans at every twist and turn - the unwritten plan is a different beast altogether:

In sum, a written plan covers only those portions of your thinking that are clear, specific, focused, thought-through, and ready to go public as a formal (and often official) document bearing the title "Plan." Unwritten plans consist of your and your group's thoughts — ranging from vague hunches to roughly-written ideas — about the future and how all of you will create it. Formal, written plans are prepared at key points, while unwritten plans are living, dynamic possibilities that constantly change as you learn more from experience and carry on discussions with your people and network.

(From Your Crucial - and Unwritten - Plan  (Linda Hill & Kent Lineback - Harvard Business Review, 10 May 2011)

In that case, I was thinking, a key point is to take time to review and evaluate your own unwritten plan. Writing the darned thing down, while time-consuming, might help - it allows you and others to critique it, and allows one to step back from it. The unwritten plan can readily become one that is obsessed by the next milestone, or fatally discounts a certain risk or suffers from other forms of myopia. But a periodic write-up may not be the whole solution - vague feelings of unease about something can be too nebulous or personal to put into a formal plan (especially if it is to be shared), but can be a really useful indication of trouble ahead. Time out to think about things seems the only way to review the unwritten plan - hard to achieve in today's busy working lives (I used to find my cycle ride to work or a lunchtime walk very useful for this).

 

"Don't stick your oar in" (Mark Radcliffe) - a tip for project managers?

I've been reading "Thank you for the Days" the autobiography of Mark Radcliffe, a radio presenter. His observations on being a  radio producer are interesting to think about for  project managers (and managers more generally). Here's what he says:

"Initially, with the native enthusiasm of relative youth by Radio 1's then standards, I did try to exert some influence on proceedings...Eventually I realized that the bands were going to do what they were going to do, and the seasoned BBC engineers were more than capable of recording it. ...As producer I wasn't responsible for the material. I was just there to make sure everyone was getting on with it, and no bits of BBC equipment got broken or stolen.... It brought home to me one of the most important lessons a producer can learn, that if everything's going smoothly, don't be tempted to stick your oar in just to satisfy your own ego".

The bit about "not being responsible for the material" reflects that the bands Mark Radcliffe was producing were being recorded to appear on the then massively influential John Peel Show  - a significant marketing opportunity at the time. I suppose the point is that if the band preferred to appear drunk (or unusually sober) or to try out a new sound, then that was a risk for them and their label, not for the BBC. Also, rock musicians are not exactly a group noted for their ability to listen to business direction. There are similarities in publishing here - I've come across some authors who have rock star status and so are edited with great caution if the idea of editing them is tenable at all.

By usually, Project Managers are usually accountable to the business for the project (as in, the business expects to take it up with the PM if the material is unsatisfactory). I've seen projects where the programming team's ideas are accepted uncritically on the rock star model, but the success is mixed. Usually, it goes wrong where specialists are making decisions that are outside their specialty. This can be hard to detect - for instance a programming decision might knowingly or unknowingly assume a certain model of customer behaviour, and the programming team may not know enough about the customers to get that right. Or (in some cases) to have the humility to ask Marketing. Of course, in other cases, the exact opposite of that problem occurs.

This means the PM is always having to make a judgment - "am I forwarding an opinion in my role as the person accountable for the delivery of the project, or am I just sticking my oar in". For example, say you are at a meeting discussing the table structure of a database. It seems to me that the PM is entirely in his or her rights to hold out for a structure that will be quicker or cheaper to build but just as good, (or conversely to champion something that is harder work for the programming team to build but will deliver better business benefits). However, like Mark Radcliffe, the PM needs to remember to shut up provided it's clear that  everything is going well.

kanban

Kanban is a progress-monitoring method that was developed in “just in time” manufacturing (especially at the Toyota motor company). It is now being adopted as a project management tool for software projects. It creates a visual display of how the project is progressing, thus allowing progress to be tracked easily and actual or potential bottlenecks to be dealt with. For example, a kanban system might be made from index cards, each representing a particular software feature being worked upon. The cards are pinned to a board, and are moved across the board to represent how the work is proceeding. Kanban is being adopted particularly by projects that use Agile methods, as it goes nicely with the agile manifesto aim of simplifying and reducing processes, tools, documentation, and plans. However, kanban could readily be used in the production-line-like situations that occur in many projects, regardless of the project management methodology the project is using.

I have been studying the explanation of kanban at the PM podcast website and wanted to work through an example of my own to cement my understanding. Download Kanban_v2Aug09 (PDF file).

[Note added 4th August 2009: I have updated my kanban example,making use of helpful criticism and advice from David Anderson - thanks David!]

Thinking about the quick visual display that kanban gives you about a project reminds me of an anecdote I read about the author P.G Wodehouse. Apparently, when he was writing a story he would assess each finished page and stick it to the wall of his room. The pages would go in order around the room, like paintings in an art gallery. If he thought a page was good, he would stick it higher up the wall; if it wasn't yet satisfactory, he would stick it lower down. As work progressed, this of course gave him an easy method of finding pages needing more thought or work. When the whole story was up to the height of the dado rail (i.e. nearly to the top of the wall), it was ready for his publisher.

How to abet poor software effort estimating

The Simula Research Laboratory have done a series of academic studies on the extent to which adding irrelevant information to an RFP can affect the estimate that comes back. The effect is scary, estimates being affected by:

• information about the client's expectations of costs, when these were clearly stated as preliminary figures that could be ignored (projects with less budget were estimated as taking less effort, possibly because of an unconscious belief the client would accept lower-quality work) 

• wording (estimates went up when the RFP was spiked with language associated with more difficult projects).

• possibility of future work (estimates went down when subjects were led to believe that the client would be impressed by efficiency of developers, and that this might lead to future opportunities)

• Total amount of information provided (estimates went up when the RFP contained information about other software systems with which the proposed project would NOT have to interact).

M. Jørgensen, and S. Grimstad (2008) How to Avoid Impact from Irrelevant and Misleading Information When Estimating Software Development Effort IEEE Software(May/June):78-83

Jørgensen, and Grimstad recommend asking for use of a set estimation methodology, eliminating irellevant information from the RFP, and trying to get it handled by an experienced estimator.

Jørgensen, and Grimstad have since carried out a "field study" (M. Jørgensen, and S. Grimstad (2009) The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment Submitted to IEEE Transactions on Software Engineering). They contacted a range of software firms asking them to estimate on various projects. The firms were not told that they were part of a research study. They were told that they were just to provide an estimate, for use in validating other estimates. The firms were simply paid for their estimation work. Apart from being ethical, this prevented firms deliberately estimating low in the hope of getting work). Estimates were only considered if drawn up by an approved method and prepared by an experienced person. Unknown to the estimators, Jørgensen, and Grimstad had a trick up their sleeve. To quote their abstract:

"The companies were allocated randomly to either the original requirement specification or a manipulated version of the original requirement specification. The manipulations were as follows:

i) reduced length of requirement specification with no change of content,

ii) information about the low effort spent

on the development of the old system to be replaced,

iii) information about the client’s unrealistic expectations about low cost, and

iv) a restriction of a short development period with start up a few months ahead (which should, rationally speaking, lead to an increase in effort).

All manipulations led to decreased median effort estimates, but only manipulation iv) led to a large, statistically significant decrease."

Rather sad that no. (iv), which could easily lead to under-estimating in circumstances where that would be tragic, was just as bad "in the field" as in a laboratory study. The other manipulations caused an effect, but not as much as expected from earlier laboratory work.

Note that  M. Jørgensen, and S. Grimstad had tried hard to divorce effort estimation from any question of costs or the other factors with which effort estimating would usually be entwined. And, because of the design of the study, there was no direct benefit of the estimator deliberately estimating low in order to put in a cheaper bid.  

SCRUM, an introduction

Here is a great video introdution to  the "SCRUM" Project management method. The video is by Hamid Shojaee from www.axosoft.com.  It is also well worth checking out John McCaffrey's commentary on the video

Software "Ecosystems"

PC Pitstop have a review of a talk by Steve Sinofsky (A Senior VP of Microsoft) at the October 2008 Microsoft Professional Developer Conference. The article reports Mr. Sinofsky's comments about Windows Vista and the forthcoming Windows 7. I found the following quote interesting:

"Even Microsoft can’t hide or ignore the cold reception that Vista has received. Sinofsky identified a few key things that caused problems. First, the Windows “ecosystem”, the third-party software, hardware, and user training, wasn’t ready for the extensive changes that came in Vista. The driver model changed, which caused lots of hardware headaches at launch. The User Account Control (UAC) feature broke applications and frustrated users who hadn’t seen the behavior in Windows XP."

The idea of an "ecosystem" seems a good one to me. While Windows probably has a bigger ecosystem than just about anything else, its a feature of any software or website these days that it will need to interact with other systems. For example in a website, you want it to:

• be searchable by engines
• provide permanent links for bloggers or other websites,
• provide RSS feeds or email alerts,
• work on a variety of browsers (including those favoured by customers with disabilities),
• Be bookmarkable by de.icio.us, Digg.it etc.

The days are long gone when you can consider your website as your own little island, accessed through the home page.

More on Wireframes

Since composing my last post on wireframes I came across a couple of articles on the subject which reinforce the point of needing to keep wireframes simple - in terms of what they are for as well as how they look.

Sarah Harrison states the problem nicely:

"Standard wireframe documents look so much like a web page layout, we ask viewers to use immense amounts of imagination to divorce that which the wireframe is trying to communicate from what its visual representation is  communicating....he main problem with wireframes is when they try to do too much, serving multiple purposes at the same time. The key, in my opinion, is to decide what the essential purpose is for your wireframe documents. Different purposes might require a different format."

Sarah Harrison: Wireframes: Struggles and Solutions, Part 1

Dan Brown (who goes on to suggest one possible solution) has had this problem:

"The conflict arose after clients had seen the wireframes. The layout, even explicitly caveated, would set their expectations, and they did not appreciate screen designs that strayed too far from them, no matter how carefully crafted. Clients also struggled to talk about information priorities, taxonomies, and functionality. Placing these concepts in a layout made them more accessible, but our conversations were too tactical, and their feedback had more to do with design than with structure."

Dan Brown: Where the wireframes are Special Deliverable #3

To me, these interesting articles reinforce the need to think ahead about the process within which the wireframing sits  That should help to keep the wireframes as a quick, disposable tool to help with the next task in hand - I don't think the one wireframe can cover all the aspects of logic, layout, emphasis and so on without losing the quick-and-cheap benefits that wireframes ought to have.

Wireframes

A wireframe is a simple visual model of a website (or CD-ROM if you want!), It is produced by a very quick and cheap method so that it can be used early in a project, for example during requirements analysis It might be made using PowerPoint, Visio, or hand-drawn pages, or in HTML, Flash or a specialized wireframe software. Wireframes are excellent for discussions within the project team, and can also be used for paper prototyping and simple usability tests. They provide a model of the website that is easier to understand than, say, a flowchart or long descriptive document. Long Documents Send People To Sleep, but even a primitive model of the website can be excellent in flushing out misunderstandings and "unknown unknowns".

I often find that I need to explain why a wireframe makes no attempt at showing the visual design (colours, fonts, graphics). The point is that these will be agreed and added later (probably after some effort and cost, which we are trying to avoid while we hammer out the early ideas and find what is going to work). For the same reason, a wireframe often does not contain much actual website copy. The point of the wireframe is to focus on what key components should be on every page and how the pages should link together (it is often possible to get the wireframe to demonstrate the latter - for example, by hyperlinks between the pages).

It is worth thinking out exactly what you want to do with the wireframe (e.g. is it merely for the project team, or will it be used for usability testing and if so how?). Being able to explain how this very primitive-looking model fits into the design process is very helpful to reassure stakeholders that the final site will not look this bland and primitive!

As I mentioned, there are a number of tools that can be used. Since the idea is to produce something quickly and cheaply that can be easily changed when it has (helpfully) uncovered misunderstandings or problems, I think the key factor is to choose a tool you can use easily. In an interesting article HTML Wireframes and Prototypes: All Gain and No Pain, Julie Stanford makes a case for making a wireframe in HTML (she uses Dreamweaver). Comments on her article give opposing points of view about the pros and cons of using HTML - helpful stuff if you are trying to decide whether to give the HTML approach a try.  If you do want to give it a try, Julie gives a primer on how to prepare a wireframe using Dreamweaver (making a lot of use of HTML tables). As I say, I think the important thing is for someone close to the requirements analysis to be able to build it quickly and easily. So probably the  choice is as much dictated by tool familiarity as by anything else. Using HTML means that you have the option of actually using some of the wireframe code to make the real site - the pros and cons of this get an airing in the comments on Julie's article.

To show that even a very humble wireframe can be useful, check out this example in PowerPoint:

Download Wireframes.ppt

This wireframe is for a (fictitious) publisher’s website – AAD Publishing Limited, publishers of Arts, Architecture and Design books. For demonstration purposes (or because AAD is only a very small  publisher requiring only a simple site) it contains only a few pages. But hyperlinks allow you to move between them in a way that gives a reasonable impression of how the site will work. I would use a wireframe like this for early discussions in the project team, and perhaps for some quick informal usability tests on any sentient human being who doesn't run away too fast (e.g. co-workers, neighbours, spouse). Even a wireframe as primitive as this can be worthwhile to throw up questions about scope, navigation and usability such as:

• •What have the project team assumed the purpose of the site is? And are they right? What have they missed?
• •How can we keep the home page interesting? It may need frequent updating – who will do this and will it be sufficiently easy for them to load copy onto the site?
• •Should the 3 categories of Art, Architecture and Design (as shown in teh current wireframe) be further subdivided?
• •Decisions will be needed about the order in which things appear on the page – e.g. for the books is it latest publications first? Bestsellers first?
• •There is no way of searching the site – is that an omission?
• •How can a customer buy a book he or she sees on the site? Is there an order form to post to us? Do we want to offer online purchase (there is currently no shopping cart in the design – is that an omission)?. Including a shopping cart would of course introduce various issues about how to take and fulfill online orders (potentially a problem for a small publisher). Do we prefer simply to be an affiliate, (e.g. of Amazon), until it is clear how much business can be generated?

Not a bad crop of questions to get out of the half-hour or so it took me to make the wireframe.