The Simula Research Laboratory have done a series of academic studies on the extent to which adding irrelevant information to an RFP can affect the estimate that comes back. The effect is scary, estimates being affected by:
-
information about the client's expectations of costs, when these were clearly stated as preliminary figures that could be ignored (projects with less budget were estimated as taking less effort, possibly because of an unconscious belief the client would accept lower-quality work)
-
wording (estimates went up when the RFP was spiked with language associated with more difficult projects).
-
possibility of future work (estimates went down when subjects were led to believe that the client would be impressed by efficiency of developers, and that this might lead to future opportunities)
-
Total amount of information provided (estimates went up when the RFP contained information about other software systems with which the proposed project would NOT have to interact).
M. Jørgensen, and S. Grimstad (2008) How to Avoid Impact from Irrelevant and Misleading Information When Estimating Software Development Effort IEEE Software(May/June):78-83
Jørgensen, and Grimstad recommend asking for use of a set estimation methodology, eliminating irellevant information from the RFP, and trying to get it handled by an experienced estimator.
Jørgensen, and Grimstad have since carried out a "field study" (M. Jørgensen, and S. Grimstad (2009) The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment Submitted to IEEE Transactions on Software Engineering). They contacted a range of software firms asking them to estimate on various projects. The firms were not told that they were part of a research study. They were told that they were just to provide an estimate, for use in validating other estimates. The firms were simply paid for their estimation work. Apart from being ethical, this prevented firms deliberately estimating low in the hope of getting work). Estimates were only considered if drawn up by an approved method and prepared by an experienced person. Unknown to the estimators, Jørgensen, and Grimstad had a trick up their sleeve. To quote their abstract:
"The companies were allocated randomly to either the original requirement specification or a manipulated version of the original requirement specification. The manipulations were as follows:
i) reduced length of requirement specification with no change of content,
ii) information about the low effort spent
on the development of the old system to be replaced,
iii) information about the client’s unrealistic expectations about low cost, and
iv) a restriction of a short development period with start up a few months ahead (which should, rationally speaking, lead to an increase in effort).
All manipulations led to decreased median effort estimates, but only manipulation iv) led to a large, statistically significant decrease."
Rather sad that no. (iv), which could easily lead to under-estimating in circumstances where that would be tragic, was just as bad "in the field" as in a laboratory study. The other manipulations caused an effect, but not as much as expected from earlier laboratory work.
Note that M. Jørgensen, and S. Grimstad had tried hard to divorce effort estimation from any question of costs or the other factors with which effort estimating would usually be entwined. And, because of the design of the study, there was no direct benefit of the estimator deliberately estimating low in order to put in a cheaper bid.
