In his 1979 Pulitzer-prize winning book Gödel, Esher, Bach (2nd edition, Penguin, 1980), Douglas Hofstadter enunciated Hofstadter’s Law, which says: ‘It always takes longer than you think, even if you take into account Hofstader’s Law.’

As many developers know, this could not be more true in the case of software development projects. The ability of software projects to overrun both time scales and budgetary requirements is now so notorious that it hardly bears repetition. Why is it that software projects overrun so often?

Many developers and researchers have addressed this problem, and some factors do appear to be emerging from this soul searching: the importance of clearly specifying requirements is often underestimated; same goes for the technical difficulty involved; often user expectations overestimate what can be achieved in the available time scale; and so on. Most of us are familiar with these explanations. What is it that unifies all of them? The key word that occurs in all these explanations is estimation. Estimation is a crucial (and poorly understood) phase of all but the most trivial software development projects.

Typically, before development begins in earnest, someone has to estimate the physical resources required, the staff involved, the project’s likely duration, the stages at which milestones can realistically be achieved, and so on. Anyone who has tried to provide reliable estimates of these system dynamics knows how incredibly difficult it is to be precise, and how flimsy is the theoretical support for any decisions and predictions we make. If we need to set an upper bound on the execution of a loop, we wouldn’t be so unscientific as to think of a sensible number and then double it to be ‘on the safe side’, but this is just what most developers and their managers find themselves doing when asked for a prediction concerning a software project.

There are several techniques that can be used to try to provide more reliability in software estimation. The most well-known of these, and that used by most project managers (if they use any technique) is called COCOMO (COnstructive COst MOdel). COCOMO is often misused, because it is not carefully fitted to the peculiarities of the local software environment. If misused in this way, the predictions it makes are often far worse than ‘think of a number and double it’. Fortunately, other more recent cost estimation techniques can offer far more accurate predictions that automatically tailor themselves to the local environment, because they base their estimates on past knowledge in just the same way that a good human estimator would do.

There is another problem with the equation itself, which cannot be overcome by tweaking the value of the constants. In order to work out the required effort for a project, we need to know the size of the project. Unfortunately, knowing the size of a project is half the battle. If we knew how much code would be required by an implementation, we could probably provide a reasonable estimate for how long the implementation would take. Even if things were a little more complicated, and we had to predict how long a team of developers would take to produce a certain number of lines of code, we could probably produce a reasonably reliable estimate. When he introduced the COCOMO models, Boehm said that one could only reasonably expect an estimate within 20% of the real value and that we could only expect this 70% of the time. In reality, even this ‘estimate of the power of estimation’ has proved to be optimistic. Empirical investigations of estimates produced by COCOMO and similar formulaic estimation systems show that estimates based on these formulae could, in the worst case, be out by as much as several hundred percent. Not very reassuring when our livelihoods could depend upon estimation precision.

Finally, there is a problem with the way in which software size is measured. Using the number of lines of code as a measure of size is rather arbitrary; 1000 lines of Java will probably go a lot further (that is, will represent more functionality) than 1000 lines of, say, Z80 assembler code. The weakness of code length as a measure of system size has lead to an increasing uptake in ‘function points’ as an alternative measure of system size, but function points are far less rigorously defined than lines of code, which introduces additional problems.

While the COCOMO method has provided developers and project managers with a much-needed handle on the problem of cost estimation, it has required a great deal of ‘tweaking’ to suit particular development environments and, even with this adjustment, it remains a tool for relating effort and duration to project size. Since its introduction in the early eighties, the COCOMO method has remained the principal technique used by project managers to estimate project costs (apart from intuition and luck, that is). While COCOMO is attractively simple, it would be wise to use it only to decide upon best- and worst-case scenarios, rather than to use it to calculate precise predictions of project attributes.

An Artificial Neural Network is a model (usually in software) of the workings of the human brain. It is currently far too demanding to expect an ANN to achieve the power, flexibility, and creativity of human intelligence, though this remains the dream of some enthusiasts. Currently achievable ANNs contain many orders of magnitude fewer components then the human brain. For example, it would take approximately 1% of all the RAM chips in existence to provide sufficient memory capacity to equal that of a single human brain. Although the replication of human intelligence is a far-off goal for the ANN research programme, more modest attempts to model simple brain-like pattern matching in particular domains have been far more successful. This work has demonstrated that ANNs are good at recognising patterns in data, and almost all applications involve some formulation of a problem in terms of pattern matching. Fortunately, many problems, including that of predicting software project attributes, can be reformulated in this way.

The network itself consists of several layers of neurons, each of which takes inputs from other neurons in the network and provides outputs to other neurons. Essentially, each neuron ‘fires’ its output connections if the sum of its input connections rises above some specific ‘threshold value’. The determination of the threshold values and the connectivity of the network of neurons determine the way in which the network will recognise patterns.

Usually, a fixed network configuration is chosen, while the threshold values are determined by ‘training’ the network with a set of well-understood patterns. A typical configuration involves an input layer, an output layer, and one or more ‘hidden layers’, which ultimately feed input through to the output. The input to each neuron in the input layer comes from the outside world. In the case of project estimation the input will be the values of known project attributes. The output from the output layer is the result produced by the neural network. For project estimation, the output will be the predicted values of unknown project attributes. Information propagates through the network from the input neurons to the output neurons, with the threshold values determining the relative significance of different subsets of input combinations.

The network is taught to recognise patterns in data during a training phase. This involves providing the network with input values for which the corresponding output value is known. The desired result is then ‘back-propagated’ from the output nodes through the intermediate nodes to the input nodes. The threshold values are modified to encourage the network to produce the right output for the input supplied. In this way the threshold values come to represent patterns in the data.

Having taught the network to recognise these patterns, it can be used to predict the likely output that will occur for a known input. In the case of project estimation, we can train the network to recognise patterns in past projects and to provide predictions based upon these for future projects.

Unfortunately, it is possible for the network to overemphasise the importance of certain attributes; essentially, the network is seeing false patterns in the data. ANNs are designed to mimic human brains and, just like human brains, the network can produce wildly inaccurate answers, due to arbitrary biases, based on the natural selectivity of past experience. There is also a problem in understanding the behaviour of the network; even if it produces good estimates, we don’t really understand how it does so. In some applications of ANNs this is not an issue, but in the case of project estimation it is often important to know why and how as well as what. For example, we shall want to know not just what the length of a project will be, but why it will last that long and how it could be changed to reduce this length.

When asked to provide an estimate as to how long an implementation will take, or how much time will be required for testing, most of us will use intuition, backed up by previous experience. This exploitation of previous experience is an example of Case-Based Reasoning (CBR). That is, we base future predictions on knowledge of past cases. Recent research by Professor Shepperd’s group at Bournemouth University has created a new automated approach to case-based estimation of project attributes. The approach is very successful in providing accurate estimates because they are based on prior knowledge and are tailored to take account of an organisation’s development environment.

Before the approach can be applied, a database of previous project attributes needs to be created. The estimates will only be as good as the quality of the data collected, and so disciplined and structured data collection is essential. Of course, this is also true of the ANN approach and, if we are to tweak it, of the COCOMO technique too.

Collecting data about projects is often regarded as a bit of a chore. It is disliked by developers because the data often goes unused, or worse, is used in a pejorative way, to evaluate programmer productivity. This is a shame, because without reliable data about previous projects it is hard to learn from them. Fortunately, project estimation using CBR uses prior project data in a different way to other techniques. Using CBR, it is more likely that the data will support the developers’ view of the project than oppose it, because CBR respects and takes account of previous development history.

The approach is essentially a codification of common sense. We try to find the set of projects conducted in the past that most closely resembles the one upon which we are about to embark. The approach can be applied equally well to whole projects or to individual project steps, such as the implementation of a GUI, or a port from one language or platform to another. In this way, the idea of case-based estimation is useful to both developers and to managers. Because the estimates are tailored to the development group concerned, they do not provide impossibly ambitious goals imposed in a top down fashion. Instead they provide realistic estimates of the likely attributes of a project.

Suppose we have collected information about some previous projects and stored this in a simple flat-file database. Suitable data will concern any attribute of the project that can be measured on a numerical scale. At first sight, this will include properties such as the size of source code used, the number of different files created, the volume of documentation, the number of developers, the duration to completion, the number of bugs reported, and so on. This data is readily available as most projects proceed; we simply need to devote (a little) extra time to its collection.

It is possible to record more qualitative project data by scaling ‘enumerable’ project attributes, for example the platform on which development took place, the customer for whom the project was developed, and so on. We can record even more ‘soft’ attributes, by simply trusting expert knowledge to determine parameters such as the maintainability of the system and the readability of its documentation.

To find the set of nearest neighbours we use ‘Euclidean distance’. This is a simple formula for determining how far two points are from one another, based on the famous Pythagorean rule for calculating the length of one side of a triangle in terms of the lengths of the other two sides. The rule generalises to an arbitrary number of dimensions, so it does not matter how many project attributes we have.

In using the nearest neighbour set to determine the estimate for new project attributes we have several choices. The simplest approach would be to take some number of projects, say five, and to estimate the unknown parameters as the mean of the corresponding known parameters for the five neighbours. This technique suffers from two problems.

First, while a neighbour might be in the nearest five, it may still be a long way off relative to the other four. Second, for some particular unknown attribute, some of the remaining attributes may be poor predictors, merely adding confusing noise.

To remedy the first problem we simply need to identify a weighting to apply to previous projects based upon how far they are from the new one. A close project will contribute far more to the estimate than one far away. If we allow weights to be zero, then we can think of the neighbourhood as containing all projects, simplifying the model further.

To remedy the second problem, we need to conduct some refinement of the model. We can ‘jack-knife’ the project set, by taking out a single project. If we ‘throw away’ one of the project’s known attributes, then we can see how good the system is at predicting the ‘unknown’ attribute of the jack-knifed project. By trying all the different subsets of attributes and jack-knifing on each project in turn, we can determine the best predictors for each attribute.

Estimating project attributes remains more of an art than a science. However, like all arts there is room for some training and for firm foundations. The evidence strongly suggests that no ‘off the peg’ estimation technique will ever be found; software projects are too diverse and complex to submit to such a crude approach.

Formulaic approaches to estimation, like the COCOMO technique, can be used to quickly determine ‘ball-park’ figures for project effort and duration, but more sophisticated (and accurate) answers can be obtained from techniques that base their estimates on local historical data. These techniques may provide better estimates, but require that we face up to the chore of collecting project attribute data.

Estimating the attributes of a software project more accurately will be of benefit to both developers and their managers. Only competitors have anything to fear from more accurate cost estimation. The techniques described in this article are not a threat to developers or their managers; at worst, we will be in a better position to say just how unrealistic our project goals are.