Saturday, 21 May 2011

Which project metrics do you use?

My favourite software project metrics:

The main two that I use at a high level are end to end cycle time, and benefit hit rate.

Business Benefit Metrics

End to End Cycle Time
the length of time from the conceptualisation of an idea, or feature to the reliable use of that feature by the end customers.
ROI cycle time
The length of time from conceptualisation of a feature to when the feature pays for itself, expressed as a rolling average
Benefit Hit Rate
As a percentage, how many of the features that start being developed go on to return more to the business than they cost.
Number of man days not spent producing the product, ie waiting for a build, waiting for a release, merging branches, waiting for broken builds, waiting for test runs to complete, waiting for hardware, attending meetings etc

Developer Productivity Metrics

Commit cycle time
Length of time that it takes a developer from completing a feature to receiving confirmation that it has been accepted.
Defect rate
Number of defects per feature that was accepted as complete, expressed as a rolling average or over a period of time.
Build time
Length of time that it takes to run the build
Commit test execution time
How long it takes to run the commit tests
Acceptance test execution time
How long it takes to run the acceptance tests
Automated test quality
When a test fails, is the fix to the system code or to the test? Represented as a percentage: num fixes to system code/num failed tests

User Experience Metrics

Use case duration
How long does it take an average user to perform a single goal
Use case step count
The number of user actions required to attain their goal
User frustration
Number of user mistakes + number of system errors*10
UI Responsiveness
Length of time from a user click and a system reponse

Live System Metrics

The length of that the system has been up without a user visible outage
Call out count
Number of times that a 'fatal' error has occurred requiring immediate support staff to take action, even if in the middle of the night
Error count
The number of errors reported in the log files
Warning count
The number of warning messages reported in the log files
Support overhead
Number of man hours required to keep the lights on
Release duration
How long does it take to perform a release, in man hours

Code Metrics

Unit test coverage
Percentage of lines of code tested by the unit tests
Acceptance test coverage
Percentage of lines of code tested by the acceptance tests
Average class Cyclometric Complexity
Indicator of code complexity, the higher the value the more complex the code base is to understand
Worst method Cyclometric Complexity
Indicator of code complexity, the higher the value the more complex the code base is to understand

Refactoring is not the same as rearchitecting

Refactoring is a small scale technique, a collection of small transformations that when applied with skill to a code base helps to improve the readability of the code without changing its behaviour. Unfortunately it has become all to common to use the term with management to hide a project rearchitecturing exercise or rewrite. Such work may be required, however the purpose of refactoring is as a low cost tool to use daily as part of ones programming practice; which helps avoid entropy in the first place.

For more information see:
High Cyclic complexity (say over 80) is like handing in the brain dump of an essay as the final essay.

Sunday, 15 May 2011

Software is not built. It is written.

This post compares the work of traditional engineers with that of software engineers, understanding where they are similar and where they are different helps me to understand the software development process. This post is a treatise on how to make waterfall work, and why Agile is here to stay.

"An author would not expect the first draft of their book to be published. Few if any publishers would accept the first draft of a book. So why then, is it so common for software projects to release first drafts of their programs?"

Summary of the problem

When first drafts of Software is released intentionally then the company making this decision will be more aware of its consequences, and so in a better position to manage the consequences than a company that is unaware that the product that they are releasing is effectively the first prototype of an engineering design. Usually the consequences of this will include the costs of stabilising the product and long delays, sometimes creating a bottomless pit of expense just to keep the system turned on; and it only follows that the cost of adding more features to such a system will be equally large and over time they will increase.

It is a large step to claim that a lot of projects release drafts of software, however it is common practice for the first design to a solution to be implemented and for the first implementation to be passed to testers; there is then often strong resistance to revise the solution to improve its behaviour and maintainability. The refactoring movement and the notion of code debt has helped to raise awareness. However it is still common for there to be a mismatch between senior management and developers in what state they believe the software product to be in. How can it be so common for smart people to release first drafts of a design without realising that it is the first draft? How effectively can such a mistake be managed if the senior management are not aware of the situation? In my experience the root cause of this mistake has often been a misunderstanding of what a Software Engineer actually does and thus as a consequence a misunderstanding in what they need in order to effectively coordinate them to deliver a successful project now AND to position said project ready for the success of the next project.

The building metaphor

I wonder whether part of the problem, part of the reason why some projects mistake a first draft as the final product is the overuse of the 'building' metaphor that is commonly used to describe software development. A weakness in this metaphor that software is built, is the comparison of the skills needed to manage a labourer involved in a task that is approximately deterministic and that of a non-deterministic task. Building sites, factories, any kind of manufacturing or building process is designed around managing tasks that are, to within some tolerance, deterministic. Consider a skilled builder who has perfected laying one brick after another with a small enough error rate that quality assurance is enough to ensure that the house will not fall down. Does it follow that the same process can be used for the author of a book? Or an Engineer designing a bridge? Ask two Engineers to design a bridge, given the same spec and you will have two different bridges. Ask two brick layers to build two walls to a spec, and you will have approximately two identical walls. Both skill sets call upon different management skills to manage; which of these examples is most like software development? A misunderstanding here will result in the wrong management tools being brought to bare.

Is the act of writing software deterministic, or is it non-deterministic? Ask two programmers to implement a timesheet program, given the same spec I guarantee to you that you will get two very different programs that will be different in just about every way that one can think to measure them. And I am not talking about the small tolerance introduced by the nature of the human hand, as in the brick laying example. I am talking about fundamental differences closer to the example of two different bridges described earlier.

In order to make this difference clearer lets take a look at the core of what a Software Engineer does, without using metaphors the very least that every programmer has to do is to write in a programming language (Java, C#, JavaScript, C, SQL etc) instructions to a computer telling it what to do, how and when. Knowledge of what instructions to give to the computer is a problem solving, design and modelling task, mapping that to one or more written languages is a translation and communication task.

The influence of Taylorism on software projects

So where is the building metaphor in this description of programming? In the 1980's there was every attempt to bring Taylorism to software development, in the same way that Taylorism removed the non-deterministic nature inherit in the craftsmanship of trades like wood and metal working on production lines, people attempted to remove it from programming too. Or more precisely the attempt was to move the non-deterministic nature of the crafts from the fabrication stage and to move it to the design stage. Thus allowing different skills, risk processes and management techniques to be brought to bare. This is the basis of the 'Big Upfront Design' project. Big Upfront Design (BUD) endeavours to raise the confidence of when a new software product will ship, and to give the clients signing the check more confidence on what they are going to receive. The rationale is that this confidence can be afforded because the uncertainties will have been ironed out during the design stage and the build stage can be reduced to an almost mechanical process.

If you have worked for a company that has employed the BUD approach, then you will have probably also witnessed a great amount of frustration as the 'build' stage never goes as smoothly as planned. The reason for this is that the design stage was not followed rigorously enough to remove all uncertainty from the 'build' stage. The rest of this post explores why the goal of separating all design from the build stage, as has been done in other engineering disciplines will never work for software development. And a BUD effort that is not aware of the difference, and expects to gain the benefits as seen in traditional engineering disciplines will, on complex projects fail before the project has even started.

Where Software Engineers have an advantage

In more traditional engineering the cost of manufacturing is extremely high, just the tooling costs alone can be significant; think bridges, cars, circuit boards and houses. The task of the traditional engineer is to design a physical solution to a problem and to then communicate how to build the thing that they have designed so a separate group of people can take those designs and fabricate them. The fabrication stage is very expensive, so a great amount of rigour is used to ensure that the plans are as accurate, expressive and correct as possible before building begins. The expectation of many software BUD efforts is to reach this level of maturity, but in order to reach this level engineering companies have to spend a lot of time and money in the design and verification of design stages before moving on to fabrication. This makes a lot of sense for engineering companies, the fabrication/build stage is extremely expensive. In fact just tooling up for production can be mega money, so they need to know that they are buying the right tools and that by following the plans to the letter that they will always build the same product. QA is then needed to control the last statistical variance of production seen in the real world. In order for BUD software projects to gain the same level of confidence as other engineering efforts, then they too must spend extra time verifying their designs with the last stage of verification is building the end design at least once before being confident to commit to tooling up for the mass production of the design. However how many software projects take verification to that extreme before committing to a build?

The idea of not committing to the build stage of a software product, until the designs have been verified by building it at least once probably sounds mad. It may be common when designing an aircraft, but software is not an aircraft. And I agree with you. Software products have a significant advantage over mechanical products, software products are extremely cheap to duplicate. There is no need to tool up for mass production within a software project in the same way as physical products. Mass production is not cheap and the great news is that software does not need it. This is a significant advantage for software projects. The reason for traditional engineers needing to use mathematical simulations, models, scaled down builds etc before committing to building the first prototype is that prototypes cost a lot of money and take a long time to build. The more expensive it is to build and test prototypes, the more rigorous the engineering disciplines become in order to verify their designs before committing to any kind of build. Software Engineers can take their designs and build the application within minutes at the cost of a few minutes electricity to drive the computers. Thats it. Reread this paragraph, at its heart is the key to understanding both the engineering side of waterfall and agile methodologies. Now reread it again and think about it.

The misunderstanding

A goal of a BUD in engineering is to make the construction stage predictable, control costs and reduce risks by knowing the outcome of the construction stage before starting. The big benefit here is for mass production. Taking the time to design everything upfront is expensive and time consuming, and because it is design, the final solution is not clear at the start of the process. The expense of BUD is offset against the savings created by reducing the costs of mass production. On a software project the mass production phase is as simple as a file copy, or a compact disk copy or a download off of the internet or the Apple store. As there are no mass production costs with software, where is the savings with which to offset a BUD effort? The cost of the first build? The first prototype? There is no build savings with which to offset the cost of BUD. No mass production stage means that software is often viewed as done when the first prototype is completed1.

Computer languages as the specification language

If a software architect could hand a developer a specification so complete that the programmer would be able to produce exactly what was requested, in the same way that a factory builds a part designed by an engineer then there would be no need for the programmer. Because the specification would be so complete, so unambiguous that a computer could follow the plans itself and build the application. This fact is worth repeating, a software architect cannot fully define the specification of a system to the point that a developer can fully build it with no ambiguity without them implementing the full solution. The software languages used by software experts is the specification language used by the software industry to describe exactly how to build an application. Software experts then use computers to perform the 'build' for us, they read the formal specifications written in languages like C and Java and they compile, verify and link them into machine code. Modern factories are working to fully automate their production lines, the software industry has had that since the fifties.

Realising that software has no build stage, in the traditional sense is extremely empowering. It explains why the industry efforts with Model Driven Design stalled, why the Agile movement gained traction in the nineties and while Agile was first considered only a fad, it explains why agile is still with us in 2011. It explains why the biggest cost savings in the software industry has been realised by improving the expressiveness of the programming languages (the specification languages) and it explains why companies that respond to software project problems (client dissatisfaction, lack of trust, failed projects etc) by strengthening their BUD efforts while at the same time being unprepared for the costs and lengthening in feedback cycles that come with it, will not experience the increase in certainty that they hope for until they have written the solution at least once2. That all important first prototype. This lesson is especially difficult to learn on BUD efforts, as the feedback cycles are so long on such projects. The failures can take a year or more before they are recognised and the first thought is often that the execution of the process was wrong. The blame game begins. The developers did not follow the plan, the developers were not good enough, the tests were sloppy or the architects and management were incompetent. The mindset then becomes that tightening up the BUD process further will solve those problems. They will not, that is not where the problem lies. Nobody was incompetent; we as an industry have been focusing on emulating other engineering disciplines without acknowledging the ways that we are different.

To make waterfall work, a company must be prepared to fully plan a solution to the point where a computer can execute it, they must know where the design of the solution really starts and ends, and accept that the nature of design is to not know what the final outcome will be. It would not need designing if we knew. Once the first prototype has been created then the company can verify it, and then decide their next steps with increased certainty; and not before. The risk is that with a long design phase, there are long periods with no feedback. If a project is cancelled, unless the first draft was completed then it is unlikely that there will be any benefit to gain from the cancelled project. Therefore to reduce project risk, reduce the feedback cycle; do not lengthen it.

In conclusion

Software, unlike materials in the real world is cheap to build and to rework. The cost is in the design, writing and verification phases, and that is the process that needs to be managed. Because of the nature of software, the craftsman who write the programs are part of the design phase and attempts to separate the design and the 'build' phases did not succeed in the eighties and they will never succeed because the build phase in software development has already been automated by relatively cheap machines.

The biggest savings in software development are realised by reducing exposure to risk by reducing feedback cycles, effectively managing labour during the design process, managing the information gained from the shorter feedback cycles, reducing labour costs by automating as much of the non-design work that occurs in order to support the main design work and improvements to the languages used to describe the plans to computers.

1 Given the analogy between releasing the first draft of a book and the first draft of a program it is worth taking a moment to compare how software engineering is and is not like writing a book. A book is usually written by one or two people, in a single language for humans to read, enjoy and learn from and after the publish date it is difficult to change again; it is difficult to see comparisons with engineering here as books written by authors are rarely rigorous instructions on how to build anything. In contrast the source code to a software system is nearly always written in multiple programming languages, by many programmers for computers and other programmers to understand and follow as literal, logical steps; instructions on how to attain a desired effect. And after release software enters a maintenance phase which can last many years and involves many changes by many more people often after the original programmers have left.

And so Software Engineering is no more like writing a novel than it is like building a house, however it is as similar to a Mechanical Engineer writing instructions on how to build their design. A Software Engineer writes instructions on how to generate a desired effect that they have designed. It is this similarity of writing, and the need for clear communication that the opening comparison of this post is designed to draw attention to. Software Engineering is as similar to writing a book as Mechanical Engineering is; and Mechanical Engineering is more like building a house than Software Engineering is. So for me, this calls into question the very premise that software is built and it is that premise and the consequences of it that are the topic of this post.

2 As a side note, once a company has paid with blood to attain the first working prototype the team that created it often disbands. All that knowledge of the problem, design and solution spaces are gone and any team that follows them have to backwards engineer the original intent or just hack it. This is how many companies with frustrating projects literally haemorrhage money.

Thursday, 12 May 2011

So what is Software Development? Art, Craft, Science or Engineering?

What assumptions, preconceptions or heuristics help you as a Software Developer? Both now, and those that guide your learning and exploration of new techniques?

Like many practitioners of my age I have flirted with SWD (Software Development) as an Art form, a Craft, a Science and Engineering practice many times over the years. I was trained at a University as a Software Engineer, by Engineers with a background in either Electrical Engineering, Mathematics, Physics or Computer Science. I have worked with developers who considered themselves to be solely artists and/or craftsman, and I have worked with scientists.

But why care? Does it matter which field Software Development is most like? Each time that I have considered SWD as just one of those forms, I took inspiration from those fields and I found that there was elements of my work that were strengthened, however I have also found that as a whole my work was weakened. At first I did not notice that the whole was weakened, at first I did not have enough breadth of experience to see the effect. Or perhaps as I was learning in the early days I lacked enough skill for there to even be a decline to observe.

However once the observation that my work was weakened as a whole was made, it has lead me to not be so interested in whether SWD is any one discipline, other than SWD is SWD. SWD is a young field dating back only five decades or so; this is young when compared to other fields that date back hundreds or thousands of years. So instead I postulate that SWD has the most to learn by infusing elements from each of these sister fields.

The view of SWD as an infusion of trades is a pragmatic one, I am not sure if it is at the essence of SWD or an acceptance of its young age. Before Science was viewed a Science its practitioners were as much Artist, Craftsman and Engineer as they were Scientist. Perhaps Artistry and Craftsmanship can be viewed as a base that Engineering, and Science are built upon.

As of the time of writing this post I find the guide that I am currently using the most is to view Artistry and Craftsmanship as a base for SWD and Engineering as an extension that helps with the coordination of larger teams of developers.

Put another way, Artistry reminds me that working without putting ones soul into it is work that is not worth pursuing, Graphic Design aids me with understanding how to improve the communication of my software designs both to the end client AND to other developers who follow after me by improving the communication of the code base itself. Craftsmanship keeps me practical and learning by doing (rather than being overly theoretical, which is a tendency of mine) and Engineering keeps the rigour in the process that helps to coordinate cross skilled teams effectively.

Saturday, 7 May 2011

Test tip with Maven Surefire

When starting a new Maven module, I strongly encourage everybody to set the unit tests to execute in parallel.

There are a few benefits to this:
  • most obviously the unit tests run faster on modern multi-cpu boxes
  • it helps to tease out concurrent bugs earlier
  • it helps to enforce good unit testing practices; specifically that there should be no side effects between tests and the order of test execution should not matter
  • enabling this feature later on in a project is possible, but it is nearly always a pain so start it early and you will get its benefits almost for free
I recently introduced this practice to a new team that I am working with and upon enabling it on a recent module whose unit tests were already in a good state it highlighted a concurrency bug in a third party library that was being pulled in that had not yet been detected. Who would have guessed it? Early detection of problems is awesome.

To enable in Maven just add the following runes to you pom.xml file:


Java Wish List

When I was young (oh man I sound old just for saying that), when I was younger (eeee not any better; oh dear). Back when I used to write compilers, and hack the Linux kernel I never thought twice about adding a new language feature to make my job easier. Over the years the amount of effort that requires, usually in tooling of various IDEs and the lack of will for companies that I work for to develop such tools I have fallen out of the habit of asking what language feature would make this job easier (it has instead been replaced with ‘what framework feature or library will make this job easier’).
So as I become increasingly itchy with Java I have found myself once again asking these questions, “what language features do I want?”. I was surprised at how long this list became. I include short one descriptions of the main features that came to mind, I will look to expand on some of them in future posts as short descriptions do not do them justice.
  • Improved GC behaviour where performance does not drop off for big heaps.
  • Software/Hardware Tranactional Memory (STM/HTM) support built into the language, ala Clojure. 
  • Simplify the try-with-resource syntax.  Something like the following should suffice, using the usual curly braces and the stack for scope. 
    • autoclose InputStream in = new FileInputStream("a.txt");
  • Type inference Scala style. This one feature in Scala, in my opinion makes Scala the language to look at these days.
  • A JVM enhancement, support constant pools across an entire jar file and not just a single class. It would reduce the file size greatly.
  • To give extra design intent information to the compiler which would give extra compile time checking, allow the developer to declare that a object is to not leave the stack.  References that escape the stack can cause thread safety problems and so receiving compile time checks when it matters would be a big help.
    • local Context ctx = new Context();
  • Compiler to error when references to this escape a constructor during instantiation. This is a common cause of concurrency bugs and so should just not be allowed.
  • Closures closures closures, functions as objects, everything as objects, objects objects objects. Closures. – steps down off of his soap box –
  • Remove statics. Naturally not achievable for Java, but like Global variables they break many of todays best practice design patterns and they should be put out of their missary. Another good point of Scala ;) .
  • Build immutable and lockable objects into the language syntax
  • Support lazy evaluation of expressions passed as arguments to methods.
  • Build object field and method reflection into the syntax of the language.
  • Clean and elegant closure support. Okay easier said than done, I’ll expand my ten pence worth on how this should look later but as a teaser I don’t like the Groovy or Scala syntax and prefer the style of using anonymous method syntax. It just feels more uniform to me.
  • Reflection can retrieve method parameter names, and even java doc
  • Refreash the internationalisation frameworks and build it more cleanly into the language. I like how Apple have done it in ObjectiveC over the Java resource approach.
  • language eval support
  • simple dynamic sql syntax, Nick Reeves taught me a wonderful syntax for this a decade ago and I would love to see it become main stream.
  • Another theft from Scala, implicit type conversions and operator overloading (okay so thats C++ style, we all borrow from each other :) . So I can write a Radian class and pass it around and get compile time type checking and use of language operators and conversions etc
  • for ( T a : list ) should not throw NPE when list is null
  • Groovy had it right when they added the .? operator for avoiding NPE exceptions when calling a.b().c()
  • Default values for method parameters (geesh is Java the only language without this)
  • Multiple line strings
  • Language support for templates
  • Language support for SQL and XML
  • Add Interner support
  • Add cache framework api built up using the delegate design pattern
  • one keyword property declaration
  • cleaner support for hashcode, equals and tostring..