A tale of two uncertainty principles

Watts S. Humphrey formulated a Requirements Uncertainty Principle as follows: For a new software system, the requirements will not be completely known until after the users have used it.

When I first came across this idea, it reminded me of another famous uncertainty principle; Heisenberg's. Superficial research revealed that Humphrey was also a physicist (besides being an important thought leader in software engineering) and it occurred to me that he had not christened his principle randomly.

Heisenberg's uncertainty principle, a cornerstone of quantum theory, states that it is impossible to measure both the position and the speed of a quantum particle at the same time, if you measure the speed you cannot accurately determine its position simultaneously and vice versa.

This apparent anomaly, and others like it, have served to make quantum theory, or at least its philosophical interpretation(s), a domain of heated, at times even hostile, debate. Uncertainty makes people, including scientists, uncomfortable. Einstein is famously misquoted as saying “God doesn't play dice” in a discussion of quantum theory. Einstein was a determinist, which means (v. roughly) that he assumed the theoretical possibility of measuring the universe well enough and thoroughly enough to be able to predict future states accurately.

There were others, including Heisenberg himself, who employed the uncertainty principle to confer the quantum scale with almost magical properties; this has become known as the Copenhagen interpretation. This interpretation leads for instance to the many-worlds problem or the more famous problem of Schrödinger’s Cat, whereby a cat exists, dead and alive at the same moment until 'observed'. The act of observing the cat will somehow send information backwards in time, determining the outcome of an event in the past, or, more fantastically, determine which of the up to the moment of observation simultaneously existing cats will sublimate, “collapsing the wave function of the alive-dead cat” to state it correctly.

Personally, I'm with Popper: the 'mystery' of quantum effects is one of measurement. Even if every measurement were theoretically possible, it would remain theoretically impossible to measure everything objectively; the universe is indeterminate. But the alive-dead cat exists only as a mathematical construct, as a statement of probabilities (propensities as Popper called them), not as an actual array of separate physical states waiting to be resolved by some act of observation. (Although there has been some progress made recently in trying to achieve superimposed quantum states with photon pairs.)

Humphrey's requirements uncertainty principle has also generated much debate, some of it heated. How would the scientists above have reacted to Humphrey's insight?

If Einstein were a developer today he might insist that it should be possible to describe a software system fully before it is built, if the requirements gathering were just done properly. He might even be willing to accept the practical (if not theoretical) impossibility of collecting perfect requirements up front but still feel that aspiring to that theoretical goal was the best way to improve.

The Copenhagen developer, maybe Heisenberg himself, would be all for scrum or some kind of agile approach as a reaction to the requirements uncertainty principle; believing that he could then dispense with estimation, planning, meetings, documentation and all those other inconvenient barriers to his creative flow. Success would be random.

Had Popper been a developer he would probably have realised that Humphrey's requirements uncertainty principle could act as his guide in his search for great software. He would work in such a way that the customer would get to see, and use, working software at as early a juncture as possible and at regular intervals thereafter. Developer and customer would together discover the requirements as the system grew. In so doing they would embrace change (uncertainty's first cousin), and maximise learning and value on the way to successful product release.

---o---

Some eighty years after the publication of Heisenberg's uncertainty principle, quantum theory continues to provoke and engage our greatest minds, and it may yet prove to be the greatest theory of them all, capable of modelling physical existence from the quantum to the galactic scales; a so-called Grand Unified Theory.

The birth of a workshop on (about) 15 sheets

While deliberating on how I might put some flesh on the bones of the idea of developing an agile coaching service incrementally - workshop by workshop - I ran into an old friend of mine, Pieter, who was having trouble with his (complex because distributed) scrum implementation at AerData. They had started well but had stalled slightly as the workload increased and the technical landscape had gotten more complex. We quickly realised that we might be able to help each other out - Pieter's team might benefit from some hands on training in particular aspects of scrum and we (Qualogy) would certainly benefit from being able to deliver a prototype workshop in a friendly environment. We agreed to give it a go. We set up a meeting such that we could discuss with those involved (the team) what might benefit them most. Wouter and I used  a questionnaire like the one displayed here (derived form a larger questionnaire of ours for assessing scrum implementations) to guide our information gathering during that meeting. Our discussion led us to collectively conclude that AerData would be best served by a workshop on user stories, and specifically on user story splitting (techniques). We agreed to make the workshop as interactive as possible.
	Back at the office, the ideas for the workshop started to flow
We thought about the essential topics
And generated a backlog
We worked on the slides, and worked on them, and worked on them...
	We searched for the ideal configuration of the workshop...
(splitting) sprint duration etc
	We tested whether or not we could expect to cover most of the splitting techniques against the sample story AerData had sent us. (As it turned out we ended up using a different story in the workshop itself; that's agility for you!)
And came up with this agenda (whereby the various splitting workshop slots would comprise, in total, 10 sprints of 15 minutes each)
	And suddenly, the day itself was upon us! Things started badly and within a half an hour we were in danger of losing our audience completely. The happiness metric confirmed our suspicions as we went into our first break; we re-organised the agenda, cutting the duration of the remaining broadcast and re-scheduling it such that we would have at least 2 interactive sessions before the next (and final) broadcast...
Things definitely got better as the day progressed. Especially satisfying were the few moments during the hands-on activities when people clearly clicked to some concept or other...
	Like the power of acceptance criteria (also as splitting technique)
We finished up the day with the perfection game. Despite our poor start everybody seemed quite pleased with the ground we had covered and how we had gone about it, we had responded well to change!
Our own conclusions in a nutshell:	Workshop structure was too waterfall! We will strive in future for strict adherence to a cycle of broadcast/interactive, broadcast/interactive. Broadcasts must be (tightly) limited in length and the ratio of sprint length to number of sprints should favour sprint length (i.e. 5 sprints of 30 minutes is more useful than 10 sprints of 15 minutes)

Under construction (user story workshop)

Hasty (and not so very random) post this week as Wouter and I are delivering our first ever workshop in a little over 30 hours from now; a one-day workshop on (splitting) user stories. If Jack were truly nimble he might relate "the making of" here, which would go something like; training with the master (scrum the scrum with a dash of serendipity), the birth of a product backlog, some friendly barter, the unforgiving timebox, the pain of pairing/the joy of pairing, the unforgiving timebox, the difficulties of testing, the happiness metric, you get the picture... In the mean time, I hope you like the handout that we have developed for iteration 1 of our user story workshop:

Keeping entropy at bay (How scrum deals with Murphy)

The 2^nd law of thermodynamics states that entropy (disorder) will tend to increase over time in any closed system. This is not only a law of chemistry but of physics and information too, and we know it intuitively; nothing lasts forever.

This does however present some problems, predicting as it would seem to, the heat death of the universe, and flying in the face of the highly ordered state of life on earth. Luckily we don't have to answer any of those great questions in this humble blog, although I will come back to the subject.

What is a closed system? Well, in effect there is no such thing (except perhaps the universe itself), and at the same time we can treat pretty much any identifiable system as approaching a closed or isolated system. Take yourself dear reader; you are a highly complex (low entropy) system, identifiable as distinct from your environment (relatively high entropy), and yet dependant on it to stay ordered. Should you be truly isolated from your surroundings (unable to extract and consume energy or dispose of waste), your system boundary, so to speak, would quickly fail. Otherwise, as you are a system approaching isolation, time will do the trick, eventually. This is also true of software projects.

Of course, disorder can also arise spontaneously, as Murphy said; anything that can go wrong probably will. The more leeway you give Murphy the greater his opportunity to strike. It's in that leeway and how it's organised that scrum, and agile approaches generally, differ from the waterfall model. The waterfall tries to keep Murphy out, scrum (with XP) builds feedback in at every level, so that Murphy is kept on a short leash. This neutralises damage early while maximising learning. A scrum project, staffed by cross-functional teams, is also enmeshed in its environment, minimising isolation thereby resisting increases in entropy.

The waterfall invites entropy by the very means it employs to combat it. Each phase of a waterfall project is both long lived and isolated. That means that the requirements documents have necessarily 'decayed' somewhat by the time the requirements gathering phase is completed. Pushing the requirements through the contract-negotiation pipe to the design phase will also introduce turbulence, raising entropy some more. By the time the design phase is nearing completion the requirement set will be a veritable paradise for Murphy. When the development phase finally starts, isolation is reaching its zenith, the programmers don't know who the client is and entropy is spiralling out of control. Many waterfall projects end in heat death.

Speaking of heat death, here's my two cents on the great questions mentioned above - if we assume there is no such thing as a (truly) closed system, there is no problem of the 2^nd law; if there are no closed systems, the universe must be infinite in time and space, an unending soup of boiling roiling big bangs and even bigger crunches. A seething mass of bangs within bangs within bangs, bumping into and off of other bangs, each bang spawning its own black holes, which accrue galaxies over time, which collapse into ever more massive black holes, which might eventually become crunches, every black hole a potential seed universe, and so on...

Why scrum works (too)

Scrum is an astutely chosen set of boundaries (constraints; time-boxes, rules etc) imposed on the behaviour of a complex adaptive system (team) catalysed by the probe of certain types of information (user stories, feedback), in short, scrum works because it is empirical and adaptive. But scrum also works because it makes people happy! 

I can feel you squirming dear reader and I understand it. The Scrum Alliance claims that it is “transforming the world of work”, Henrik Kniberg, scrum guru, has introduced something called the happiness metric. What is going on? Management fads are annoying enough but when promulgated by evangelising purists proselytising the way to happiness, they're insupportable! And yet...

According to Dan Pink, it's not money that motivates, it's purpose, autonomy and mastery. Scrum delivers of itself autonomy and the opportunity for mastery, and as Meat Loaf said, two out of three ain't bad. Now, I'm as sceptical as the next man, especially when Pink requires that people be paid well enough to take the “issue of money off the table”. Then again, how many of us interested in transforming our world of work actually have any influence over how (other) people get paid? I certainly didn't when I floated the idea of 'going agile' in the sadly toxic environment I was working in a few years ago:

I had ended up in a typical matrix organisation running the waterfall (telco; IT built & maintained enterprise fulfilment system), complete with silos and gates and all that other waste. Third party integrators were in-sourced to deliver large scale change, maintenance updates were delivered by the in-house development team. Hand-off was built into every facet of this company's processes, contract negotiation was ingrained in its culture. Being a young and ambitious company however we also suffered from wishful thinking which meant that people were permanently disappointed, frustrated in their ambition by their own way of working.

Of course software delivery (I started there as the project manager responsible for the maintenance updates) was a nightmare. On hearing some of my colleagues one day bemoan yet another delay in the delivery of the requirements for the “next generation” system being built at the time, I suggested to them that they consider delivering the system in increments, at least to the acceptance environment. This reasonably conservative suggestion inspired vigorous rebuttal; agile methodologies (which I hadn't directly mentioned) might be okay for building GUI-centric client applications, but business critical enterprise systems were another matter, and innovation was not simple change management anyhow.

Despite this cool reception I made up my mind at that moment to pursue a transition to scrum. Shortly thereafter a major budgetary crisis conveniently paved the way for me to make a start; a cash flow problem meant that all external contracts were terminated forthwith and all open projects were put on hold. Some of those projects were however regarded as essential so after a short period of re-orientation the maintenance development team was asked to pick up the further expansion of the next gen system.

This was no sinecure, the development organisation (developers, testers, analysts) had suffered serious turnover in the months prior to these events and had undergone a major re-organisation. We were in many regards a new team. We did not know the system and it was being expanded for the offering of new services based on state-of-the-art network technology – domain knowledge was also at a premium. Also, some of the tooling in use in the next generation system was unfamiliar to the team (e.g. business rule engine). As is to be expected in such a situation, the new team also had strongly held opinions as to the architecture of the system they were assuming responsibility for – it was decided that we needed to include an architectural overhaul in any expansion of the system. And, of course, the new services mentioned were to be ready for sale within six months; there was a deadline before we had ever seen any requirements or delivered our first estimate! Oh, and the maintenance releases need to continue as before.

The advantage to all this should have been how easy it was to convince everybody involved that an agile approach was the only one with any hope of success. The environment in which we were operating however was toxic, trust was spread thin on the ground and the blame game seemed to be everybody's favourite past-time.

The business could live with some kind of phasing (levels of automation linked to product maturation) but our own architects said the system would have no meaning if not delivered in its entirety. Some of our stakeholders were willing to work closely with the development team but not to prioritise their wishes in a given phase (using MoSCoW for instance), they 'knew IT' and not getting something immediately meant never getting it. All involved, including most of the managers in IT, regarded estimates as commitments, despite regular warnings to the contrary. The team members themselves had invested quite a lot of effort in becoming specialists and felt threatened by the implication of generalisation inherent in agile methodologies, not only were testers loath to help analysts for instance, database programmers didn't want to know anything about Java and vice versa.

The development team lead was one of the colleagues with whom I had had the original conversation on iterations, he was process owner and saw the waterfall process with all it's in-built controls and sign-offs as the only thing shielding him and his team from total chaos. “And anyway”, as he said to me one day in all seriousness, “everybody knows that the project schedule is impossible, that's how we do things here, otherwise we would never get any budget”. This was our starting point.

Over the next 15 months or so we formed as a team and then stormed, stormed and stormed some more. The transition, largely because it was officially not happening, was painful in the extreme. For a significant part of the journey we had the worst of both worlds as 'agile' was given the blame for everything that went wrong while simultaneously being seized upon to excuse all sorts of sloppy practice. This was all taking place while the company itself got into ever deeper problems, external contractors were suddenly let go on a number of occasions along the way, we were put up for sale and the mother company had parachuted in a hostile CEO to effect the transaction.

Despite the tumult, we delivered three major releases in the time the business was used to receiving one (without missing a single maintenance update along the way). 'Simply' by reducing the size of the updates to our system while actively challenging the silo mentality and unifying the inputs to the team (one product backlog over all projects and change requests) we had improved our ability to deliver business value, in an extremely volatile situation, beyond all recognition.

The formal decision to go 'full scrum' was finally taken. In so doing, it was not the further improvements in transparency and predictability, nor the steadily increasing velocity of the team that impressed most. It was the broad smiles that made people stop and take note. Especially when it was taken into account that the sale of our company had just been effected. Our future was more uncertain than ever and yet the team was clearly happier (and more productive) than at any stage in the previous three years! 

Be happy, scrummify!