Why scrum works (too)

Scrum is an astutely chosen set of boundaries (constraints; time-boxes, rules etc) imposed on the behaviour of a complex adaptive system (team) catalysed by the probe of certain types of information (user stories, feedback), in short, scrum works because it is empirical and adaptive. But scrum also works because it makes people happy! 

I can feel you squirming dear reader and I understand it. The Scrum Alliance claims that it is “transforming the world of work”, Henrik Kniberg, scrum guru, has introduced something called the happiness metric. What is going on? Management fads are annoying enough but when promulgated by evangelising purists proselytising the way to happiness, they're insupportable! And yet...

According to Dan Pink, it's not money that motivates, it's purpose, autonomy and mastery. Scrum delivers of itself autonomy and the opportunity for mastery, and as Meat Loaf said, two out of three ain't bad. Now, I'm as sceptical as the next man, especially when Pink requires that people be paid well enough to take the “issue of money off the table”. Then again, how many of us interested in transforming our world of work actually have any influence over how (other) people get paid? I certainly didn't when I floated the idea of 'going agile' in the sadly toxic environment I was working in a few years ago:

I had ended up in a typical matrix organisation running the waterfall (telco; IT built & maintained enterprise fulfilment system), complete with silos and gates and all that other waste. Third party integrators were in-sourced to deliver large scale change, maintenance updates were delivered by the in-house development team. Hand-off was built into every facet of this company's processes, contract negotiation was ingrained in its culture. Being a young and ambitious company however we also suffered from wishful thinking which meant that people were permanently disappointed, frustrated in their ambition by their own way of working.

Of course software delivery (I started there as the project manager responsible for the maintenance updates) was a nightmare. On hearing some of my colleagues one day bemoan yet another delay in the delivery of the requirements for the “next generation” system being built at the time, I suggested to them that they consider delivering the system in increments, at least to the acceptance environment. This reasonably conservative suggestion inspired vigorous rebuttal; agile methodologies (which I hadn't directly mentioned) might be okay for building GUI-centric client applications, but business critical enterprise systems were another matter, and innovation was not simple change management anyhow.

Despite this cool reception I made up my mind at that moment to pursue a transition to scrum. Shortly thereafter a major budgetary crisis conveniently paved the way for me to make a start; a cash flow problem meant that all external contracts were terminated forthwith and all open projects were put on hold. Some of those projects were however regarded as essential so after a short period of re-orientation the maintenance development team was asked to pick up the further expansion of the next gen system.

This was no sinecure, the development organisation (developers, testers, analysts) had suffered serious turnover in the months prior to these events and had undergone a major re-organisation. We were in many regards a new team. We did not know the system and it was being expanded for the offering of new services based on state-of-the-art network technology – domain knowledge was also at a premium. Also, some of the tooling in use in the next generation system was unfamiliar to the team (e.g. business rule engine). As is to be expected in such a situation, the new team also had strongly held opinions as to the architecture of the system they were assuming responsibility for – it was decided that we needed to include an architectural overhaul in any expansion of the system. And, of course, the new services mentioned were to be ready for sale within six months; there was a deadline before we had ever seen any requirements or delivered our first estimate! Oh, and the maintenance releases need to continue as before.

The advantage to all this should have been how easy it was to convince everybody involved that an agile approach was the only one with any hope of success. The environment in which we were operating however was toxic, trust was spread thin on the ground and the blame game seemed to be everybody's favourite past-time.

The business could live with some kind of phasing (levels of automation linked to product maturation) but our own architects said the system would have no meaning if not delivered in its entirety. Some of our stakeholders were willing to work closely with the development team but not to prioritise their wishes in a given phase (using MoSCoW for instance), they 'knew IT' and not getting something immediately meant never getting it. All involved, including most of the managers in IT, regarded estimates as commitments, despite regular warnings to the contrary. The team members themselves had invested quite a lot of effort in becoming specialists and felt threatened by the implication of generalisation inherent in agile methodologies, not only were testers loath to help analysts for instance, database programmers didn't want to know anything about Java and vice versa.

The development team lead was one of the colleagues with whom I had had the original conversation on iterations, he was process owner and saw the waterfall process with all it's in-built controls and sign-offs as the only thing shielding him and his team from total chaos. “And anyway”, as he said to me one day in all seriousness, “everybody knows that the project schedule is impossible, that's how we do things here, otherwise we would never get any budget”. This was our starting point.

Over the next 15 months or so we formed as a team and then stormed, stormed and stormed some more. The transition, largely because it was officially not happening, was painful in the extreme. For a significant part of the journey we had the worst of both worlds as 'agile' was given the blame for everything that went wrong while simultaneously being seized upon to excuse all sorts of sloppy practice. This was all taking place while the company itself got into ever deeper problems, external contractors were suddenly let go on a number of occasions along the way, we were put up for sale and the mother company had parachuted in a hostile CEO to effect the transaction.

Despite the tumult, we delivered three major releases in the time the business was used to receiving one (without missing a single maintenance update along the way). 'Simply' by reducing the size of the updates to our system while actively challenging the silo mentality and unifying the inputs to the team (one product backlog over all projects and change requests) we had improved our ability to deliver business value, in an extremely volatile situation, beyond all recognition.

The formal decision to go 'full scrum' was finally taken. In so doing, it was not the further improvements in transparency and predictability, nor the steadily increasing velocity of the team that impressed most. It was the broad smiles that made people stop and take note. Especially when it was taken into account that the sale of our company had just been effected. Our future was more uncertain than ever and yet the team was clearly happier (and more productive) than at any stage in the previous three years! 

Be happy, scrummify!