Project managers as scrum masters? (Know your attractors!)

Inspired by a flattering response to a couple of tweets of mine at the weekend, which were a condensed paraphrasing of the Cynefin model's approach to complex problems, and keeping in mind @0x17h's (via @RonJeffries) paraphrasing of Arthur C. Clarke, “Any sufficiently advanced catchphrase is indistinguishable from wisdom”, I decided to try and illustrate my statement with an example problem and how it might be dealt with.

The (combined) tweets:

Similarly constrained self-organising systems will tend to organise towards one of a few attractors (patterns), the trick is getting the constraints right & then damping, or encouraging, patterns as they emerge. Know your attractors!

Problem statement

What to do with the project manager when transitioning to scrum?

Constraints

• There are only three roles in scrum; PO, scrum master & team
• Team should plan it's own work for a sprint and meet daily during the sprint to take stock and make any adjustments necessary to stay on track

Project managers often have former lives, meaning that many see a transition to scrum as a natural way for them to return to the work they actually love without losing face, i.e. they become team members. Others are naturally inclined to assume the Product Owner role. Some consider themselves good candidates for scrum master, or are cajoled into it, as the scrum master role is often, erroneously, seen as equivalent to the project management role.

Whether or not a PM will be a good scrum master is a problem in the complex or chaotic domain because, even if it were possible to identify all the variables involved, they are usually fuzzily defined and (therefore) difficult to measure – e.g. scrum master competencies – which means that causality is often discernible only in retrospect, if at all (retrospective coherence).

According to the Cynefin model's probe-sense-respond approach to problems in the complex domain it should be possible to observe the scrum for one of a few known attractors (this is what differentiates action in the chaotic & complex domains – act vs probe – probe implies that you have some idea of the range of possible consequences of your actions) and amplify positive developments and/or dampen undesirable ones appropriately.

For the sake of brevity we will specify two (possibly overly?) simplified attractors and a number of patterns by which you might identify which attractor your situation is headed for, and how you might intervene.

Attractors

• Command-&-Control (alpha male/the way things were)
• Subsidiarity/Distributed cognition (agility; the path you want to follow)

Probe: PM transitioning to Scrum Master

• Pattern: Scrum master as power broker, boss (Attractor – Command-&-Control)
  • Sense
    • Scrum master determines sprint scope (in committee with PO)
    • Scrum master dictates tasking
    • Scrum master assigns tasks
    • Scrum board not a source of truth
    • Scrum master unilaterally decides on action in the event of interruptions
    • Little regard for sustainable pace
    • Estimates as commitments, regular overtime
  • Respond
    • Dampen
  • How
    • Insist that PO prioritises and Team forecasts
    • Insist that scrum teams must organise their own work to be successful
    • Insist that tasks be pulled, during the sprint, and not assigned (pushed) during planning
    • Ensure that only work represented on the scrum board gets done
    • Ensure that interruptions are triaged with team input, and resultant outcomes agreed with the team
    • Ensure that variance in velocity is recognised as normal, coach the team and the PO to create stories small enough (~10 per sprint) such that (occasional) failure to fully meet the sprint forecast should not cause any serious business issue
• Pattern: Ingrained habits and perspectives difficult to displace (Attractor – Command-&-Control)
  • Sense
    • Scrum master plays an active role in task planning
    • Daily stand-ups always convened by scrum master
    • During stand-ups, team members look to scrum master when 'reporting status'
    • Burndown updated by scrum master
    • Burndown 'flatlines' till last day or two of sprint
    • Stakeholders ask scrum master for status updates
    • Scrum master coordinates effort to be able to demo during review
    • Scrum master demonstrates new functionality during review
    • Team communicates with other teams through scrum master
  • Respond
    • Dampen
  • How
    • Scrum master leaves the room while story is being tasked
    • Schedule stand-ups for the same place & time daily, ask the scrum master to hang back until a few team members have approached the board
    • Have team members pass a token during stand-up, whomsoever has the token, has the floor. Ask team members to update the scrum board as they talk. Have scrum master stand laterally to the group.
    • Ask team members to rotate this duty, agree that the last to speak updates the burndown
    • Encourage swarming, introduce explicit WIP limits
    • Ensure scrum board is big, visible and kept current
    • Make it clear that the scrum master's role is to facilitate the team's demo, not run it
    • Introduce subject matter experts and leave the conversation, facilitate setting up communities of practice etc.
• Pattern: Scrum master facilitating & delegating (Attractor – Distributed Cognition)
  • Sense
    • Planning meetings not dominated by scrum master
    • Retrospectives not about the scrum master
    • Impediment list ordered, current & highly visible
    • Team decides on sprint scope
    • Tasking completed by the team
    • Tasks pulled during the sprint
    • Number of open stories less than number of team members
    • Stand-up happens of its own accord, at the same time every day, in front of the scrum board
    • Trust between PO and team evident
    • Communities of practice emerging
  • Respond
    • Amplify
  • How
    • Ensure the scrum master does intervene when necessary (e.g. PO pressuring team)
    • Have scrum master change format of retrospective on occasion to prevent stasis, ensure that PO allows for (a minimum of) one improvement story per sprint
    • Ensure impediments are being cleared (list isn't just getting longer and longer)
    • Ensure the team continues to challenge itself – organise it such that that variance in velocity is recognised as normal, coach the team and the PO to create stories small enough (~10 per sprint) such that (occasional) failure to fully meet the sprint forecast should not cause any serious business issue
    • Challenge tasking outcomes; are we missing anything? etc.
    • Remain alert to power distributions within the team
    • Continue to try and minimise the number of concurrent 'open' stories
    • Have the scrum master be the first to approach the board on occasion
    • Continue actively encouraging transparency
    • Bring people together, don't get in their way

This is anything but an exhaustive list dear reader, and in a real-life situation you will probably be both dampening and amplifying on different tracks simultaneously, but I'm sure you get the picture.

Princely agile

Whether running an agile project or a Prince 2 project, the principles at work are the same, in both cases a project is broken up into small pieces and regular formal inspection is applied to ensure alignment. It is in the detail of the how & why of that decomposition, and the interpretation & exercise of control, that important differences can be found. Another important difference is the level of prescription, Prince 2 attempts to describe the complete set of possible project contexts and states that the method should be pared to fit. Agile methods, like scrum, tend to prescribe very little and insist that context will determine how the method or framework should be extended for any given situation.

Ironically enough the majority of so-called Prince 2 projects are PINO (prince in name only), have few or no (non-project management) products defined and never include stages (which should not be confused with phases), nor the associated formal decision moments on how to proceed, or not. This means that stopping a project often only becomes an option when original deadlines and/or budgets have been thoroughly and spectacularly violated, more often than not without the delivery of any usable product whatsoever. The PID (project initiation document) often remains live deep into the project lifecycle, sometimes being finalised only when the project draws to a close (talk about gaming the numbers!).

Product based planning is a cornerstone of Prince2 and, if applied appropriately, can provide a link to agile methodologies. Appropriate application would involve hi-level specification of products during the Initiation phase, whereafter detail is added incrementally and iteratively, just-in-time and through collaboration with the team. Minimum viable product (or feature) should be the guiding principle in designing a product breakdown structure (PBS) - not process flow like requirements document, analysis document etc.

But if scrum is a framework for project management, why would you need Prince 2 at all?

Well, scrum is indeed sparsely prescriptive and basically says nothing about (financial) governance other than that it is up to the Product Owner. In fact the role of Product Owner is one of the most problematic aspects of scrum. Scrum says that the PO should have vision & authority and be (always) available to the team thereby ensuring maximum value delivery. This is a difficult assignment, and if the stakeholder and/or user landscape is complex enough, it requires nearly superhuman abilities to execute satisfactorily. A sparse version of Prince 2 could provide a solution here. Specifically, a steeringboard could provide a workable escalation channel in priority disputes (as long as it is still employed according to the principle of management by exception, i.e. at the discretion of the PO), and the maintenance of a formal business case might provide an explicit policy for the PO to work from from day to day.

I have used Gojko Adzic's (most excellent!) Specification by Example to create a mapping between Prince 2 phases (always mandatory != stages) and agile execution:

Project Start

• State goals (Brief/Mandate)
• Appoint Steering Board
• Appoint Project Manager or PO
• Assign existing team(s) to the project

Project Initiation

• Collaboratively determine scope (hi-level PBS e.g. in the form of Story Mapping/Initial creation of Product Backlog, consisting necessarily of primarily rough-grained stories)
• Create business case
• Collaboratively elucidate hi-level risks

An initiation phase could easily consist of a small number of sprints, this would ensure a much better understanding of scope & risk, and allow for using actual velocity in provisional (release) planning etc

Project Execution

• Managing stages & stage boundaries: Each iteration is closed with the demonstration of working software, actual progress is tangible and concrete which means that the validity of the business case can be continually reviewed and assessed with confidence. The definition of formal stages to this end may be overkill.
• Work package creation & acceptance (and associated wasteful hand-offs) are replaced by the ongoing collaborative activities of backlog grooming (“refining the specification”) & sprint planning. A project manager (if there is one) therefore has no role in activity planning.

Closing a project

• Lessons learned – continually generated during the project (review & retrospective)
• Product handover – not necessarily applicable if every sprint realises (potentially) shippable product
• Review goals

Et voila – princely scrum!

Of course, there are those who suggest that management generally (Radical Management, Management 3.0) and financial governance specifically (Beyond Budgeting) would also benefit hugely from an agile approach. I agree, princely scrum (if and when contextually appropriate) should be but a step on the road to imperious agility.

The trouble with success

This week the first sprint of the pilot transition at our new client was completed successfully. Kudos to the team, if they learned as much about scrum as I learned about coaching then things will turn out alright!

Before embarking on this transition I had imagined (in my wisdom!) that teams transitioning to scrum (or any flavour of agile) would generally be coming from either chaos or toxic and petrified waterfall, so that anybody in an executive role (as opposed to management or admin) would be only too happy to try something new. 

But what about a start situation wherein success is the problem, so to speak? The team we are working with is part of a young company which has grown so rapidly in the last twelve months that it's quite literally bursting at the seams; space in the office is at an absolute premium.

Initially I thought that this would make the job of coaching a transition to agile that bit easier – especially when it became apparent that the existing processes had evolved such that they included some obviously agile elements already. For example, scope was derived from goals via a series of workshops involving the client, this scoping phase was followed by a definition phase, which included (high level) requirements gathering, and subsequent to that, a cycle of design/development iterations was entered into. Finally there was an acceptance & delivery phase whereby a portion of the team would be on-site at the client. Releases tended to fall into the three to six month range and teams were organised by client/project and not by function.

Sure it would be just be a case of crossing the t's and dotting the i's and they would be equipped to deal with phenomenal growth for years to come. If nothing else, in the past two weeks I've learned that naiveté is a good learning stance!

To start with, there is the 80/20 rule: The last 20% of any job will cost 80% of the effort. To take one example from the list above – a feature team is not yet a feature team simply by its comprising all the skills required to deliver a feature; if responsibilities are still strictly demarcated by role then you retain silos, queues and handoff, and it remains exceedingly difficult to genuinely work as a team according to priority (even with the best will in the world).

So, the first sprint has been completed and all the stories have been delivered, in fact, stories were added during the sprint and still the entire scope was delivered. The burndown flat-lined all the way to the second last day of the sprint however and on that last day there was a sheer drop all the way to zero – a typical indicator that the sprint was in fact a mini-waterfall. This posed a challenge for me in setting up the retrospective. I wanted to ensure that the team enjoyed its success and that the people took the time to recognise their achievement, yet I felt it was imperative to discuss the risks (getting a lot of work done without delivering anything) inherent in the approach that had led to the burndown profile described.

I started by focussing explicitly on the successful delivery and then defined the retrospective goal as generating at least one improvement story (for inclusion in the next sprint backlog) which would help the team along the road to delivering sprint scope story by completed story. As you might expect, resistance to the suggestion that there had been anything 'wrong' was strong. Discussion was energetic and I was repeatedly faced with 'the way things are done here'.

“The way things are done here” is a notoriously difficult notion to challenge at the best of times - scrum is incomplete and adaptive, every implementation of scrum is unique and it is most certainly so that some things are best done in a particular way in a particular situation. That said, much of “the way things are done here” during the early stages of a transition is just general resistance to change.

If most of the people are unhappy and the projects are generally disastrous then “the way things are done here” can be rejected relatively easily as obviously the wrong way to do things. If however your team is motivated and successful such that projects are generally delivered on time (even if stress levels do sky-rocket as delivery nears) dealing with the “way things are done here” requires careful thought, concentrated listening and skilled discussion.

I'm working on it!

The power of play

These days, more a coach than a player, I've been thinking not only about scrum, agile and all the rest but how best to communicate on these topics and how best to generate and/or impart knowledge generally. The power of scrum lies in self-organisation and that complicates the coaching role; even if it were possible, there is no value in telling your team from moment to moment how they should tackle the challenges they will face along the way – you need to get people on the right track and help them develop whatever skills or practices are necessary to stay on that path.

In this regard I am beginning to understand the potential power of play. For instance, Innovation Games can be a powerful tool in getting those mental cogs a-grinding. The sail boat (variation on the speed boat) exercise can be used both for envisioning risks & leverage at the start of a project and as a tool to facilitate retrospectives. Formulating an elevator pitch or designing a product box are also great ways to generate collective focus and get the creative juices flowing.

In developing my coaching skills (and our service generally) I am lucky to be able to interface with a product development group here at our own company; the Qafé team. If I want to try out any ideas that I come across, and the Qafé team and I can identify some reciprocal advantage, then we collaborate accordingly. For instance, having read Derby & Larsen's excellent Agile Retrospectives, I asked the team if I could facilitate their sprint retrospectives for a set period. I gained through practical experience and the team gained through the generally refreshing approach to retrospectives.

We (Wouter and I) also 'tested' our user story (splitting) workshop on the Qafé team. Apart from all the excellent feedback that we got on the workshop itself we also learned that the Qafé team was having a problem relating to potential customers. Strange as it may seem, it is not at all unusual that developers do not explicitly think about who their users might be and what they might expect from the software being built. I decided to run a pared down version of Jonathan Rasmusson's Inception Deck exercise – even though Qafé was long past inception, in fact it is already in production and about to take off in a big way, more of that in a moment – to focus the development team's mind on the why of their daily what.

The inception deck is a technique for envisioning projects which comprises, among other things, several popular innovation games. Some elements of the inception deck were not applicable given the timing of our exercise but in a little under 2 and a half hours we went through Why are we here?, Create an elevator pitch, Design a product box, Show solution and What keeps us up at night? It proved to be a very valuable exercise and thereafter the team was generally much more focussed on who they were building for and why.

Some weeks later the company CEO arrived in the team room with good news and a serious challenge. Several large integrators had voiced an interest in Qafé and had undertaken to invest in some training and, assuming the product matched the potential of the idea behind it, to add Qafé to their list of supported/preferred development frameworks for enterprise systems. “But,” warned the CEO, “realise that this is make or break, if these guys don't like the system, that's it, you won't get another chance and the bad publicity will kill the product before it takes off. How confident are you that the product is ready, that it's stable enough?”.

The answer was a less than resounding “Quite”.

Don't get me wrong; this team is confident that they have a good product, they know that it's well conceived, designed and executed. The problem lies in the limited feedback they have been able to generate up until now – several enterprise applications have been built (by a sister organisation) using Qafé, each one resulting in invaluable feedback in the form of bug reports, feature requests etc. but the team might have been more comfortable with one or two more implementations under their belt before being faced with such a make or break opportunity.

The PO jokingly recalled the inception deck exercise, “Do you still wonder what keeps us up at night?” he asked of me. Suddenly inspired, he instructed the team to think on what would keep them up at night in the specific context of the news they had just heard. At the following planning meeting, with a focus on the release to the big vendors, the team detailed what their greatest concerns were. These were then collated and prioritised. Starting at the top of the list the team translated their concerns into actions and so determined, in cooperation with the PO, what their product backlog should be for the coming period. Sprints were reduced from 2 weeks to 1 and the team has gone about methodically addressing and eliminating the risks identified. 

Confidence has soared.

The virtues (or otherwise) of feedback

Agile methodologies are successful because they are empirical, incremental and iterative; every action is examined and feedback is generated such that improvement is attained by taking small refining steps. Running scrum with XP generates feedback (TDD, pairing) as often as every few minutes!

It is abundantly clear to me why this results in great software better aligned with the customer's needs. However, in a recent conversation on the merits of frequent (structured) feedback I decided to play the devil's advocate and asked if it weren't possible that frequent feedback might sometimes in fact stymie some of the very best minds. This suggestion was met with appropriate derision (yes, it was a conversation among 'the converted').

I wasn't to be put off that easily however, what about Van Gogh I asked? He never sold a single a painting while he was alive, and when he paired with Gauguin it ended in tears. If Van Gogh had heeded the feedback he received on the fruits of his artistic endeavours he would have given up long before he got to Arles and the world would have been a poorer place for it. Art is not science was the rejoinder. There is a science to art though, and an art to science.

What about music then, it's still art but art with an explicitly mathematical character, how do great musicians deal with feedback? Beethoven was stone deaf by the time he wrote his majestic 9^th symphony, a symphony which to this day sets the standard for all other symphonies. When Stravinsky's Sacre du Printemps was first performed some ninety years later there was a riot! The composer (and the choreographer, who was also a touch too modern for the tastes of Paris' beau monde) were nearly run out of town. The press was almost unanimous in its condemnation of this newfangled noise and the weird dancing that went with it. These days, Sacre du Printemps is regarded as one of the masterpieces of twentieth century music. As for Schoenberg...

I wasn't convincing however, art was far too subjective (we hadn't even touched on literature yet), and that's exactly why creative geeks get into software; a bit is either on or off. Feedback is always good. And Schoenberg really is unadulterated noise!

However, new research would seem to indicate that I may not have been entirely misguided in my playful challenging of the virtues of feedback. Creative ideas tend to generate resistance, even among people explicitly seeking innovative breakthroughs, and regardless of objective evidence supporting the new idea!

So, when is negative feedback a sign that you need to change your ways and when is it a sign that you are most definitely on the right track? There is no simple answer to that question. As a rule I would err on the side of responding to feedback at face value but try and allow for the occasions when feedback says one thing but means another. In fact, this is a specific instance of a generic problem with all process guidelines, and a topic of hot debate in the scrum community;

• When does a product owner challenging a team become a manager pressuring the team?
• When does guarding the process become procedural nit-picking?
• When does positive conflict become damaging mudslinging?
• When does the avant garde represent the emperor’s new clothes?
• Etc.

In all cases the answer is (a sometimes seriously unsatisfying), “it depends”. And that is the art to this science of software development.

Fixing to get flexible

When adopting scrum, one of the more subtly difficult issues that can arise, derives from the varying expectations generated by the word agile. Customers who are used to long lead times and cumbersome procedures are delighted at the prospect of 'going agile' – they will get to call the shots and there will be no penalty for change. This lightweight interpretation of agile, inspired by a craving for flexibility, can cause problems when customers decide mid-sprint that they need something new, tomorrow.

When faced with the situation outlined above, the scrum master will of course state that the customer needs to talk to the PO and that, in all likelihood, (some portion of) the new functionality will be delivered no earlier than at the end of the next sprint (which could be up to 4 weeks hence!). The customer's response will often range from disappointment to disillusion; what's so agile about that?

Or, what to do as freshly minted scrum master when the sprint ends and the last story or two (!) on the sprint backlog are not done? The freshly minted PO, who is also still learning, and his customer base are pushing hard for an extension to the sprint – surely adding a day or two to the sprint is better than delaying delivery of the story in question for (at least) another whole sprint; where is the agility in (such) strict time-boxing?

Management might also react negatively when it transpires that agility does not involve people hopping nimbly from one team to the other as the (perceived) need arises. How to ensure efficient use of resources; what's so agile about all this rigidity?

So, how should a scrum master explain to his or her environment that a measure of intransigence is in fact conducive to flexibility?

They might point out that four weeks is still significantly shorter than 12 or 26 weeks but I wouldn't recommend it.

They could delve into the theory of complex adaptive systems and explain how scrum is an astutely chosen set of boundaries (constraints; time-boxes, rules etc) imposed on the behaviour of a complex adaptive system (team) catalysed by the probe of certain types of information (user stories, feedback).

Or they could point out that scrum's basic measure of progress and unit of prediction, velocity, requires fixed sprint length and fixed team composition to have any meaning.

Otherwise they might pontificate on the magic of cadence and appeal to their listeners' sense of (universal) rhythm. Or they could point out that if the team does not have to constantly busy itself with a fluid agenda that they can concentrate on what really matters; delivering working software that the customer actually wants with astonishing regularity.

For the PO, whose time is also usually at a premium, stable sprint configuration means that he or she knows months in advance what possible delivery dates are, but also when they need to be in the office as planning meetings, grooming sessions and reviews take place in the same locale, at the same time & for the same duration and on the same day of the week, sprint in, sprint out.

If all else fails, a story might help; it is often said that Einstein's wardrobe was filled with identical suits because he felt that (mental) energy expended on deciding what to wear was energy wasted. As with much of the myth of Einstein it is difficult to establish the veracity of this anecdote but recent research indicates that decision fatigue is indeed a force to be reckoned with. Developers, scrum masters, POs, line managers and customers will surely all agree that a team's (decision) energy quotient is far better spent on coding questions than on deliberations as to the best time for this week's grooming session.

What scrum is (not) – a meme

Although it's difficult to test or otherwise corroborate in any way, I have the feeling that 'the agile community' has been trending negatively over the last couple of weeks. At the very least there has been much to-do in my personal infosphere on the evils of agile (aficionados). The debate as to what agile, and specifically scrum, is seems to have been revived via a couple of slightly bad tempered posts by Ken Schwaber. Even well-loved fixtures of scrum lore like the chicken and the pig are coming under fire.

A framework that has inspect&adapt as one of its central tenets should obviously itself be under constant review, but what surprises me about some of the negative utterances I have come across recently is the vehemence with which they are delivered. Onediatribe in particular caught my attention, not only because (in a discussion of scrum) it ranged – in a series of three articles - over politics, (social) history and religion (subjects close to my heart), but most especially because of its toxicity. This guy really has a bee in his bonnet. His central metaphor, of priests and followers, is not without its worth though, and the unthinking uncomprehending dogmatists that seemed to have riled him so are definitely not a figment of his imagination – I have met them too. As for the parasitic consultants; I don't doubt they're out there either.

However.

Lambasting scrum as a religion, when it is explicitly constructed on empiricism, is a bit like creationists claiming that the theory of evolution is just one more belief system.

One of the issues that came up in discussions engendered by said article (also re-posted to LinkedIn) was the Scrum Alliance and its certification (specifically the Certified Scrum Master). I do not want to get into a discussion on the Scrum Alliance but I will defend the CSM, at least as delivered by Jeff Sutherland; not that everybody who attends the course comes away a good scrum practitioner but then again not everybody with a university degree in computer science is a good programmer.

Two days is enough time to get through the basics of scrum, scrum is simple and incomplete; that's the beauty of it. More importantly, Sutherland starts by introducing scrum as, above all else, a frame of mind; the way. He then goes on to explain shu ha ri and states that although it is good to follow the rules to the letter initially, gaining understanding is the goal and that once that goal has been attained the rules can be left for what they are; useful guidelines. He also explicitly includes XP and Lean as means to maximising the efficacy of scrum. Scrum as taught by Sutherland is therefore anything but an exclusive collection of dogma and rites. 

Further; the belief that agile development equates to no documentation, no planning and no commitments abounds. I have met teams that claimed to be running scrum when sprints were not fixed-length, team composition was wildly variable, random managers could change the 'sprint backlog' whenever it suited them, and most importantly, delivery of working software was haphazard at best. I have known team leads and scrum masters who 'itched' at the idea that 'their' charges were not all fully allocated – that their team was thus inefficient – and therefore assumed responsibility for sprint planning. With so much confusion out there some attempt at certification is absolutely necessary. If someone is a CSM then you know at least that they have been exposed to a version of scrum that will work if applied as learned.

But the CSM is not enough, Sutherland & Schwaber on the topic: certifications do not guarantee excellence

That is to say; scrum is a meme. And just as with the its physical cousin the gene, it is as good as impossible to make a perfect copy of a meme. Everybody who has ever learned anything about scrum has their very own unique scrum meme, some of which will be instantly recognisable as incomplete or otherwise faulty, some of which will be more difficult to assess, unless you test them. In that regard, it could be a good idea to view certification as a necessary, but not a sufficient, condition for recognising the owner of a good scrum meme. An agile hiring process could be even more useful – new scrum masters are hired initially on a one or two month contract, their performance (basic understanding, facilitation & communication skills, removal of impediments) can then be tested over a small number of sprints whereupon the team can decide whether the new scrum master is the scrum master they have been looking for.

I sincerely hope that the agile community succeeds in keeping its conflicts positive.

Introduction to scrum (A retrospective)

A few weeks ago I wrote about how our first workshop came about and what we had learned from it. As luck would have it, the same day that we ran that first workshop, I received a message from an ex-colleague of mine, Leonie, who was working as a business analyst at Second Floor. This small company is dealing with the difficulties of success – growth is so strong that their existing processes, which have worked well up to now, are beginning to hinder progress. One of the things they were considering in an effort to manage their success, was a transition to scrum (which some teams had already partially implemented) and they were looking for project managers versed in same.

Although I had to disappoint Leonie in as much as I am not for hire as a project manager we quickly agreed that there was potential for mutual benefit in our respective situations; Qualogy could assist Second Floor in a transition to scrum, while Second Floor could help Qualogy through structured feedback on our expanding workshop portfolio.

A meeting was arranged to discuss possibilities. We agreed, with the delivery manager and technical lead at Second Floor, to kick-start possible further cooperation with an afternoon workshop as a general introduction to scrum. If that delivered enough learning and was generally well received we would talk about collaborating for the entire transition. At that point we had already agreed in broad terms on what a transition would look like; it would progress team per team (teams already comprised all the skills required to deliver working software) and would comprise a series of workshops and some targeted coaching.

First things first though; we needed to create a “Scrum basics” workshop that would leave a bunch of very smart, semi-initiated (in terms of scrum and agile) and fully dedicated professionals with an appetite for more. From our previous experience we were very aware of the need to run any workshop as a series of broadcast-interaction cycles, but we only had a half a day for the session and we wanted to ensure that the basics were thoroughly covered. The urge to take the easy way out and create a presentation (broadcast) with a high wow-factor (assuming we could do such a thing) was strong!

Our discussion with the delivery manager and the technical lead at Second Floor had indicated that there were some obvious (and relatively inexpensive) opportunities for process improvement, quick wins if you like. We were considering the possibility of creating hooks in our presentation-to-be for these basic improvement experiments (e.g. keeping sprint length stable) when Wouter made the inspired suggestion of setting the workshop up along the lines of a retrospective. The fact remained that it would not be trivial to involve our audience actively enough while ensuring that we covered the basics, but a retrospective framework (we used Derby & Larsen's Set the Scene-Data Gathering-Generating Insights-Decide What to Do-Wrap) provided an excellent starting point for attempting just that.

We set the scene in a half an hour by introducing ourselves, relating how the workshop had come about and how things might develop thereafter. We explained how the agenda had been set up, how we intended to make the material covered as relevant as possible to Second Floor, and how we wanted to garner feedback throughout the afternoon. This was followed by a short broadcast including definitions and an extremely brief history of both scrum and agile.

To gather data, we used a (familiar) brainstorm; we asked everybody to think about the process at Second Floor and to organise their thoughts by writing down (on sticky notes) what they felt was good about their process, what was bad and what could be improved. Notes were added to a prepared flip chart in the relevant category. We then asked the group to collate and de-duplicate the notes in the various sections. Using dot voting we generated a shared view of what was most important to keep doing, and what needed changing most urgently.

Then followed three Generating Insights/Decide What To Do cycles. Each cycle was built around one of the scrum roles whereby a short broadcast was followed by some Q&A and finally an explicit effort to connect some of the material just covered with specific points raised via the brainstorm. Envisioning how exactly we would make that explicit connection beforehand was difficult; we obviously we did not know what would come out of the brainstorm. On the other hand we knew that there were some quick wins to be had. At the same time, we wanted to avoid obviously guiding people to pre-defined conclusions using leading questions.

Eventually we decided to formulate a list of quick wins per role/section as a list of Try.../Avoid... pairs (à la Larman & Vodde) and play it by ear. If during discussion following a given broadcast, an obvious opportunity to make a link to the Try.../Avoid... pairs presented itself we would capitalise on it. Also, and alternatively, if discussion stalled we would introduce the appropriate Try.../Avoid... pairs as a catalyst for conversation. Organising the information according to the scrum roles worked well; it provided a natural framework for covering required ground on artefacts, ceremonies and rules while simultaneously detailing varying perspectives. However, in spite of (or maybe precisely because of) energetic discussion with the group, we struggled to connect to the results of the brainstorm in the fashion that we had hoped.

We wound the workshop up with a retrospective. We had chosen a combination of the Happiness Metric and the Perfection Game as feedback mechanisms. Feedback from the Perfection Game was again very useful, if challenging. Positive feedback regarding the brainstorm and the general structure of the workshop was tempered by the fact that we had struggled to explicitly incorporate the results of the brainstorm into further proceedings. This was not lost on our audience who complained via the Perfection Game of occasionally missing the relevance to their own situation.

Another interesting aspect of the feedback from this session was that it seemed to contradict itself at times. While some participants felt that we had spent too much time on certain basics, others complained that we had raced through concepts that were not yet familiar. The Dreyfus Model (a modern variant of Shu Ha Ri if you will) provides insight here, although not necessarily a neatly packaged solution! The Dreyfus Model works on the premise that a person's level of training or expertise on a given subject will dictate the best way for them to gain further proficiency in that field. In the extreme that means that the kind of instruction best suited to beginners (context-free instruction to be followed to the letter) is useless, even potentially damaging, to experts honing their intuition. And vice versa. As yet, we are not decided on how to deal with this insight.

This workshop was important for us on several levels. Particularly satisfying is that fact that we have added another increment (Scrum Basics workshop) to our product (Agile Coaching service) while refining (that is; iteratively improving on) already existing elements of same on the basis of (potential) customer feedback. We are hoping that our extremely collaborative approach will serve the double function of setting us apart in an increasingly competitive market while also ensuring that our customers get what they want, much as that may change along the way.

A question of measurement

As stated somewhat flippantly in last week's post, the mystery of quantum effects is in my opinion, a question of measurement. Besides uncertainty, there is this problem with the observer, whereby the observation of a quantum system somehow resolves it's probabilities into actualities. This implies that measuring a system determines the state of that same system. If the system-to-be-measured, is not the system-once-measured, what do measurements mean?

It's a topsy-turvy world down there at the quantum level.

But all is not necessarily as it seems at the macro level either. Think of taking the temperature of a volume of liquid using a mercury thermometer – the thermometer generates a reading because there is heat exchange between the liquid being measured and the thermometer. The act of measuring has changed the system, however slightly. For many purposes the thermometer can be regarded as accurate; the change in temperature that the thermometer effects in the liquid is negligible compared to the smallest unit of measurement.

At the quantum scale however particles are so small and travel at such high speeds (close to the speed of light) that even the infinitesimal is not negligible. This makes it basically impossible to take any measurements without significantly altering that being measured.

Complex adaptive systems display the same resistance to objective observation; for instance, it's notoriously difficult to measure the behaviour of human beings without modifying the behaviour under observation – have you ever dear reader suffered a blackout when taking an exam? Another example is the stock market; measuring consumer confidence can lead directly to a dampening or buoying of the market. Otherwise there is the stock indication itself - the very fact that stock is falling is often the reason that it continues to fall (or rise and rise as the case may be).

But doesn't that make an empirical approach to anything inherently problematic? Well, yes, but it's like Churchill reputedly said of democracy; it's the best we've got! There are steps that can be taken to reduce the subjectivity of a given observation or measurement and its impact on the system being measured, experiments can be set up whereby measurements approach objective non-invasive observations. This is an arduous process, which can be deceptively difficult to get right, and it breaks down quickly in the face of complexity.

How should we mere mortals effectively inspect (let alone adapt) then in the wonderfully complex arena that is software development?

We should proceed with care, measure only the bare minimum and always interpret metrics as approximations, avoid false precision. We need to recognise that metrics will drive behaviour, and be on the lookout for how. The measurements we do decide to take, we can make as trustworthy as possible by keeping as many other important variables as stable as possible. Measuring in relative quantities also helps, as does changing only one thing at a time. In short; apply a light weight version of the scientific method, remember Occam's razor and, be humble!

For example; to make velocity measurements meaningful/useful (for planning purposes), we should fix the sprint duration and keep the configuration of the team stable. We should estimate in story points and include only the points for completed stories in our measurements. Even having done all that, we should allow for the imprecision of our measurements by predicting only on the basis of velocity ranges.

Further we should remember that, while measuring team velocity can have the side effect of driving performance, measuring individual 'velocity' will have a detrimental effect on the team's results – especially if it's a manager doing the measuring!

Keeping entropy at bay (How scrum deals with Murphy)

The 2^nd law of thermodynamics states that entropy (disorder) will tend to increase over time in any closed system. This is not only a law of chemistry but of physics and information too, and we know it intuitively; nothing lasts forever.

This does however present some problems, predicting as it would seem to, the heat death of the universe, and flying in the face of the highly ordered state of life on earth. Luckily we don't have to answer any of those great questions in this humble blog, although I will come back to the subject.

What is a closed system? Well, in effect there is no such thing (except perhaps the universe itself), and at the same time we can treat pretty much any identifiable system as approaching a closed or isolated system. Take yourself dear reader; you are a highly complex (low entropy) system, identifiable as distinct from your environment (relatively high entropy), and yet dependant on it to stay ordered. Should you be truly isolated from your surroundings (unable to extract and consume energy or dispose of waste), your system boundary, so to speak, would quickly fail. Otherwise, as you are a system approaching isolation, time will do the trick, eventually. This is also true of software projects.

Of course, disorder can also arise spontaneously, as Murphy said; anything that can go wrong probably will. The more leeway you give Murphy the greater his opportunity to strike. It's in that leeway and how it's organised that scrum, and agile approaches generally, differ from the waterfall model. The waterfall tries to keep Murphy out, scrum (with XP) builds feedback in at every level, so that Murphy is kept on a short leash. This neutralises damage early while maximising learning. A scrum project, staffed by cross-functional teams, is also enmeshed in its environment, minimising isolation thereby resisting increases in entropy.

The waterfall invites entropy by the very means it employs to combat it. Each phase of a waterfall project is both long lived and isolated. That means that the requirements documents have necessarily 'decayed' somewhat by the time the requirements gathering phase is completed. Pushing the requirements through the contract-negotiation pipe to the design phase will also introduce turbulence, raising entropy some more. By the time the design phase is nearing completion the requirement set will be a veritable paradise for Murphy. When the development phase finally starts, isolation is reaching its zenith, the programmers don't know who the client is and entropy is spiralling out of control. Many waterfall projects end in heat death.

Speaking of heat death, here's my two cents on the great questions mentioned above - if we assume there is no such thing as a (truly) closed system, there is no problem of the 2^nd law; if there are no closed systems, the universe must be infinite in time and space, an unending soup of boiling roiling big bangs and even bigger crunches. A seething mass of bangs within bangs within bangs, bumping into and off of other bangs, each bang spawning its own black holes, which accrue galaxies over time, which collapse into ever more massive black holes, which might eventually become crunches, every black hole a potential seed universe, and so on...

Why scrum works (too)

Scrum is an astutely chosen set of boundaries (constraints; time-boxes, rules etc) imposed on the behaviour of a complex adaptive system (team) catalysed by the probe of certain types of information (user stories, feedback), in short, scrum works because it is empirical and adaptive. But scrum also works because it makes people happy! 

I can feel you squirming dear reader and I understand it. The Scrum Alliance claims that it is “transforming the world of work”, Henrik Kniberg, scrum guru, has introduced something called the happiness metric. What is going on? Management fads are annoying enough but when promulgated by evangelising purists proselytising the way to happiness, they're insupportable! And yet...

According to Dan Pink, it's not money that motivates, it's purpose, autonomy and mastery. Scrum delivers of itself autonomy and the opportunity for mastery, and as Meat Loaf said, two out of three ain't bad. Now, I'm as sceptical as the next man, especially when Pink requires that people be paid well enough to take the “issue of money off the table”. Then again, how many of us interested in transforming our world of work actually have any influence over how (other) people get paid? I certainly didn't when I floated the idea of 'going agile' in the sadly toxic environment I was working in a few years ago:

I had ended up in a typical matrix organisation running the waterfall (telco; IT built & maintained enterprise fulfilment system), complete with silos and gates and all that other waste. Third party integrators were in-sourced to deliver large scale change, maintenance updates were delivered by the in-house development team. Hand-off was built into every facet of this company's processes, contract negotiation was ingrained in its culture. Being a young and ambitious company however we also suffered from wishful thinking which meant that people were permanently disappointed, frustrated in their ambition by their own way of working.

Of course software delivery (I started there as the project manager responsible for the maintenance updates) was a nightmare. On hearing some of my colleagues one day bemoan yet another delay in the delivery of the requirements for the “next generation” system being built at the time, I suggested to them that they consider delivering the system in increments, at least to the acceptance environment. This reasonably conservative suggestion inspired vigorous rebuttal; agile methodologies (which I hadn't directly mentioned) might be okay for building GUI-centric client applications, but business critical enterprise systems were another matter, and innovation was not simple change management anyhow.

Despite this cool reception I made up my mind at that moment to pursue a transition to scrum. Shortly thereafter a major budgetary crisis conveniently paved the way for me to make a start; a cash flow problem meant that all external contracts were terminated forthwith and all open projects were put on hold. Some of those projects were however regarded as essential so after a short period of re-orientation the maintenance development team was asked to pick up the further expansion of the next gen system.

This was no sinecure, the development organisation (developers, testers, analysts) had suffered serious turnover in the months prior to these events and had undergone a major re-organisation. We were in many regards a new team. We did not know the system and it was being expanded for the offering of new services based on state-of-the-art network technology – domain knowledge was also at a premium. Also, some of the tooling in use in the next generation system was unfamiliar to the team (e.g. business rule engine). As is to be expected in such a situation, the new team also had strongly held opinions as to the architecture of the system they were assuming responsibility for – it was decided that we needed to include an architectural overhaul in any expansion of the system. And, of course, the new services mentioned were to be ready for sale within six months; there was a deadline before we had ever seen any requirements or delivered our first estimate! Oh, and the maintenance releases need to continue as before.

The advantage to all this should have been how easy it was to convince everybody involved that an agile approach was the only one with any hope of success. The environment in which we were operating however was toxic, trust was spread thin on the ground and the blame game seemed to be everybody's favourite past-time.

The business could live with some kind of phasing (levels of automation linked to product maturation) but our own architects said the system would have no meaning if not delivered in its entirety. Some of our stakeholders were willing to work closely with the development team but not to prioritise their wishes in a given phase (using MoSCoW for instance), they 'knew IT' and not getting something immediately meant never getting it. All involved, including most of the managers in IT, regarded estimates as commitments, despite regular warnings to the contrary. The team members themselves had invested quite a lot of effort in becoming specialists and felt threatened by the implication of generalisation inherent in agile methodologies, not only were testers loath to help analysts for instance, database programmers didn't want to know anything about Java and vice versa.

The development team lead was one of the colleagues with whom I had had the original conversation on iterations, he was process owner and saw the waterfall process with all it's in-built controls and sign-offs as the only thing shielding him and his team from total chaos. “And anyway”, as he said to me one day in all seriousness, “everybody knows that the project schedule is impossible, that's how we do things here, otherwise we would never get any budget”. This was our starting point.

Over the next 15 months or so we formed as a team and then stormed, stormed and stormed some more. The transition, largely because it was officially not happening, was painful in the extreme. For a significant part of the journey we had the worst of both worlds as 'agile' was given the blame for everything that went wrong while simultaneously being seized upon to excuse all sorts of sloppy practice. This was all taking place while the company itself got into ever deeper problems, external contractors were suddenly let go on a number of occasions along the way, we were put up for sale and the mother company had parachuted in a hostile CEO to effect the transaction.

Despite the tumult, we delivered three major releases in the time the business was used to receiving one (without missing a single maintenance update along the way). 'Simply' by reducing the size of the updates to our system while actively challenging the silo mentality and unifying the inputs to the team (one product backlog over all projects and change requests) we had improved our ability to deliver business value, in an extremely volatile situation, beyond all recognition.

The formal decision to go 'full scrum' was finally taken. In so doing, it was not the further improvements in transparency and predictability, nor the steadily increasing velocity of the team that impressed most. It was the broad smiles that made people stop and take note. Especially when it was taken into account that the sale of our company had just been effected. Our future was more uncertain than ever and yet the team was clearly happier (and more productive) than at any stage in the previous three years! 

Be happy, scrummify!