Fractals at the office

I have been struggling with the idea for this post since LKBE11 in October of last year. For most of that period, suffering as I was from a bulging disc in my lower back, I was on large doses of morphine and I attributed my lack of progress to my induced state of mind. The one post I did complete in the same period was meandering and maudlin, and I took that as proof that the problem was the author, not his subject matter.

Now, clear-headed for almost a fortnight, I have only succeeded, despite concerted effort and a sense of urgency (I wanted to post prior to CALM Alpha), in bloating my notes. I've had to accept that the basic problem is that I do not in fact know what it is that I want to say. That is, as long as I do not try to examine my thoughts directly, I 'know' intuitively what it is that I want to express but, as soon as I try to look my subject in the eye, examine it in detail so to speak, it seems to shimmer & fade and dissolve into a million tiny pieces. Each fragment may be open to investigation but it seems that I can only grasp the whole obliquely.

I have played with a number of titles such as “The magic of feedback; the roux to your secret sauce”, “The key to Emergence (Feedback)”, “On Fractals, Feedback and Emergence (Why IT is where it's at)”, “Connectedness; Complexity simplified” and so on. To no avail.

So I have decided to proceed as follows: I will revisit LKBE11 and the topics touched there that are close to my heart, propose some auxiliary reading - inspired by the reading list for Calm Alpha - and wrap up with a brief deliberation (and an outlandish extrapolation) on an emergent order that some feel is on its way, if it isn't in fact already nascent. In doing so I hope to give you, dear reader, some idea of why I think IT is such an exciting field to work in at the moment.

Probability

At LKBE11 Don Reinersten & Alan Shalloway separately made a case against Deming's 95% (as related in an earlier post) while John Seddon wound the conference up with a stinging attack on the idea that individuals could win against the system. In fact this is a (useful) false dichotomy.

Stating that the probability of a person failing to change a system approaches 1, when he or she is a constituent of that system, would seem to preclude any individual successfully effecting change in any system. But we now know that small changes in initial conditions (of any system incorporating recursive feedback) can lead to large scale differences in output over time; the (by now slightly infamous because so widely misunderstood/frequently overstated) butterfly effect.

We humans are not good at probability. The Buddha (karma) and Jesus (Do unto others..., Live by the sword, die by the sword...) knew that the system (society in their case) was a function of the individuals composing it and their interactions. I'm pretty sure that they themselves also understood that their preaching was fundamentally flawed. Doing good acts may well increase the probability of meeting good acts but it's not a direct relationship (certainly not in the asymmetrical power distributions of old). Living by the sword most definitely increases the chances of dying violently, and yet there's many a warrior has passed away peacefully in their sleep.

Both Christianity and Buddhism dealt with this problem by introducing a non-falsifiable reckoning. If a Christian lived a good life they would see paradise in the afterlife, if a Buddhist lived a less than exemplary life he or she would be punished by dint of reincarnation (as a lower form). Don't blame the ancients though, probability continues to trouble us - see for instance the Copenhagen interpretation of Quantum Mechanics (as parodied by Schrodinger's famous thought experiment).

I have found Popper's “Quantum Theory and the Schism in Physics” and, more recently, Nassim Nicholas Taleb's “Fooled by Randomness” to be very helpful in trying to avoid probability pitfalls in my thinking.

And back to LKBE11. Despite the rhetorical enmity, the three speakers mentioned proposed similar methods for improving systems – the way to create an effective and humane system is to give those governed by it the power to adjust same according to their (quantified) experiences working in it. What they might have agreed on was that it was in fact all about the feedback...

Feedback

I was first introduced to the power of feedback, self-similarity (fractals) and all that wonderful stuff by James Gleik's seminal “Chaos”. In the days and weeks after LKBE11 I realised that the principles of the various agile methodologies and lean, and the basic practices they imply, represent in fact a fractal! Delivering value from the start; expand incrementally, refine iteratively, create a trusting/trusted environment, seek out and incorporate feedback early and often – these edicts apply equally well at all levels whether you are writing software, coaching or setting up a new business venture. They are loosely worded, ambiguous even, such that they must be adapted to any given concrete context or scale; no two “instances” of a fractal pattern are the same

And it gets more exciting, for instance, TDD is in many ways a fractal instance of the agile principles; hypothesise, test, reflect, repeat (or OODA, PDCA, do+ inspect&adapt, whatever your flavour or context). Further, Kent Beck's 4 simple design rules are:

1. Run all the tests (create a trusting/trusted environment)
2. Contain no duplicate code (expand incrementally)
3. Express all the ideas the author wants to express (seek out and incorporate feedback early and often)
4. Minimize classes and methods (refine iteratively)

So, if, for instance, a team is working scrum + lean, with XP inside, guided by A-TDD, self-similar patterns can be observed throughout the system. Multiple levels of fast feedback are then built in, allowing magic to happen, through the emergence of strange loops. Strange Loops? I only began to truly understood the implications of chaos theory, fractals etc. through reading Douglas Hofstadter's “Godel, Escher, Bach; An eternal golden braid”. To do a whopping intellectual feat absolutely no justice I will summarize it by saying that Hofstadter makes the case in this book for emergence as a function of feedback.

Emergence, Evolutionary Change

At LKBE11 I was introduced to the Marshall Model. In his original paper, Marshall states that “The basic premise of Rightshifting is that organisations can improve their effectiveness incrementally (and with little turbulence / disruption), until such time as they approach a transition zone.” but also “At these transition zones, major upheaval – in the form of a shift of mindset – is required to proceed further to the right i.e. to continue to improve the effectiveness of the organisation.

Personally, I suspect that this major upheaval is a symptom of the phase transition as much as an agent thereof. Things proceed incrementally or they do not. Think of heating water through its various physical states, from liquid to solid to gas. The change applied is the addition of heat. Heat may be applied constantly but the behaviour of the system varies. It displays punctuated equilibrium, ice remains ice even as its temperature creeps upwards, as it approaches 0 C strange things start to happen, some liquid water is produced but the rise in temperature seems to stall even as the heat applied remains constant. This is called latency (heat supplied goes into loosening bonds, in this example freeing molecules from their crystal straitjacket instead of raising temperature) and it occurs at all phase transitions. Now, there is increased turbulence too, the newly freed water molecules in our example are sliding around banging into bonds, imparting energy and amplifying the instability of the remaining crystal. So, even though the system displays major disruption and or 'sudden' state changes (large shifts), and allowing for significant qualitative difference on either side of a transition zone (liquid/gas, synergistic/chaordic mindset), it remains so that the change process (heat supply, conscious adaptation of practice) evolves step by step.

For a truly enlightening discussion on (biological) evolution by increment, and much more, see Daniel Dennett's “Darwin's Dangerous Idea” and his “skyhooks & cranes” analogy. Further understanding may be gleaned form John D. Barrow's “Impossibility” which is a delightfully accessible book on limits, tipping points and so on.

Another concept that I was introduced to at LKBE11 which struck me as oddly familiar was Real Options. I was sure I had come across it before. And I had. The central argument of Dennett's (loftily titled) “Consciousness Explained”, is that mind is an emergent property of the brain. A model for consciousness is proposed which boils down to the self as “narrative centre of gravity”; consciousness as an assemblage of mechanistic tricks including a “multiple drafts” model for action. This has largely been borne out by neuroscience since with one important caveat – we do not apply “best fit” when choosing from multiple drafts but “first fit”. For a less speculative discussion on the mechanics of mind, see the excellent and more recent “We are our Brains” by Dick Swaab.

Complexity theory

Dave Snowden also spoke at LKBE11. Cynefin and the sense-making model were an exotic beast to me, angry socialist Celts with a sense of humour less so as it happens. I did not understand everything, and am still quite vague on practical detail even having done pretty much all my reading homework for CALM Alpha, but I was immediately sold on the general framework. I undertook to re-read “Frontiers of Complexity” by Peter Coveney & Roger Highfield but haven't gotten round to it yet. What stuck with me from that book was the idea that complexity at one scale often yields to simplicity at another and vice versa. It is possible to know something about attractors, to recognise patterns, but not necessarily when or where they will arise. On the other hand, setting simple boundaries on a system with a sufficiently high number of agents can gave rise to complex patterns, emergent properties. Snowden cites the example of a flock of birds (simple rules: fly to the centre of the flock, match speed, avoid collision) in this instance saying that you can be sure they will avoid an oncoming mountain peak, just not whether they will go past it on the left or the right. For a spectacular example of flocking in action see the following video of a starling murmuration.

Elsewhere Snowden talks informatively and entertainingly about order & chaos since antiquity, which got me thinking.

This idea of patterns which arise again and again also surfaces over and over throughout the history of human thought. From the Buddha (reincarnation) to French folklore (plus ça change, plus c’est la même chose) and beyond to Nietzsche’s “eternal recurrence” and Camus' search for meaning in the Myth of Sisyphus, humanity has seen this tendency to (almost)repetition as something negative, fateful, at the very least something to be faced down, overcome. Should we despair? According to Snowden there's no need; with the right tool set, the beast can be tamed! I look forward to learning more.

To conclude, at the risk of indulging my apophenia: if mind is an emergent property of the brain with its fifty to a hundred billion neurons and their trillion connections, constrained locally by a relatively small number of variables (potential barriers, neurotransmitters), has humanity not been building its superbrain(s) since the advent of language? Or at least since agriculture was first practised? Every advance in communications since, from domesticating the horse to the printing press to the telegraph, and on to the internet, has increased our interconnectedness. Our ability to share and exchange narrative. The internet may as yet have a relatively small number of physical nodes and display only loose connectedness but phenomena like Facebook and Twitter are adding another dimension.

If you were to view users as equivalent to neurons and the internet 'merely' as infrastructure, then that one facet of the human super brain alone already has 2 billion neurons... Supra-identities like nation, tribe or corporation (that may pre-date internet) could get stronger and display more cohesion with ever more internet capacity and connectedness. I wouldn't go as far as those who talk of Gaia, or The Technium, or The Singularity. I'm not suggesting a global consciousness. For one individual human mind the self is already a much trickier idea than might appear at first glance. I'm thinking more in terms of a fragmented and multidimensional space where initially rough-hewn, badly bound and shifting personalities (consciousness as an extremely loose narrative centre of gravity) interact. But even at that scale (decidely smaller than global) some problems already seem a lot less of an issue. Inter galactic travel for example; if a consciousness can expect to exist coherently for centuries what's a few light years' travel between friends?

The virtues (or otherwise) of feedback

Agile methodologies are successful because they are empirical, incremental and iterative; every action is examined and feedback is generated such that improvement is attained by taking small refining steps. Running scrum with XP generates feedback (TDD, pairing) as often as every few minutes!

It is abundantly clear to me why this results in great software better aligned with the customer's needs. However, in a recent conversation on the merits of frequent (structured) feedback I decided to play the devil's advocate and asked if it weren't possible that frequent feedback might sometimes in fact stymie some of the very best minds. This suggestion was met with appropriate derision (yes, it was a conversation among 'the converted').

I wasn't to be put off that easily however, what about Van Gogh I asked? He never sold a single a painting while he was alive, and when he paired with Gauguin it ended in tears. If Van Gogh had heeded the feedback he received on the fruits of his artistic endeavours he would have given up long before he got to Arles and the world would have been a poorer place for it. Art is not science was the rejoinder. There is a science to art though, and an art to science.

What about music then, it's still art but art with an explicitly mathematical character, how do great musicians deal with feedback? Beethoven was stone deaf by the time he wrote his majestic 9^th symphony, a symphony which to this day sets the standard for all other symphonies. When Stravinsky's Sacre du Printemps was first performed some ninety years later there was a riot! The composer (and the choreographer, who was also a touch too modern for the tastes of Paris' beau monde) were nearly run out of town. The press was almost unanimous in its condemnation of this newfangled noise and the weird dancing that went with it. These days, Sacre du Printemps is regarded as one of the masterpieces of twentieth century music. As for Schoenberg...

I wasn't convincing however, art was far too subjective (we hadn't even touched on literature yet), and that's exactly why creative geeks get into software; a bit is either on or off. Feedback is always good. And Schoenberg really is unadulterated noise!

However, new research would seem to indicate that I may not have been entirely misguided in my playful challenging of the virtues of feedback. Creative ideas tend to generate resistance, even among people explicitly seeking innovative breakthroughs, and regardless of objective evidence supporting the new idea!

So, when is negative feedback a sign that you need to change your ways and when is it a sign that you are most definitely on the right track? There is no simple answer to that question. As a rule I would err on the side of responding to feedback at face value but try and allow for the occasions when feedback says one thing but means another. In fact, this is a specific instance of a generic problem with all process guidelines, and a topic of hot debate in the scrum community;

• When does a product owner challenging a team become a manager pressuring the team?
• When does guarding the process become procedural nit-picking?
• When does positive conflict become damaging mudslinging?
• When does the avant garde represent the emperor’s new clothes?
• Etc.

In all cases the answer is (a sometimes seriously unsatisfying), “it depends”. And that is the art to this science of software development.

Keeping entropy at bay (How scrum deals with Murphy)

The 2^nd law of thermodynamics states that entropy (disorder) will tend to increase over time in any closed system. This is not only a law of chemistry but of physics and information too, and we know it intuitively; nothing lasts forever.

This does however present some problems, predicting as it would seem to, the heat death of the universe, and flying in the face of the highly ordered state of life on earth. Luckily we don't have to answer any of those great questions in this humble blog, although I will come back to the subject.

What is a closed system? Well, in effect there is no such thing (except perhaps the universe itself), and at the same time we can treat pretty much any identifiable system as approaching a closed or isolated system. Take yourself dear reader; you are a highly complex (low entropy) system, identifiable as distinct from your environment (relatively high entropy), and yet dependant on it to stay ordered. Should you be truly isolated from your surroundings (unable to extract and consume energy or dispose of waste), your system boundary, so to speak, would quickly fail. Otherwise, as you are a system approaching isolation, time will do the trick, eventually. This is also true of software projects.

Of course, disorder can also arise spontaneously, as Murphy said; anything that can go wrong probably will. The more leeway you give Murphy the greater his opportunity to strike. It's in that leeway and how it's organised that scrum, and agile approaches generally, differ from the waterfall model. The waterfall tries to keep Murphy out, scrum (with XP) builds feedback in at every level, so that Murphy is kept on a short leash. This neutralises damage early while maximising learning. A scrum project, staffed by cross-functional teams, is also enmeshed in its environment, minimising isolation thereby resisting increases in entropy.

The waterfall invites entropy by the very means it employs to combat it. Each phase of a waterfall project is both long lived and isolated. That means that the requirements documents have necessarily 'decayed' somewhat by the time the requirements gathering phase is completed. Pushing the requirements through the contract-negotiation pipe to the design phase will also introduce turbulence, raising entropy some more. By the time the design phase is nearing completion the requirement set will be a veritable paradise for Murphy. When the development phase finally starts, isolation is reaching its zenith, the programmers don't know who the client is and entropy is spiralling out of control. Many waterfall projects end in heat death.

Speaking of heat death, here's my two cents on the great questions mentioned above - if we assume there is no such thing as a (truly) closed system, there is no problem of the 2^nd law; if there are no closed systems, the universe must be infinite in time and space, an unending soup of boiling roiling big bangs and even bigger crunches. A seething mass of bangs within bangs within bangs, bumping into and off of other bangs, each bang spawning its own black holes, which accrue galaxies over time, which collapse into ever more massive black holes, which might eventually become crunches, every black hole a potential seed universe, and so on...

Mother Nature as exemplary engineer (Why does Scrum work?)

Evolution in a (super simplified) nutshell is: descent with modification, acted upon by natural selection.

That is to say, organisms pass properties from one generation to the next via some process or other. This process is never perfect and therefore each new generation of a given organism will differ slightly from its parent generation. These differences will be selected upon by the environment; changes that offer some advantage to an organism in a given environment will eventually spread throughout a population (survival of the fittest).

However, most (significant) modification will be disadvantageous, certainly if change is only achieved by (random) errors in the copying process. This should mean that almost every trial will end in (fatal) error. In that case, even the 3 billion or so years that have passed since the first unicellular organisms appeared on earth would not represent near enough time to produce life as we know it today in all its complex diversity.

A 'recent' discovery in physics helps address this problem; complex adaptive systems self-organise.

What?!?

A discussion of this phenomenon of self-organisation, and/or complex adaptive systems, is way beyond the scope of this piece - not to mention how monumentally challenging it is to your author's intellectual capacities - so I will just give a (hopefully compelling) example to illustrate the phenomenon.

Consider the termite colony. Termite mounds are built without a site director or a project manager or in fact any kind of management structure. How do all those termites, scurrying around in their own little restricted worlds, so obviously manage to work together as successfully as they do? They, that is to say, the complex adaptive system that is a termite colony, self-organise(s).

This tendency to self-organise introduces another possible source of variance in evolution. The environment selects on these experiments in self-organisation in precisely the same way as it does on the changes in the genetic code of the individual organisms, blindly but steadily (Mother Nature literally takes aeons to realise her handiwork), weeding out that which does not work well in favour of that which does.

All well and good, but what good could all this possibly be to us in software development?

Well, if a few extremely smart software engineers were to get together to see how they could best build good software, they might consider how Mother Nature goes about her work. If it was possible to speed up the process of selection acting upon modifications they might even have been on to something...

Well, a bunch of software engineers did go through an approximation of this thought process, and they came up with... Scrum!

Scrum ( in a super simplified) nutshell is; self-organising units engaged in constant experimentation, the results of which are acted upon by (short) formal feedback cycles at multiple levels

In short, Scrum works because it is a modified copy of how that greatest of all engineers, Mother Nature, goes about her job.