Organisational learning as artificial intelligence

A recent webinar by the APM was titled “Project knowledge management​”, but took a very different approach to our regular understanding of that term.

The speaker was Dr Atif Ansar, a specialist in understanding the performance of major projects. His lecture compellingly framed organisational learning as being akin to machine learning. As an experienced academic, the lecture was well delivered and his enthusiasm for the subject very infectious. The whole lecture is available on YouTube but was so interesting that I have chosen to write it up in some detail.

Collective intelligence vs individual intelligence

Organisations have collective intelligence. Many bright employees may learn lessons independently, but the organisation has its own level of learning that depends on being able to draw on those individual lessons. How do we know, then, whether our organisation is actually learning? There are certain signs that we can look for:

  • Does performance improve over time?
  • Do lessons learned get shared effectively?
  • Are mistakes repeated or do they disappear?

An organisation has certain characteristics similar to a large learning computer, with the potential to realise lessons that the component parts cannot. “Collective superintelligence is a system composed of a very large number of smaller intellects, such that the system’s overall performance across many very general domains vastly outstrips that of any cognitive system.”

Failing to learn from experience

By way of example, Dr Ansar showed a graph from his own research, detailing the cost overruns on 250 large dam-building projects from around the globe, stretching back over a century. He asked us to draw some conclusions about our ability to learn from previous experience of similar projects.

Scatter graph showing cost overruns associated with large dam projects globally over the past hundred years
Cost overruns of large dam projects

Clearly, the builders of dams are not learning lessons. Not only are the cost overruns not reducing or disappearing overall, but even the variance between projects isn’t significantly decreasing.

Why do organisations fail to learn? Dr Ansar quoted from Levinthal and March’s paper The Myopia of Learning. They identify three blind spots in organisational intelligence:

  • Temporal myopia. We tend to ignore older data, just looking at recent projects. The dam-building graph above shows that this approach is a mistake: older projects clearly have relevant lessons which we would miss if we constrained ourselves only to examining the most recent few projects. There is also the phenomenon of “catastrophic forgetfulness”, itself characterised by two causes:
    • Institutional amnesia: The most important lessons are not carried over from one large project to the next. Even very experienced project managers may only work on a small number of large projects over the course of their careers. Worse, they retire at the absolute peak of their knowledge and expertise. (We have poignant examples of that at my own place of work, with some programmes spanning an entire career. I know of a colleague who joined a large programme at its inception, starting in the PMO, who will likely still be with the programme when he retires.) Additionally, and commonly, there are imperfect or incomplete knowledge hand-overs between stages of the same project, where these are led by different people.
    • Well-meaning amateur: This relates to projects directed by someone for whom it is a pet (or political) project, rather than in their sphere of expertise.
  • Spatial myopia. We tend to ignore data that seems geographically distant – dismissing the experience of those that are further away. To illustrate this fallacy, Dr Ansar showed the dam-building overrun data plotted on a world map. Some geographical areas perform significantly better than average, and some significantly worse; therefore, we should learn lessons from both. For example, were we to learn lessons only for projects undertaken in the UK, we would miss the lessons both of the worse-performing projects in South America, and the relative successes in North America.
  • Failure myopia. Many people and organisations are uncomfortable discussing failure in an honest and constructive way – perhaps because it is threatening to individuals’ professional standing or job prospects. The notion of “radical candour” is well-embedded in software projects, and in the principles of agile, but not so much in larger projects.
World map showing large dam projects and cost overruns
Cost overruns by geography, demonstrating spatial myopia

How machines learn

Dr Ansar then turned to a well-known machine learning problem: classifying images. In a collection of photographs showing either chihuahuas or else muffins that happen to resemble chihuahuas, human beings can reliably identify which is which with 95% accuracy. (The remaining 5% might be because there really are muffins that look exactly like dogs, or maybe the human being was just in a hurry and pressed the wrong button.) Machines, on the other hand, classically find this problem very hard. An untrained computer will just guess, and get 50% of the answers correct. However, since about 2007, machines have got better and better, and can now be trained to an accuracy of 97% – better than the humans. They are also very much faster.

How did this happen? A leader in the field is Fei-Fei Li of Stanford, responsible for ImageNet, in which millions of images were hand-classified so that computers can learn from them. In machine learning, trying to specify to the machine what a dog looks like – deductive modelling – doesn’t work. What works is providing the machine with huge data sets and then allowing the machine to make sense of the data. By extension, Dr Ansar reasons, in organisational intelligence, it’s not a good idea to try to specify the parameters of a successful large project in advance: don’t force the project into a template that might not be a good fit. Instead, collect as much data as possible on comparable projects, and get people to make sense of that data. This requires huge effort and a lot of trial and error, or simulation, but the dividends are massive.

Another example was Google DeepMind. This demonstrated that a machine trained on simple tasks could then teach itself more complex tasks. Learning reinforces – it is cumulative. Thus ImageNet can now diagnose complex heart defects that even the most highly-trained human doctors would find difficult. That cumulative learning can lead to unexpected breakthroughs – the machine appears to improvise, leading to “serendipitous discoveries”. For example, when DeepMind was taught to play the complex game of Go, it not only set a milestone by beating human players, but it did so in a way that was utterly unexpected by both the game’s most skilled players and the very people that programmed it.

Are we a learning machine?

Does an organisation – does our organisation – meet the criteria for being a “learning machine”? Dr Ansar provided a checklist:

Checklist for learning machine organisations
Checklist: Is my organisation a learning machine?

He concluded by observing that overall performance is multiplicative, rather than additive, across the categories of people, systems (tools and technology), and process. Organisational learning therefore needs to cover all of these categories, as reflected in the checklist. He was emphatically not suggesting that an organisation would be effective without experienced professionals, any more than it would be without tools and processes for learning.

Q&A

Dr Ansar had some interesting thoughts on how to capture and retain the experiences of those leaving an organisation, perhaps due to retirement. Perhaps surprisingly, he emphasised the very human experience, rather than a data gathering exercise:

  • Rather than trying to fill out a template, allow the leaver a blank canvas on which to paint the project’s strengths and weaknesses. Classify the picture into data later.
  • Human beings love to feel useful and love seeing that their expertise matters. So stay in touch after retirement – perhaps get them back to consult from time to time.

Somebody asked a question about breaking down the taboo of talking about failure. Dr Ansar suggested that a reluctant team could begin by discussing other people’s failures – looking to competitor projects. Alternatively, conduct a “pre-mortem” (truly the most excruciating example of corporate neologism) in which the team imagines a future in which the project has gone well, and a future in which the project has failed, and collectively identifies the reasons for each outcome.

Finally, somebody asked how to learn lessons if a project is ground-breaking or unique. Dr Ansar proposed, uncontroversially, that one should minimise the uniqueness between projects, so that lessons can still be learned from the common factors.

Final thoughts

I should like to reconcile Dr Ansar’s views with the traditional understanding of organisational knowledge, especially that described in an internal blog post by my colleague John, a specialist in this area.

Dr Ansar’s definition of knowledge relates to drawing experience from huge data sets – treating knowledge as a “thing”, albeit a vast one built from many tiny component “things”. Alternatively, my colleague would consider knowledge as a flow of ideas that, if managed successfully, cyclically increases both the amount of explicit, codified knowledge and the tacit, socialised knowledge available to an organisation and the communities that comprise it. However, some of their conclusions do overlap very nicely. They certainly agree that knowledge is costly as well as disruptive; and that experimentation and simulation develops new knowledge, or innovation.

A knowledge approach based on collecting and analysing past data may work for complicated systems – those characterised by many parts, but which can be deconstructed and examined. However, the paper [Snowden, D – Complex Acts of Knowing: Paradox and Descriptive Self-Awareness] at the heart of John’s blog post explicitly refutes that knowledge of complex systems (those characterised by a range of interactions between irreducible components, including humans with their messy, unpredictable feelings and motivations, leading to unknowable future behaviour) can be gained through developing models based on past behaviour.

Despite their differences, I suspect that there are many other common conclusions to be gained from these two perspectives, and indeed they may end up being mutually reinforcing. For example: where Snowden’s paper demands regular disruptions in “knowable space” to avoid “entrainment of thinking”, these could perhaps be provided by the kinds of serendipitous breakthroughs Dr Ansar discussed. If so, perhaps the DeepMind example shows that – far from undermining the established experts – such breakthroughs might even be considered exciting, joyous events.

By David A

Imagination of a poet - communication skills of an engineer. In no particular order, I like old cinemas, Japanese gardens, playing music, giraffes, technology, films, ducks, narrow Italian streets, and trying to write stuff that's interesting to people. I've been blogging irregularly since 2005 and have had some form of website since 1998.