The Problem With Knowing

If you read my recent article, That Sinking Feeling, you’ll know I’ve been thinking rather a lot about the Titanic since reading a number of inquiries, including the Thirlwall Inquiry. I promise I’m not planning to turn this into a series of articles about every unfortunate decision made before some travel disaster, although there probably is enough material for one. What interests me about the Titanic is that we all know the ending. That makes it very easy to look back at what happened and wonder how people could have made the decisions they did. With hindsight, the warning signs seem obvious, the consequences seem inevitable and some of the choices made that night can look almost inexplicable. But that’s because nobody on the Titanic had the benefit of knowing how the story ended.

That got me thinking about something I’ve encountered throughout my working life. We spend an enormous amount of time collecting information, analysing it, checking it and presenting it in ever more impressive dashboards. All of that is useful, but there comes a point when the information has done its job and someone has to make sense of it. The interesting question isn’t always whether we have enough information. Sometimes it’s what happens when the information is sitting in front of us and our experience, assumptions and expectations tell us something rather different.

And that’s where this article picks up from That Sinking Feeling. In that first article, I was interested in what happens when the warning is there, the information is available and the organisation still fails to respond. This time I’m interested in what happens before we even get that far. Information doesn’t arrive in a vacuum. We interpret it through what we already know, what we’ve experienced before and what we expect to happen next. A warning that looks obvious to us might not look quite so obvious to someone who has heard similar warnings before and seen nothing happen, or to someone whose previous experience tells them that the situation is probably manageable. So this isn’t really about ignoring information. It’s about how we decide what information means and how much weight we give it.

Information doesn’t arrive in a vacuum. We interpret it through what we already know, what we’ve experienced before and what we expect to happen next.

I came across this quite early in my career. I was asked by a senior member of an NHS Trust to get some more data to show that their area didn’t have a problem with coronary heart disease mortality. We’d already calculated the figures over a number of months and the rate was consistently higher than we’d have wanted to see, with some indication that it was increasing. The request, then, wasn’t really to find out what was happening. I was being asked to look again and prove that the apparent problem wasn’t a problem. I’ve always been uncomfortable with the word “prove” when it’s used around scientific analysis anyway. We can produce evidence, test hypotheses and establish how strongly the evidence supports a conclusion, but “prove” promises a level of certainty that science rarely gives us. I’ve written about that elsewhere in The Proof of the Pudding and more recently in Method to the Madness, so I won’t wander off down that particular rabbit hole here. What stayed with me from this episode was the fact that the desired answer had already been decided before the analysis had been completed. The data was being asked to find a different conclusion rather than to tell us what was happening.

That experience has probably influenced the way I approach analysis ever since. I’m always interested in what the numbers say, but I’m at least as interested in what people thought they were going to say before they saw them. We all bring assumptions to information. We bring previous experience, organisational history and memories of what happened the last time someone raised a similar concern. An analyst can present a result that is perfectly sound and still find that the first response is not “What should we do about this?” but “How can that be right?” Sometimes that’s exactly the response we need. Data can be wrong. Definitions can change. Samples can be misleading. A relationship between two things can look convincing until you discover that something else is sitting underneath it. Anyone who has worked with real data for long enough will have seen numbers behave beautifully until somebody opens the bonnet.

The difficulty is that the same healthy scepticism can become something else if we aren’t careful. There’s a point where checking the evidence is good analysis and a point where we’re simply looking for a reason not to believe an answer we don’t like. I’ve probably done a bit of that myself over the years. Analysts like certainty because uncertainty makes our lives awkward, particularly when someone wants a clear answer and the evidence comes with a collection of caveats. We can keep improving the model, checking the methodology and discussing the limitations almost indefinitely. There’s always another question we could ask. At some stage, though, we have to decide whether the evidence is good enough to inform a decision. Scientific analysis rarely gives us perfect certainty, and waiting for it can become an excuse in its own right.

There’s always another question we could ask. At some stage, though, we have to decide whether the evidence is good enough to inform a decision.

The Titanic passengers provide a useful reminder of why this isn’t simply a question of whether people were being sensible. Some passengers were initially reluctant to enter the lifeboats, which can seem extraordinary when we know what happened next. But they didn’t know what happened next. They were looking at a ship that was still largely upright and lit, and which, to many of them, still appeared capable of carrying on. They were being asked to leave the relative safety and comfort of a huge ocean liner and climb down into a small open boat on a dark Atlantic night. The Titanic had also been presented as an exceptionally safe ship. People had boarded it expecting to arrive in New York, not to be told a few hours later that they should abandon ship. Their experience and everything they could see around them were therefore part of the way they interpreted the warning. A lifeboat could look like a strange place to be when the ship you were leaving still appeared safer than the boat you were being asked to get into.

That’s easy to miss when we look backwards. We know the ship was going to sink, how quickly it would happen and what the consequences would be. The people on board had none of that information. They knew there had been a collision and that there was a problem, but they had to work out what that problem meant while it was happening. The Titanic was still afloat. The lights were still on. People were still moving around the ship. There was no way for someone standing on deck to see the complete sequence of events that we can now see simply by looking back at the history.

The same applies to the warnings that had reached the bridge. Ice in the North Atlantic wasn’t an unfamiliar problem, and a warning about ice therefore arrived with a context. It wasn’t a message saying that the ship was about to sink. It was another piece of information received during a voyage that had, until then, been proceeding normally. Previous experience helped determine how that information was understood. That doesn’t mean the warnings were ignored or that anyone knew what was going to happen. It means that information rarely arrives with its significance already attached to it.

information rarely arrives with its significance already attached to it.

That’s the part of the story that interests me. We tend to think that information tells us what something means, when in reality we bring quite a lot to the interpretation ourselves. Previous experience, expectations and what we can see happening around us all influence the weight we give to a new piece of information. Someone who has encountered the same warning several times without anything serious happening may interpret the next warning differently from someone encountering it for the first time. Neither person necessarily has better information. They’re bringing different histories to the same information.

That also explains why hindsight can be so misleading. Once we know the outcome, the earlier decisions seem to form a neat chain leading towards the disaster. We can identify the iceberg, the warnings, the speed of the ship, the lifeboats and everything that followed. For the people involved, those things didn’t arrive as a completed story. They arrived as separate pieces of information whose significance had to be worked out at the time. They were trying to decide what was happening without knowing which of the things they were seeing and hearing would turn out to matter most. Experience becomes part of the evidence we use to interpret new information.

The old story of the boy who cried wolf is a good illustration of this. A shepherd boy looks after his sheep and, apparently bored with the job, repeatedly raises the alarm that a wolf is attacking the flock. The villagers come running to help, only to discover that there’s no wolf. After this happens a few times, they stop believing him. Then, when a wolf really does appear, the boy raises the alarm again, but the villagers ignore him because they’ve learned from experience that his warnings haven’t meant anything before. By the time the genuine danger arrives, the message has acquired a history. The word “wolf” still means exactly what it did before, but the people hearing it no longer respond to it in the same way because they remember what happened last time.

Organisations can fall into exactly the same trap. A warning is raised, people respond, meetings are held and perhaps nothing much changes. The same thing happens again a few months later, and eventually the next warning arrives carrying all the previous ones with it. It may be accurate. It may even describe a more serious situation than the earlier warnings. But people don’t encounter information in isolation. They encounter it with a memory of what happened the last time they heard it, and that history can change the weight they give to the next warning.

people don’t encounter information in isolation. They encounter it with a memory of what happened the last time they heard it, and that history can change the weight they give to the next warning.

I’m always surprised at how quickly yesterday’s abnormal can become today’s baseline within an organisation. A measure gets a little worse. People notice. It gets worse again and there’s another discussion. After a while, the new figure becomes the number against which the next figure is compared. The question changes from “Why has this got so bad?” to “Is this month better or worse than last month?” without anybody having formally decided that the original position no longer matters. The reference point has moved because everyone has become used to where they are.

You see we can often adapt to circumstances without ever consciously deciding to accept them. A service can deteriorate gradually and the deterioration becomes part of everyday life. A waiting time that would once have caused alarm becomes something staff know how to work around. A recurring problem becomes the subject of jokes. A report starts showing an awkward number every month and, after a while, the number itself is no longer news. People know it’s there. They’ve talked about it. They may even have become very good at explaining why it’s there. The existence of an explanation can sometimes feel remarkably like having solved the problem.

As a bit of an aside, the same thing happens with processes. A report is created because somebody needs particular information. People use it, decisions are made and everyone is happy. A few years pass, the people change, the service changes and the original question disappears, but the report continues arriving every month. Somebody still produces it because that’s what they do. Someone still checks it because that’s part of the process. It gets circulated to people who may no longer remember why they’re receiving it. The really impressive thing is that it can continue for years without anybody doing anything particularly wrong. The process works. The report arrives. The dashboard refreshes. Everything is functioning exactly as designed, even though the reason for having designed it in the first place has vanished somewhere along the way.

That’s a useful lesson for organisations because we often look for problems in the way a task is being carried out when the more interesting question is whether the task still needs to be carried out at all. The person producing the report may be doing exactly what they’ve been asked to do. The analyst monitoring the measure may be doing a perfectly good job. The manager may be running the meeting exactly as they always have. None of them necessarily has a reason to stop and ask whether the circumstances that created the work in the first place still exist. We can become remarkably efficient at maintaining something without noticing that the thing itself has stopped being useful.

We can become remarkably efficient at maintaining something without noticing that the thing itself has stopped being useful.

This is one of the things that makes the use of AI within organisation interesting to me. We continue to build systems that are more and more capable of looking at vastly more information than any individual could reasonably examine in their lifetime. They can identify unusual patterns, generate forecasts and produce warnings at incredible speed. And that gives us a remarkable ability to find new things. But I’m less convinced that finding more things automatically means we’ll make better decisions.

If a system produces ten warnings, somebody might reasonably look at ten warnings. If it produces ten thousand, we have a different problem. The machine hasn’t run out of capacity, but the people receiving the results certainly have. We can end up with an AI system spending the night finding things that might need attention and humans spending the following day working through the list of things the AI has found. At some point you start wondering whether we’ve automated the difficult bit or just found a very efficient way of creating even more difficult bits.

And there’s an obvious danger in treating every machine generated warning as though it deserves the same response. Human attention is limited. If everything is urgent, then nothing is particularly urgent for very long. Someone still has to decide which findings matter, which ones are credible, which ones need investigating and which ones actually change what should happen. That decision can’t be handed over simply because the information arrived from an impressive piece of software. The technology can find patterns in the data, but the significance of those patterns still depends on the situation in which they appear.

If everything is urgent, then nothing is particularly urgent for very long. Someone still has to decide which findings matter, which ones are credible, which ones need investigating and which ones actually change what should happen.

That’s why I’ve never been especially keen on the expression “data driven”. It makes it sound as though the data gets into the driving seat, adjusts the mirrors, checks the route and heads off towards the destination while the rest of us sit in the back and admire the scenery. I’ve never encountered quite that arrangement, although it seems to be the destination that AI developers want to reach. Personally, I think we’d be better calling it “data informed”. Yes, information can tell us something we didn’t know, but people still have to interpret it in the context of what they’re trying to achieve and what else is happening around them. The number doesn’t make the decision. It gives the person making the decision something to think about.

That’s where things can get problematic in large organisations. Different people can each be doing exactly what they’re supposed to do while the overall situation still moves in the wrong direction. The person collecting the data can collect it correctly. The analyst can analyse it correctly. The manager can run the service correctly. The report can be produced on time. Everyone can point to the thing they were responsible for and show that it was done properly. Meanwhile, the information sitting across all those activities can be telling a story that nobody has really taken ownership of.

That’s not the same thing as saying that everyone needs to know everything. Large organisations couldn’t function that way. We need people with different responsibilities and different areas of expertise. What we also need, though, is somebody willing to step back far enough to see when all those perfectly reasonable activities are adding up to something that isn’t reasonable any more. Captain Edward Smith couldn’t personally do every job on the Titanic. He didn’t need to. He did, however, remain the person responsible for the ship as a whole. The point isn’t that modern organisations need a captain standing on a bridge shouting instructions. It’s that dividing work doesn’t divide the consequences of what happens when all that work comes together.

And that becomes difficult when the information points towards a decision that people don’t want to make. An analysis may suggest that a project isn’t achieving what it was intended to achieve. A service may be producing an outcome that nobody expected. A process may have survived long enough to become almost impossible to question. Changing any of those things has consequences for people who have spent time, money and effort making the existing arrangement work. The numbers don’t create those consequences. They simply make it harder to pretend they aren’t there.

The point about knowing is that it changes the nature of the problem. Before we know, we have a question. Once we know, we have something to interpret. The answer may still be uncertain, the evidence may still have limitations and there may still be good reasons to gather more information. But at some point the issue stops being about what the data says and becomes a question of what we are going to do with what it says.

The point about knowing is that it changes the nature of the problem. Before we know, we have a question. Once we know, we have something to interpret.

That’s why the Titanic continues to be such a useful story. The iceberg is the obvious bit, but it isn’t really the bit that we should pay most attention to. We need to remember the people before the iceberg, trying to make sense of information with the experience they already had, becoming accustomed to things that were changing and making decisions without the benefit of hindsight. That’s much closer to the world most of us work in.

We can spend an awful lot of time trying to make sure the information is right, and we should. Eventually, though, there comes a point where the uncertainty has been reduced as far as it reasonably can be, the question is the right one and the evidence is telling us there's a metaphorical iceberg ahead. At that stage, asking for more information may just become a way of postponing the inevitable. Knowing doesn’t automatically lead to action. People still have to interpret what they know, decide how much weight to give it and, eventually, accept whatever consequences come with acting on it. We often think that the problem is not knowing enough, because that’s a problem we know how to work on.

Knowing doesn’t automatically lead to action. People still have to interpret what they know, decide how much weight to give it and, eventually, accept whatever consequences come with acting on it.

We can always collect more data, run another analysis and build another dashboard. But knowing what the information is telling us can leave us with a much less comfortable task, because once we know, we have to decide what we’re prepared to do about it. And that, unfortunately, is the problem with knowing.