That Sinking Feeling

This week I’ve been reading yet another inquiry report. To be honest, I’ve lost count of the number of reports I’ve read that contain some variation of the same conclusion. The warning signs were there. Information existed. Concerns had been raised. Questions had been asked. Nobody was entirely unaware that there might be a problem. The details obviously change from one report to the next. The names and circumstances differ, but the underlying pattern is usually remarkably familiar.

We like to believe that failures happen because people didn’t know. It’s a comforting explanation because it suggests the solution is simple. Gather more information. Commission more analysis. Build a better reporting system. Create a dashboard. Introduce a new technology platform. Yet many of the reports that attract headlines tell a rather different story. The issue wasn’t that information was absent. The issue was that information existed and somehow failed to change what happened next.

Reading that report this week reminded me of something I’ve seen repeatedly during my career. Organisations can become surprisingly comfortable living alongside information that ought to make them uncomfortable. A concern is raised and gets added to a risk register. A performance measure starts heading in the wrong direction and appears in monthly reports. Staff begin voicing concerns. Customers make complaints. An audit identifies weaknesses. A piece of analysis suggests a problem is emerging. Everyone acknowledges that the issue exists. Meetings are held. Updates are requested. Actions are recorded. Yet somehow the underlying problem remains remarkably unchanged.

The greatest danger isn’t always a lack of information. Sometimes it’s becoming so accustomed to information that we stop responding to it. Warnings become familiar. Risks become routine. Things that would once have prompted urgent action become accepted as normal simply because they’ve been discussed for so long. Which is why the story of the Titanic remains surprisingly relevant to anyone who works with data, analytics, dashboards, risk registers or artificial intelligence. It isn’t really a story about an iceberg. It’s a story about information, judgement and what happens when organisations become so familiar with risk that they stop responding to it.

The greatest danger isn’t always a lack of information. Sometimes it’s becoming so accustomed to information that we stop responding to it.

I’m going to split this story into two parts, partly because I’ve already managed to get from an ocean liner in 1912 to modern organisational governance, and I suspect that trying to cram the rest into one article would produce something long enough to require its own risk assessment. So, the first part is about the Titanic, the warnings it received and the rather dangerous comfort we can get from believing that having safeguards means we’re safe. The second will move into the world I’ve spent most of my career working in, where the iceberg is replaced by data, dashboards, risk registers, AI and the rather awkward business of deciding what to do when the information tells us something we don’t want to hear.

The interesting thing about the Titanic is that there wasn’t a scarcity of data. In 1912 they wouldn’t have called it data, of course. People hadn’t yet discovered that giving perfectly ordinary things technologically sounding names makes them seem far more impressive than they really are. They had messages, observations, reports and people whose job was to collect information and pass it to those making decisions. We call much the same thing data today, although we’re generally inclined to put it into a dashboard, surround it with coloured circles and appoint somebody to maintain it.

During 14 April 1912, the ship received a number of wireless warnings about ice. The Caronia reported icebergs and field ice. The Baltic sent information about ice, and later the Mesaba reported heavy pack ice and a large number of icebergs in an area that included the Titanic’s intended route. The subsequent British inquiry established that the officers knew they were entering waters where ice was to be expected.

This is important because it challenges the way the story is often remembered. It’s tempting to think of the disaster as a consequence of not knowing what lay ahead. In reality, information about the risk was arriving throughout the day. The warnings weren’t absent, and they weren’t hidden away where nobody could find them. Information was being received, communicated and discussed. The more interesting question is what happened after it arrived, particularly if you spend your working life thinking about data, decisions and the curious things organisations do with both.

It’s tempting to think of the disaster as a consequence of not knowing what lay ahead. In reality, information about the risk was arriving throughout the day.

There’s another part of the story that’s worth considering. The Titanic had acquired a reputation for being extraordinarily safe and was widely described as effectively unsinkable. The National Archives itself refers to it as a “supposedly ‘unsinkable’ vessel”, and the contemporary inquiries examined why warnings about ice had not resulted in greater precautions.

It would be too easy to say that people ignored the warnings simply because they thought the ship couldn’t sink. The evidence doesn’t really support such a simple explanation. But confidence in the thing you’re protecting has to affect how you think about warnings. If you believe you’re sitting on one of the safest ships ever built, an iceberg warning may not carry quite the same psychological weight as it would if you believed the ship was inherently fragile.

That has a rather uncomfortable modern equivalent. Organisations can become so confident in the safeguards they’ve put in place that the existence of the safeguard starts to become evidence, at least in their own minds, that the underlying risk must therefore be under control. We have a risk policy, so risk is being managed. We have a risk plan, so someone has thought about what happens if things go wrong. We have a risk register, so the important risks have been identified. We have a governance framework and a compliance process, so there are clearly people somewhere keeping an eye on things. And now, increasingly, we have an AI strategy, which presumably means we’ve thought about the risks associated with AI as well as the opportunities.

Organisations can become so confident in the safeguards they’ve put in place that the existence of the safeguard starts to become evidence, at least in their own minds, that the underlying risk must therefore be under control.

At some point, though, all of those reassuring things can become substitutes for actually asking whether the risk is being managed. The policy exists, the plan has been approved, the risk is sitting there in amber on the register, and the governance meeting has a standing agenda item, so everyone can point to something that demonstrates that the organisation has taken the matter seriously.

Somewhere in the process there’s probably even a human in the loop called Bob.

Bob has been identified as the person responsible for providing human oversight, which sounds reassuring until you start asking what Bob is actually expected to do. He’s supposed to notice when the AI does something odd, understand why it has done it, decide whether it matters, intervene if necessary and explain what happened afterwards. Unfortunately, Bob is also responsible for three other projects, two committees, the monthly performance report and something called “business readiness”, so he’s unlikely to be sitting beside the algorithm waiting for it to produce its next questionable decision.

If the system makes ten thousand decisions overnight, Bob presumably needs to review them before breakfast, despite nobody having explained which ten thousand he is supposed to look at or how he is expected to distinguish the sensible ones from the nonsense. If he spots a problem, he needs the authority to intervene. If he doesn’t have that authority, he isn’t really providing oversight. He’s providing commentary, which is rather different.

And this is where Bob can become particularly useful to the organisation while being remarkably useless to the system. If anybody asks whether there is human oversight, the answer is yes. Bob exists. Bob has a role. Bob has probably even completed the training. The fact that he wasn’t available when the system made the decision, couldn’t understand why it had produced the output or had no authority to stop it is rather less convenient, but those details don’t fit quite so neatly into a governance report.

There’s a rather important difference between having a control and having control. A policy doesn’t manage a risk simply because the policy exists. A risk register doesn’t mitigate anything merely by containing a beautifully worded entry in amber. A human in the loop doesn’t provide meaningful oversight simply because a human has been placed somewhere in the process diagram. An AI strategy doesn’t make an organisation safe from AI any more than the Titanic’s reputation made an iceberg less capable of sinking it.

There’s a rather important difference between having a control and having control.

I’ve spent quite a lot of time around organisations where somebody says that they need more data. Sometimes they genuinely do. If we’re trying to understand something and don’t have the information we need, then collecting better information is a fairly sensible thing to do. But there are plenty of occasions when the organisation already knows more than it wants to admit. The difficulty isn’t finding the data; it’s deciding what to do when the data tells you something inconvenient, and that’s a rather different problem.

We’ve all sat in a meeting where somebody says that there’s a significant risk that a project might fail. Nobody particularly wants the project to fail, so everyone agrees that the risk needs to be monitored. Somebody adds it to the risk register, another person agrees to provide an update at the next meeting and, because we’re a modern organisation, somebody probably gives it an amber status.

Three months later, the risk is still there. It gets discussed again, perhaps with a slightly different shade of amber. Somebody asks whether the mitigation is working. The answer is that it’s being reviewed. Another meeting is arranged. Eventually the project fails, and everybody starts looking backwards through the minutes to establish when the organisation first knew there was a problem.

There’s something strangely reassuring about recording a risk. Once it’s written down, it feels as though we’ve taken responsibility for it. We haven’t necessarily done anything about it, but at least it’s in a spreadsheet somewhere, which I’m sure will be a great comfort when everything goes wrong.

There’s something strangely reassuring about recording a risk. Once it’s written down, it feels as though we’ve taken responsibility for it. We haven’t necessarily done anything about it, but at least it’s in a spreadsheet somewhere

The Titanic has another useful little story tucked away in the lifeboats. The ship carried 20 lifeboats, with stated accommodation for 1,178 people, even though there were more than 2,200 people aboard. The awkward part is that the Titanic actually exceeded the lifeboat capacity required by the regulations at the time, which gives us a slightly uncomfortable question about the difference between being compliant and being safe. They aren’t necessarily the same thing.

There’s another uncomfortable feature of the lifeboat story, which is what happened to different groups of people. “Women and children first” is probably the most familiar phrase associated with the evacuation, although it wasn’t applied in precisely the same way everywhere. The survival figures show a very substantial difference between groups, particularly between First and Third Class passengers.

It’s tempting to reduce that to a simple story about class and location. The people in the cheaper accommodation were further away from the boat deck and faced additional barriers in getting to the lifeboats. But that’s really another example of what happens when we turn a complicated problem into a single explanation. “They were below deck” sounds like sensible reasoning. It just doesn’t explain everything.

But there is a data lesson in that. We’re very fond of averages and aggregate measures because they give us something manageable to report. The organisation as a whole might be doing reasonably well while a particular group is doing very badly. The overall survival rate might look acceptable while somebody has quietly disappeared underneath it. We can produce a perfectly respectable organisational average while the people at the bottom of the distribution are having a very different experience.

It’s one of the reasons I’ve always been suspicious of the phrase “the average patient”, “the average customer” or “the average employee”. I’ve never met one. They tend to exist mainly in spreadsheets. Real people are distributed inconveniently across the data, and sometimes the bit of the distribution we’ve decided not to look at is rather more important than the average.

There’s also an important lesson in the story of the Titanic’s musicians. The band continued playing as the evacuation took place and all eight musicians died. The exact final piece they played has become a matter of historical debate, but the evidence for the band continuing to play during the evacuation is sufficiently established that the story has become one of the most enduring parts of the disaster.

It makes for a heroic story, and there’s no reason to doubt the courage involved. But there’s another way of looking at it. Everyone on the ship had a role, and when things went badly wrong people continued doing the roles they understood. The musicians played. The crew loaded lifeboats. Engineers worked below deck. Officers organised the evacuation. People did what they had been trained and expected to do, even though the circumstances had changed dramatically.

Organisations do this all the time. We design a process for normal circumstances, train people to follow it and then become surprised when they continue following the process when circumstances are no longer normal. The procedure says this, so somebody does this. The dashboard says amber, so somebody carries on. The model has approved it, so somebody assumes it must be safe. The governance framework says that Bob provides oversight, so Bob provides oversight, even though nobody has quite worked out what useful oversight looks like at three o’clock in the morning when the system is making decisions faster than Bob can read them.

We design a process for normal circumstances, train people to follow it and then become surprised when they continue following the process when circumstances are no longer normal.

There’s nothing inherently wrong with having defined roles. Quite the opposite. Organisations need them. But a role can become a trap when it becomes more important to perform the role than to understand what is actually happening around you.

There’s an even more interesting example in the Titanic’s design. The very feature that contributed to its reputation for safety, its system of watertight compartments, was based on assumptions about how flooding would behave. The compartments could contain flooding to a point, but once enough water entered and the ship began to pitch forward, water could flow over the tops of the bulkheads into adjoining compartments.

The design wasn’t simply bad. It was an ingenious safety feature operating beyond the circumstances for which it had effectively been designed.

That feels remarkably familiar in the world of technology and data. A control designed to prevent one type of failure can create vulnerability somewhere else. A security system can make one route into a system harder while making another route more attractive. A governance process designed to prevent reckless decisions can make people believe that anything which has survived the governance process must therefore be safe. A model designed to reduce human bias can reproduce the biases contained in the data used to train it.

The awkward thing about safeguards is that we tend to judge them by whether they exist rather than by what happens when reality behaves differently from the assumptions behind them. The Titanic had watertight compartments. The organisation has governance. The AI system has human oversight. The project has a risk register.

And none of those statements tells us what happens when the thing we designed the safeguard for turns out not to be the thing that actually happens.

The organisation has governance. The AI system has human oversight. The project has a risk register. And none of those statements tells us what happens when the thing we designed the safeguard for turns out not to be the thing that actually happens.

There’s one other Titanic detail that I find particularly interesting because it involves something that was already known before the iceberg was struck. A coal bunker fire had been discovered before the ship sailed and continued to be dealt with during the voyage. It wasn’t a secret fire that nobody knew about. It was a known problem which, at the time, was considered manageable. There has been subsequent debate about whether dealing with that fire affected the ship’s speed or contributed to the eventual disaster, but the important point for this discussion is rather simpler. The problem was there; people knew about it, and it became part of the normal operation of the ship.

That is another familiar organisational pattern. We don’t necessarily ignore problems. Sometimes we manage them. We monitor them. We put a control around them. We agree that they’re being dealt with. Then something else happens and the supposedly manageable problem becomes part of a much bigger failure.

The danger isn’t always the thing nobody knows about. Sometimes it’s the thing everybody knows about that has gradually become normal.

We have a remarkable ability to turn complicated questions into tick boxes. Has the policy been approved? Has the training been completed? Has the risk assessment been carried out? Has the committee reviewed it? Has the appropriate form been signed by someone who is sufficiently senior to make the signature reassuring? All answered yes? Excellent. We’ve complied. Whether we’ve actually solved the problem is, apparently, a separate question.

I’m not suggesting that rules and standards are pointless. They’re not. We need them. Without standards, we’d spend half our time arguing about what good looks like and the other half discovering that everybody had interpreted it differently. But compliance can become an end in itself. Once the requirement has been met, people can stop asking whether the requirement was actually enough.

The Titanic met the applicable requirements for lifeboats. That didn’t make the lifeboats sufficient for what actually happened. There’s a lesson there for pretty much every organisation I’ve ever encountered, although I’m not sure it needs turning into one of those inspirational leadership quotations with a picture of an iceberg underneath it.

And perhaps that’s the uncomfortable bit. We can build policies, processes, registers, committees and frameworks until the organisation is practically drowning in evidence that it takes risk seriously. We can identify the risks, record them, review them and report them. We can even put Bob in the process diagram and call it human oversight.

None of that changes what happens when the warning arrives.

The safeguards only matter if somebody is prepared to act when they’re needed. Once the warning is sitting in front of us, the question isn’t whether we have a process for dealing with it. It’s whether anyone is actually going to change course.

The safeguards only matter if somebody is prepared to act when they’re needed. Once the warning is sitting in front of us, the question isn’t whether we have a process for dealing with it. It’s whether anyone is actually going to change course.

Because you can have all the information you need, all the controls you’ve carefully designed and all the people whose names appear beside the relevant boxes on the governance chart, and still carry on sailing in exactly the same direction.

And by the time you realise where that’s taking you, you may have rather more than a warning to deal with. You may have that sinking feeling