This Way Up

There are some things in life which have become almost impossible to parody because the reality is already doing the work for you. Parcel delivery is one of them.

You order something online. You’re informed once you’ve paid that the parcel will be delivered by the delivery firm Yodvri (a fictional firm for the purpose of this article) and you spend the next few days watching its progress around the country. It arrives at some point when you aren't expecting it, usually after you've received a message telling you that nobody was in, despite the fact that you were sitting six feet from the front door. When you finally find it, it's in the wheelie bin, under the hedge or balanced somewhere on the garden wall in public view, which the delivery driver has apparently decided constitutes a safe place.

Sometimes the box is perfectly fine. At other times it looks as though it has had a difficult afternoon. One corner is crushed, the tape is coming away, it looks like part of it has been gnawed by a small rodent and there's often a muddy footprint across the side. You pick it up to see whether anything has fallen out and usually you can't quite believe somebody thought it was ready to be delivered.

And then you see the words alongside two arrows.

THIS WAY UP

You look at the arrows. You look at the box. You look back at the arrows. And you wonder whether the delivery driver saw them. If they did, they certainly didn’t pay attention with the arrows pointing defiantly toward the ground.

There’s now a bit of a tradition of making jokes about parcel companies which tells us something about how widespread the experience has become. Even if we all have our own preferred villain, there is enough going on for the jokes not to have been invented entirely from thin air.

But let's forget our fictional delivery company, Yodvri, for a moment and imagine that the parcel actually matters (because I’m pretty sure that some delivery firms think it doesn’t).

Suppose there's a bottle inside which has to remain upright. Or something fragile. Or something that’s perfectly safe providing it's handled in a particular way but becomes a very different proposition if somebody drops it, turns it upside down and then throws it over a wall. The contents haven't done anything wrong. The bottle didn't decide to become dangerous halfway through the journey. The vase isn't responsible for being broken. It was the way the parcel was handled that caused the problem. And, if we're honest, we don't normally need to be told that.

If you open the box and find a broken delicate vase inside, you don't pick up the pieces and say, "Well, that's disappointing. Perhaps the vase should have been more resilient. It’s the fault of the vase." No, you ask what happened to the parcel, and why it was handled in the wrong way. You might, of course, look at the box and immediately have a pretty good idea. That's one of the useful things about physical objects. They tend to leave evidence behind. If somebody has handled a parcel badly, you can usually see it. The crushed corners, the broken contents, the leaking bottle, the muddy footprint. By the time it reaches you, the damage, and by implication the journey, is visible.

Data is much more accommodating. You can handle data badly (and many people and organisations do) and still produce a perfectly respectable looking spreadsheet. You can use a measure outside the context in which it was created and get a number. You can combine two things which shouldn't really be combined and get a number. You can misunderstand a definition, fail to document a change, lose some of the provenance or make an assumption about what a field means and still get something which looks remarkably like an answer. There's no smashed vase sitting at the bottom of the spreadsheet to tell you that something went wrong on the journey. That makes the delivery person, the person handling the data, rather important.

If I'm the person collecting it, moving it from one system to another, interpreting it, transforming it or preparing it for somebody else to use, I'm effectively the delivery driver. The data is the contents of the parcel. The instructions about how it should be handled are the governance, definitions, context, provenance, purpose and all the other things which tell me what I can and can't safely do with it. And the person receiving it is, in some cases, the customer. In other cases a parcel may be delivered by one company to a depot, and then another from picks it up to deliver to the customer

If I'm given a dataset in a “delivery depot” and I know what it should look like, I may spot something wrong. I might notice that the numbers don't make sense. I might know that two categories have been mixed up. I might wonder why something which should be there isn't there. If I've got enough context, I can look at the parcel and say, "Hang on, this doesn't look right." I can even send it back. It's not perfect, because people make mistakes and sometimes people don't notice mistakes, but there is at least the possibility of somebody in the depot checking the state of the box, or the customer opening the box and looking at what's inside before they use it.

We've been doing this with data for years. We put checks into processes. We document definitions. We try to understand where information came from. We ask whether it's appropriate for the purpose. We have data quality rules and governance arrangements and people whose job is, at least in theory, to ask awkward questions before something gets sent on its way. But despite all of this we also get used to ignoring some of those things. And that's probably because data is so forgiving.

If a parcel arrives upside down and the contents are still intact, you might get away with it. Do it again and the same thing happens. And again. After a while, somebody might reasonably conclude that the words THIS WAY UP aren't actually all that important. They’re still there. Nobody has removed them. They're just no longer being taken seriously.

The same thing happens with data. A definition isn't quite followed, but nothing obvious goes wrong. A quality check is skipped because everyone is busy, and the report still goes out. A dataset is used for something slightly different from the purpose for which it was collected, but the results look plausible. A workaround becomes part of the process because it was only supposed to be temporary, and then three years later nobody remembers why it was ever considered a workaround. Eventually, the handling becomes normal. That's probably one of the more dangerous things about data governance. Not that there aren't rules, but that we can break them repeatedly without anything immediately exploding. And before long, the arrows become decoration.

Then we introduce AI into the process and something rather interesting happens. Without realising it, we change the delivery driver.

Instead of a person taking the parcel, reading the instructions, making some decisions about how to handle it and eventually handing it to another person, we've got a machine which can take an enormous number of parcels, process them at speed and deliver something at the other end without necessarily having any understanding of what's inside.

That sounds like a useful development, and it can be. The problem is that the delivery driver has become considerably faster without necessarily becoming any better at understanding the instructions. And then there's the customer. With AI the customer isn't always a person. Sometimes the immediate recipient of the output is another AI system or agent

The first system takes the data, processes it and produces an output. Another system receives that output, processes it again and produces something else. An agent passes something to another agent. A model generates a summary which is fed into another model. A system makes a recommendation which becomes an input into the next stage of a process. The parcel is being delivered, opened and passed on without anybody necessarily standing there saying, "Hang on a minute, this bottle is broken." And that's a very different situation from a human receiving the parcel, because a human might notice. Not always, obviously. We can all be spectacularly bad at noticing things when we're in a hurry, and there's no shortage of evidence that people will happily accept information because it confirms what they expected to see. But humans at least have the possibility of context. We might know that a particular number looks odd. We might remember what happened last month. We might know that a particular dataset isn't normally used that way. We might simply have enough experience to think, "That doesn't look right."

AI doesn't necessarily have any of that. It can be remarkably good at producing something which looks as though the parcel arrived in perfect condition.

We often talk about AI hallucinating, as though the main problem is that the machine occasionally invents something. Of course that matters, but there's something more mundane going on here which may be more difficult to deal with. AI can take something which has been badly handled, do exactly what it has been asked to do with it, and produce an output which looks like it’s right.

Imagine the parcel contains a vase. Somewhere along the journey, somebody has ignored the instructions, the box has been turned upside down and the vase has fallen over and broken. A human receiving the parcel is likely to recognise what has happened. They know there was supposed to be a vase in the box, and what they're looking at is a broken vase. They may not know exactly where in the journey it happened, but they can see that something has gone wrong.

An AI system may see something quite different. It may receive an image of the contents and identify them as pieces of ceramic arranged in a particular pattern. Perhaps it describes them as mosaic pieces. From its perspective, that might be an entirely reasonable description of what is in front of it. It doesn't necessarily know that these pieces were once a vase, that the vase was supposed to arrive intact, or that somebody accidentally broke it somewhere along the delivery route. The broken vase has effectively become the new truth. The AI hasn't necessarily made a mistake in identifying what it can see. The mistake happened earlier, when the parcel was handled incorrectly. But because the AI doesn't have the context to understand that something has gone wrong, it can happily take the damaged contents, give them a new label and send them on to the next customer. The next system then receives "mosaic pieces", rather than a broken vase. And it doesn't know to ask why, so it simply carries on.

That's the bit that is more concerning than the idea of AI suddenly producing something completely bizarre. The really interesting failures may be the ones where the AI produces a perfectly plausible description of something that has gone wrong, without having any idea that it has gone wrong at all. It doesn't know that the box was upside down through part of its journey. So it doesn't necessarily know that the contents have been damaged.

And if the next AI system in the chain receives the output, it may have no reason to question it either. As far as it is concerned, another parcel has arrived. The label says what it says. The contents look like something it can work with. So it carries on. But by this point the damage has already happened.

By the time a human eventually sees the final output, the original parcel might have travelled through several systems. The context which would have allowed somebody to spot the problem may be long gone. The original definition may not have travelled with the data. The reason a particular field was collected may have disappeared somewhere along the way. The caveat that somebody knew about but never documented might have vanished completely.

And the final answer can still look very good.

The better AI becomes at producing polished outputs, the easier it may be to overlook the fact that something went wrong much earlier in the delivery process. If I receive a parcel with a broken vase sticking out of the side, I'm probably going to question it. If I receive a beautifully presented answer explaining exactly what the data supposedly tells me, I'm much more likely to read it. (Especially if I wanted the answer in the first place.) And if that answer has already been consumed by another system, there may not even be a human equivalent of the customer standing at the door with the broken vase in their hands. The next process simply accepts the delivery and carries on.

This doesn't mean that AI is inherently bad at handling data. Quite the opposite. It can be an extraordinarily powerful way of moving information through organisations and doing things with it that would have taken people enormous amounts of time. But the more capable the delivery mechanism becomes, the more tempting it is to assume that the handling problem has been solved simply because the delivery is faster. If the data requires a particular context, the AI still needs that context. If a definition matters, it still matters. If information shouldn't be combined with something else, the fact that a machine can combine it doesn't mean that it should. If there are restrictions on how information should be used, putting an AI system in the middle of the process doesn't make those restrictions disappear.

The arrows are still on the box. But unfortunately we seem to be distracted by the speed of the van.

There's a lesson in that which goes beyond AI. We have spent years making data move faster through organisations. New systems, better integration, dashboards, automation, APIs, data platforms and now AI. Each development promises to make the journey quicker and easier. But there’s a danger in concentrating so heavily on the delivery that we stop asking what happens to the parcel on the way. With a human delivery driver, there is at least somebody who can potentially read the label. There is somebody who can decide that perhaps the box shouldn't be thrown over the wall. There is somebody at the other end who can open it and discover that the contents have been damaged.

When the delivery driver is AI, and the immediate customer is another machine, that human pause can disappear completely. The parcel still gets delivered, and the next one. And everything appears to be working. Until eventually a human opens the final box and wonders what on earth happened to the contents. By then, of course, the delivery driver has moved on. The original box may have disappeared. Nobody can quite remember who handled it first and the words THIS WAY UP are sitting there, printed in large letters, pointing towards the floor.

They were never particularly difficult instructions. We’ve just become used to thinking that because the parcel usually arrived intact, they probably weren't all that important. And that's the mistake we're now in danger of making with data and AI. Assuming that because the technology can handle the parcel, it must also understand how the parcel needs to be handled. But those are two very different things.

The contents of the box haven't changed, and neither has the responsibility that comes with handling them properly. What we've done is build an increasingly sophisticated delivery service which can move information faster, pass it between more systems and produce an answer before anyone has had time to ask whether the parcel arrived in one piece. A human might eventually open the box and realise that the vase has become mosaic pieces. Another AI system might simply accept the mosaic pieces, give them a perfectly reasonable name and send them on their way. By then, the original mistake has become part of the next delivery, and perhaps the next one after that. Nobody has necessarily done anything deliberately wrong. Nobody has necessarily even realised that anything has gone wrong. The parcel has simply kept moving.

That's why the old instruction printed on the side of the box deserves a little more attention than we usually give it. It was never there to make the cardboard look more interesting. It was there because somebody knew that what was inside mattered, and that the way it was handled could determine what eventually arrived at the other end. We can make the delivery driver faster, give it better technology and remove more and more people from the journey, but we can't remove the need to understand what's in the box or how it needs to be handled. At some point, somebody still has to know whether the vase is a vase or a pile of mosaic pieces. And if nobody bothers to look until the parcel reaches the customer, it may be too late to work out where the damage happened.

Which brings us back to those two rather unremarkable arrows printed on the cardboard, pointing upwards. They weren't put there because the person who packed the box had nothing better to do. They were there because somebody knew what was inside, knew what could happen if it was handled badly, and thought it worth telling the person carrying it.

THIS WAY UP.