Looking at the Right Thing

A while back I wrote an article called When Perception and Reality Clash in Data Analysis. The starting point of this was the familiar phrase “perception is reality”. It sounds profound, but it’s also rather misleading. Perception is how we interpret what we see. Reality is what’s actually happening, whether we see it or not.

That difference in definition becomes important when we’re working with data because it’s very easy to confuse the thing we’ve measured with the thing we’re actually interested in. We can calculate something accurately, produce a beautiful graph and write a convincing explanation, only to discover that we weren’t looking at the right thing in the first place.

That was the point of my original article. At the time, I was mainly looking at statistical concepts such as p-values, confidence intervals, statistical power, sensitivity, specificity and Type 1 and Type 2 errors. Those concepts are still important, but the world of data analysis has changed considerably since I wrote it. We now have AI systems that can analyse datasets, find patterns, produce charts, write reports and even suggest explanations for what the data appears to be telling us.

That's potentially very useful. It also makes the question of whether we're looking at the right thing rather more important. After all, an AI system can process the wrong data just as efficiently as the right data. It can analyse the wrong question faster than any human. It can produce a beautifully written explanation of something that isn’t actually what we needed to understand. Said another way, having better analytical tools doesn’t necessarily mean we’ll get better answers. Sometimes we just get the wrong answer more quickly.

having better analytical tools doesn’t necessarily mean we’ll get better answers. Sometimes we just get the wrong answer more quickly.

The problem starts long before we get to AI, or even to statistics. Data for analysis doesn’t simply appear. Somebody decides what to collect, how to define it, when to collect it and who should be included. Data has provenance, and those decisions can have a huge effect on what we eventually see.

Take something as apparently straightforward as NHS waiting times. If somebody says that the average waiting time has increased, I would immediately want to know what they mean by “waiting time”. Is it from referral to first appointment? Referral to treatment? The time somebody has been waiting for a particular procedure? What happens when somebody cancels an appointment? What happens when they move between providers? The number might be completely accurate, but if I’m interested in the experience of patients and I’m measuring something else, I’m not looking at the right thing.

This is one of the themes I explored in When Perception Gets Ahead of Reality. We can become so accustomed to treating a measure as though it represents the thing itself that we forget it’s only a representation. The number isn’t waiting time. It’s a measurement of something we’ve decided to call waiting time. That might sound like semantics, but it can make a considerable difference to the conclusions we draw. It also explains why data quality is more complicated than checking whether a spreadsheet contains errors.

In A Good Data Die, I looked at what happens when organisations retain data simply because they can, even when it has become old, irrelevant or inappropriate for the purpose for which it’s subsequently used. The fact that information exists doesn’t mean it’s useful information. That becomes particularly relevant with AI. We’re increasingly told that AI can make sense of vast quantities of data, which is true. But quantity isn’t the same thing as quality, and processing more data doesn’t solve the problem if we’re looking at the wrong thing. In fact, it can make the problem harder to see.

quantity isn’t the same thing as quality, and processing more data doesn’t solve the problem if we’re looking at the wrong thing.

One of the best demonstrations of this is Simpson’s Paradox.

Suppose we’re comparing two treatments. We look at two groups of patients separately and discover that Treatment A has a better success rate than Treatment B in both groups. It would seem fairly obvious that Treatment A is the better treatment.

Then we combine the groups and discover that Treatment B has the better success rate overall.

The arithmetic can be completely correct, and that's the fascinating part. The apparent contradiction happens because the groups aren't the same size or composition.

Perhaps Treatment A has mainly been given to patients with a better prognosis, while Treatment B has been used more frequently with patients who are much harder to treat. Once we combine the groups, the different proportions can reverse the overall result.

So which answer is right? The answer is that both sets of calculations can be right. The problem is that they're answering slightly different questions.

If we only look at the overall result, we might conclude that Treatment B is better. If we only look at the two individual groups, we might conclude that Treatment A is better. To understand what’s actually happening, we need to look at why the groups differ. We need to look at the right thing.

This is more than an interesting statistical curiosity. It’s a warning about how we use analytical tools, and particularly AI.

Imagine giving the combined dataset to an AI system and asking it to determine which treatment is most effective. It may correctly identify that Treatment B has the better overall success rate. It could produce a graph, calculate the percentages and write a convincing explanation of the result. Nothing necessarily went wrong. But unless we ask it to examine the individual groups, it might never discover that Treatment A performs better within both of them. The machine has analysed the data we gave it; the problem is that we may not have been looking at the right thing.

The machine has analysed the data we gave it; the problem is that we may not have been looking at the right thing.

This is also why I’m wary of the way statistical significance is sometimes interpreted. Suppose we conduct a clinical trial and obtain a p-value of 0.03. It’s very tempting to say that the treatment has been shown to work because the result is statistically significant. But statistical significance isn’t the same thing as importance.

If a drug produces a tiny reduction in cholesterol and we have a sufficiently large sample, the difference might be statistically significant. That tells us something about the evidence against the null hypothesis, but it doesn’t tell us whether the effect is large enough to make a meaningful difference to patients.

We need to look at the size of the effect, the confidence interval and the wider clinical context. Most importantly, we need to ask whether the result matters outside the spreadsheet. Once again, we need to look at the right thing.

Confidence intervals are useful because they move us away from the rather simplistic idea that statistical analysis produces a yes or no answer. If a treatment reduces cholesterol by 10% and the confidence interval is from 5% to 15%, we have considerably more information than we would get from simply being told that p is less than 0.05. We can see the estimated size of the effect and something about the uncertainty surrounding it.

But even here, there’s a temptation to turn an uncertain result into something that sounds certain. We often want statistics to tell us what is going to happen, but they can’t do that. They can help us estimate what might happen and quantify some of the uncertainty, but there will always be things we don’t know. That isn’t a failure of statistics; it’s the nature of reality.

We often want statistics to tell us what is going to happen, but they can’t do that ... That isn’t a failure of statistics; it’s the nature of reality.

The same applies when a study produces a result that isn’t statistically significant. A p value of 0.08 doesn’t prove that there’s no effect. Perhaps the study was too small. Perhaps the data was highly variable. Perhaps the effect is real but relatively modest.

That’s where power comes in. A study with low power may simply fail to detect something that’s really there. Saying “we didn’t find a statistically significant effect” is very different from saying “there is no effect”. Once again, we have to make sure we’re looking at the right thing.

With AI we’re no longer simply asking software to perform a calculation. We can ask an AI system to interpret the results for us. We can give it a dataset and ask it to identify trends, find anomalies, explain relationships and recommend conclusions. And that can be enormously helpful when the person using it understands the problem. It's much more questionable, though, when we assume that the system's ability to produce an answer means it understands what we're trying to find out.

I wrote about this in Going Beyond the Tools to Technique. There’s a big difference between knowing how to use a tool and knowing what to do with it. You can know every function in Excel and still be a poor analyst. You can create sophisticated Power BI dashboards and still be presenting the wrong information. You can use AI to generate an impressive statistical analysis and still have no idea whether the analysis answers the question you actually needed to answer. The technology doesn’t remove the need for analytical judgement. It makes it easier to hide the absence of it.

You can use AI to generate an impressive statistical analysis and still have no idea whether the analysis answers the question you actually needed to answer.

There’s an important distinction here between an obvious AI mistake and a subtle analytical mistake. If an AI system invents a statistic or cites a paper that doesn’t exist, we have a problem that’s relatively easy to identify once we check it. The more worrying errors are the ones that look perfectly reasonable. The system might use an appropriate statistical method on inappropriate data. It might identify a genuine correlation and interpret it as causation. It might analyse one population and assume that the findings apply to another. It might combine groups that shouldn’t be combined. It might even produce the correct answer to the wrong question. That’s much harder to spot because nothing necessarily looks broken. The numbers might add up, the graph might look good, the prose might be excellent and the conclusion might sound convincing, yet we still might not be looking at the right thing.

There’s another human problem that AI can make considerably easier: confirmation bias. If I already believe that waiting times have increased, I can ask an AI system to analyse my data and explain why waiting times have increased. It will probably give me an explanation. But I haven’t actually asked it to establish whether waiting times have increased. I’ve assumed the answer and asked it to explain it.

That’s the problem I explored in When Decisions Come First. The tendency to decide what we believe and then find evidence to support it isn’t new. Researchers, managers, politicians and organisations have been doing it for years. AI simply makes it much easier. We can now generate a plausible justification for our existing view almost instantly. That means the wording of the question becomes increasingly important. “Why have waiting times increased?” and “What has happened to waiting times?” aren’t the same question. The first assumes something has happened, whereas the second asks us to find out what happened. If we’re serious about analysis, we need to be very careful about the assumptions we build into our questions.

If we’re serious about analysis, we need to be very careful about the assumptions we build into our questions.

What I wrote in How Guesses Become Gospel becomes relevant here. An estimate can appear in one report, get repeated in another and eventually become accepted as fact. Once something has been repeated often enough, people can forget that it was originally an estimate or an assumption. AI doesn’t make that problem disappear. If anything, it can amplify it. An AI system has access to vast quantities of information, but information that has been repeated many times isn’t necessarily information that’s true. A mistake doesn’t become correct because it appears in thousands of documents. Sometimes it just becomes familiar. And familiarity can be remarkably persuasive.

There’s a similar problem with the assumption that computers are objective. We often trust a computer-generated number more than we trust somebody’s opinion because the number appears to be objective. But somebody still decided what the computer should measure. Somebody selected the data. Somebody decided what counts as success. Somebody chose the variables. Somebody designed the model. And somebody eventually decides what the output means.

AI doesn’t remove human judgement from the process; it just moves it around. That’s particularly important when we start using predictive systems. In The Dreamer and the Oracle, I looked at the difference between generative and predictive AI and the way predictions can influence the future they’re supposed to predict.

Imagine an algorithm identifies somebody as being at high risk of hospital admission. Because of that classification, they receive additional monitoring and support. Their chance of being admitted then falls. Was the prediction wrong? Perhaps not. But the prediction has influenced what happened. The system didn’t simply observe reality. Its output became part of the decision-making process and helped change reality. This is one of the reasons that simply reporting the accuracy of a predictive model isn’t always enough. We need to understand what happens after the prediction has been made. Once again, we need to look at the right thing.

simply reporting the accuracy of a predictive model isn’t always enough. We need to understand what happens after the prediction has been made.

The same issue is appearing with evidence itself. In When Seeing is No Longer Believing, I looked at how AI-generated images, video and audio are making us reconsider what we mean by evidence. We used to assume that seeing something was reasonably strong evidence that it had happened. That assumption is becoming much harder to sustain.

We’re now dealing with a similar issue when AI produces written analysis. A fluent explanation can feel authoritative. A well-designed graph can look convincing. A detailed report can give the impression that somebody has thought carefully about the evidence. But presentation isn't evidence. A polished explanation can still be wrong, a sophisticated model can still be inappropriate and a beautifully designed dashboard can still be showing us the wrong thing.

This brings me back to what I think is the most important part of the whole discussion.

I don’t think we should respond to AI by refusing to use it. I use it myself, and I find it extremely useful. It can help me explore ideas, challenge my thinking, find alternative approaches and work through problems much faster than I could do alone. But I need to understand enough about the subject to recognise when the answer doesn't make sense. That's why The Responsibility to Think remains relevant. We can use technology to help us think, but we shouldn’t use technology as an excuse not to think. If an AI system gives me an answer and I accept it, I’m still responsible for what I do with that answer. The machine isn’t.

We can use technology to help us think, but we shouldn’t use technology as an excuse not to think. If an AI system gives me an answer and I accept it, I’m still responsible for what I do with that answer. The machine isn’t.

That responsibility becomes particularly important when the consequences of getting something wrong are serious. Healthcare, recruitment, financial decisions, education and public services all involve situations where a misleading analysis can affect real people. The answer isn’t to ask whether AI is right or wrong. It’s to ask whether we’re asking the right question, using the right data and interpreting the result in the right context. That sounds less exciting than talking about artificial general intelligence or machines replacing human beings, but it’s where many of the practical problems actually sit.

We've become very good at collecting data, processing it and asking machines to analyse it. The danger is that we start to believe that because we can analyse something, we must understand it. But a lot of the time we don’t.

A p value doesn’t tell us whether a treatment matters to a patient. A confidence interval doesn’t remove uncertainty. A statistically significant result doesn’t automatically represent an important result. A high accuracy score doesn’t necessarily mean that a predictive model is useful in practice. And a large language model doesn’t magically know which questions we should have been asking.

That’s why I think the title of this article is more important than the phrase “perception is reality”.

When we analyse data, we need to ask ourselves whether we’re looking at the right thing. When we use AI, we need to ask whether we’re looking at the right thing. When somebody presents us with a statistic, a graph, a prediction or an apparently convincing explanation, we need to ask whether we’re looking at the right thing.

Sometimes the answer will be yes. Sometimes we’ll discover that the interesting story was hidden underneath the headline number. Sometimes Simpson’s Paradox will be waiting for us. Sometimes we’ll find that the statistically significant result isn’t clinically important. Sometimes the apparently negative result is simply the consequence of an underpowered study. Sometimes the AI has produced a perfectly coherent answer to a question we didn’t really mean to ask.

So before we trust the numbers, the graph, the model or the AI, perhaps the first question we should ask is a very simple one.

Are we looking at the right thing?