Obsession with the How

Over more than 30 years of working with statistics, analysis, research and information, I’ve probably made just about every mistake it’s possible to make with data, and I’ve certainly seen plenty of other people make them too. One thing I've seen repeatedly is that analysis can be completely correct and still be completely unhelpful. I’ve seen technically sound analysis, beautifully presented charts and carefully calculated statistics that looked impressive but failed to answer the question anyone actually needed answering. Sometimes the problem was the question, sometimes it was the data, and sometimes nobody had stopped to ask what the numbers were really telling us. The longer I’ve worked with data, the more convinced I’ve become that getting the calculation right is only the beginning. The harder part is knowing what you’re trying to find out, whether you’ve got the right information to find it out, and what to make of the answer when you finally get it.

That’s probably why so much of what I’ve written over the last couple of years, particularly about AI literacy and data literacy, has kept coming back to the importance of asking the right question. I’ve always been convinced that getting the question right is harder than getting an answer. An answer can be completely correct and still be of very little use if it answers a different question from the one we actually needed to ask. And of course, AI has made that problem much more obvious because it can produce an apparently convincing answer to almost anything in seconds. But I don't want to make this article about AI because the underlying problem is much older. We’ve always been rather easily impressed by answers that look authoritative, contain lots of numbers or have been produced using sophisticated methods. I’ve certainly been guilty of that myself at times. Something can look clever, complicated and convincing without actually telling us anything particularly useful.

Having spent so much time thinking and writing about getting the question right, I’ve realised that I’ve probably given less attention to what happens next. While important, getting the right question is only the beginning. We still have to understand the data we’re using, decide whether it’s appropriate for the question we want to answer, think about how it’s been collected and coded and understand its limitations. We then have to choose an appropriate way of analysing it and, perhaps most importantly of all, work out what the result actually means. It's that working out that I want to spend more time exploring over the next few articles. But before I get to discussing the statistics themselves, there’s another issue I think is worth considering - the way we’ve come to think about what it means to be good at working with data.

While important, getting the right question is only the beginning. We still have to understand the data we’re using, decide whether it’s appropriate for the question we want to answer, think about how it’s been collected and coded and understand its limitations. We then have to choose an appropriate way of analysing it and, perhaps most importantly of all, work out what the result actually means

Nowadays there seems to be an increasing fascination with the mechanics of analysis. If you look at discussions about becoming a data analyst or moving into data science, the conversation often starts with software and technical skills. Learn SQL. Learn Python or R. Learn Power BI or Tableau. Learn how to manipulate data, build dashboards, automate processes and create visualisations. And now there’s another skill to add to the list, learning how to prompt to get the best results from AI. Now, there’s nothing intrinsically wrong with any of that. If you’re going to work with data, you need to be able to use the tools available to you, and I wouldn’t want an analyst working for me who couldn’t work with modern analytical software.

What concerns me is the order in which we tend to approach this. We seem remarkably keen to decide what somebody needs to learn before establishing what they need to understand. I recently came across a discussion on LinkedIn about whether the first thing somebody wanting to become a data analyst should learn is SQL or Power BI. It was a perfectly reasonable question, and both are useful skills, but I found myself wondering why we were choosing between tools before talking about the thing that actually matters - understanding the data and the problem it’s supposed to help us solve. You see, knowing how to manipulate a dataset isn’t the same as knowing whether the dataset is appropriate, and being able to produce a sophisticated visualisation doesn’t mean that we necessarily understand what we’re looking at.

That distinction is important because analytical work involves making decisions long before we get to the point of pressing a button, or writing a prompt. We have to decide what information matters, which variables are relevant, how things should be defined and grouped, what comparisons are meaningful and what the resulting numbers might actually tell us. Some of those decisions might be made in SQL, some in Power BI, some in Excel and some in a piece of code, but the software doesn’t make them analytical decisions simply by providing a way of carrying them out. The tools are ways of doing things; the "How" of data analysis. They don’t, and can't, tell you whether the thing you’re doing is sensible (the "Why".)

There’s a related point here that I’ve seen repeatedly in my career. People tend to talk about analytical work as though it starts when you open the software, when in reality much of the important thinking has already happened before you get anywhere near it. If you’ve decided which data to use, what you’re trying to measure and what you’re going to compare, you’ve already made analytical judgements. The software is simply the means by which you carry them out. That’s why I’m less interested in whether somebody’s first choice is SQL, Power BI, Python or R than I am in whether they understand what they’re going to do with the information once they’ve learned how to use the tool.

I’m less interested in whether somebody’s first choice is SQL, Power BI, Python or R than I am in whether they understand what they’re going to do with the information once they’ve learned how to use the tool.

Power BI can do an awful lot for you, but it can’t tell you whether what you’re asking it to do makes sense. It’ll calculate a measure because you’ve told it to. It’ll produce a percentage without questioning whether the denominator’s appropriate, and it’ll draw a trend without knowing whether a change reflects something happening in the world or a change in the way the data was collected, defined or coded. You can produce a dashboard that looks extremely professional from data that was never appropriate for the question you wanted to answer. In fact, the better the dashboard looks, the easier it can sometimes be to overlook the problem because people tend to associate presentation quality with analytical quality.

That brings me to something I think should come much earlier in analytical training. Before somebody starts deciding whether SQL, Power BI, Python or R should be the first thing they learn, I’d want them to spend some time understanding the thing they’re actually going to be analysing. I’d start with the data itself.

That understanding goes well beyond knowing what the columns are called. You need to know where the data came from and how it was produced. You need to understand what the variables actually represent, how they’ve been defined and coded, what their data types are and what level of detail’s been retained. You have to have some idea of the provenance of the information and whether different sources have used the same definitions. You should also think about validity and whether the data in front of you actually represents what you believe it represents. None of this is particularly glamorous compared with building an interactive dashboard, but it’s much closer to the substance of analysis and vitally important.

Data type is a good example. It’s not enough to know whether something’s been stored as a number or as text. We need to understand the nature of the variable itself. Is it nominal, ordinal, interval or ratio? Is it discrete or continuous? Those distinctions affect what we can legitimately do with the data and how we should interpret the result. A value being stored numerically doesn’t automatically make it a numerical measurement in the statistical sense, just as storing something as text doesn’t tell us everything we need to know about what it represents. The computer’s interested in the format. The analyst needs to be interested in the meaning.

The computer’s interested in the format. The analyst needs to be interested in the meaning.

Granularity presents a similar issue. Data’s collected and stored at particular levels of detail, and those levels have consequences for what we can subsequently learn from it. Sometimes information’s collected at a very detailed level and then aggregated. Sometimes the detail was never collected in the first place. Either way, the level at which information exists needs to match the question we’re asking. It’s surprisingly easy to take perfectly accurate information, aggregate it to a particular level and then try to use it to answer a question that requires information at a different level.

Validity takes us further into the question of what the data actually means. We need to understand the definitions behind the variables and the rules governing what can legitimately be recorded. We need to know whether the way the information’s been collected was appropriate and whether the values we’re seeing are meaningful in the context in which we intend to use them. A value can look perfectly plausible and still be wrong for the purpose of the analysis. That’s a different problem from simple inaccuracy and one that can be much harder to spot.

Coding matters for much the same reason. Categories, missing values and other information are represented according to conventions, and those conventions need to be understood rather than simply accepted. Codes can look like numbers without actually being measurements, and two datasets can use apparently similar coding systems to represent different things. If we don’t understand how the data’s been coded, we can easily make assumptions that find their way into the analysis without anybody noticing.

All of this makes the idea of data quality rather more complicated than the familiar checklist of completeness, accuracy, consistency and currency. Those things matter, of course, but I’ve never found the question “Is this good quality data?” particularly helpful on its own. The question I’m more interested in is “Good enough for what?”

I’ve never found the question “Is this good quality data?” particularly helpful on its own. The question I’m more interested in is “Good enough for what?”

That distinction matters because data quality isn’t an absolute property of a dataset. It depends on the "What". What do you want to use the data for? What question do you want to answer? Information can be perfectly adequate for one purpose and unsuitable for another without anything about the underlying data having changed. The level of accuracy, precision, detail or timeliness that’s needed depends on the decision being made and the consequences of getting it wrong.

Precision’s a good illustration of this. There are situations where a broad estimate is entirely adequate and others where a relatively small difference can have serious consequences. There’s little point insisting on unnecessary precision simply because a computer’s capable of providing it, just as there’s little sense in accepting a rough approximation when the decision depends on a small difference. The question isn’t whether the data could be more precise. It’s whether it’s precise enough for the purpose for which we intend to use it. As I say in my training "You need to know WHAT you're collecting and HOW you're collecting it."

This is one of the many reasons I get nervous when people start analysing data simply because it happens to be available. We’ve got enormous amounts of information at our disposal and increasingly powerful tools for doing things with it, which makes it very tempting to start with the data rather than the question. Put a dataset into Power BI, drag a few fields onto the screen, look for something interesting and, before long, there’s a chart suggesting that something’s changed. Someone then notices a difference between two groups and decides it needs investigating. Perhaps it does even though we will always find things through chance. The more comparisons we make, the more chance relationships we will find. And perhaps the data was never collected for that purpose, the definitions don’t quite match, the level of detail’s wrong or the measure itself doesn’t mean what we’ve assumed it means.

The temptation to start with the data we have, rather than a question, is understandable because finding something’s much easier than deciding whether what we’ve found is worth finding. A large dataset can give the impression that there must be an answer hidden somewhere inside it, waiting for the right combination of filters, charts and statistical techniques to reveal it. Sometimes there is. Sometimes we’re simply finding patterns because we’ve gone looking for them. The availability of data in itself doesn’t create a useful question, and the ability to analyse something doesn’t make it appropriate to analyse.

The availability of data in itself doesn’t create a useful question, and the ability to analyse something doesn’t make it appropriate to analyse.

This is where the obsession with the mechanics starts to become a problem. We can spend a great deal of time teaching people how to interrogate databases, write code / prompts and construct visualisations while giving much less attention to the judgement that has to sit around all of those activities. A technically capable person can produce an answer very quickly. That doesn’t necessarily make them a good analyst.

AI makes this particularly interesting because it’s getting very good at the mechanics as well. It can write SQL, produce Python and R code, manipulate data, suggest visualisations and carry out statistical calculations. That can save enormous amounts of time and, used properly, can make analytical work accessible to people who might previously have struggled with some of the technical barriers. But it also means that knowing how to make a computer perform a particular task is becoming a less useful way of judging whether somebody understands what they’re doing.

If I ask an AI system to calculate an average, it’ll probably manage it perfectly well. What it can’t establish simply from seeing a column is whether an average’s an appropriate summary, whether the values have the right meaning, whether the variable’s at the right level of measurement or whether the data’s suitable for the question I have in mind. The code can be flawless and the answer can still be meaningless. The same applies to SQL, Power BI, Excel, R, Python or any other analytical tool. The software can perform the operation. It can’t take responsibility for the judgement behind it.

That’s why its essential that we broaden what we mean when we talk about analytical skills. Technical competence matters, but so does the ability to question the data, understand its limitations and recognise when an apparently straightforward calculation’s answering a question that we didn’t really mean to ask. Someone who can build an impressive dashboard but doesn’t understand the data behind it isn’t necessarily a better analyst than someone who can’t build the dashboard at all. The dashboard’s only useful if the thinking behind it’s sound.

Someone who can build an impressive dashboard but doesn’t understand the data behind it isn’t necessarily a better analyst than someone who can’t build the dashboard at all. The dashboard’s only useful if the thinking behind it’s sound.

And even when we’ve done all of that properly, there’s still another stage that’s often overlooked. Suppose we’ve asked the right question, understood the data, established that it’s appropriate, checked the definitions and coding, considered the level of detail and selected a suitable analytical method. We carry out the analysis and get a result. What then?

Its here where I want to concentrate over the next few articles, because producing statistical results has become remarkably easy while understanding them remains surprisingly difficult. A computer can calculate a p value almost instantly. It can produce a confidence interval, correlation coefficient or regression coefficient without us having to understand the mathematics involved in the calculation. That’s enormously useful, but it can also encourage us to think that the difficult part of statistics is obtaining the number. In practice, the harder question’s often what the number means and what we should do with it.

The p value seems a good place to start because it’s one of the statistics I’ve seen misunderstood most often. The familiar 0.05 threshold has acquired almost mythical status, as though a result with a p value of 0.049 somehow belongs to a completely different category from one with a p value of 0.051. We see statements suggesting that a result below 0.05 proves something, while a result above it means that nothing’s been found. Neither interpretation’s particularly helpful.

There are also some much more basic misunderstandings about what a p-value actually represents. It isn’t the probability that the result happened by chance, and it isn’t the probability that a hypothesis is true. Nor does it tell us whether an effect is large, important or useful in practice. Those distinctions matter, particularly when statistical results are being used to inform decisions rather than simply to populate a research paper.

That’s the territory I want to explore next, and I want to do it in a practical way. Most people who use statistics don’t need to become mathematical statisticians. They need to be able to look at a result, understand what it’s saying, recognise what it isn’t saying and decide how much weight they should give it. Being able to explain a statistical result in ordinary language is often far more useful than being able to reproduce the formula that generated it.

Most people who use statistics don’t need to become mathematical statisticians. They need to be able to look at a result, understand what it’s saying, recognise what it isn’t saying and decide how much weight they should give it.

That, really, is what’s brought me back to the title of this article. There’s nothing wrong with learning SQL, Python, R, Power BI, Tableau or any of the other tools that have become part of modern analytical work. I’ll continue to use them myself and encourage other people to learn them. The problem comes when we confuse knowing how to use a tool with knowing how to analyse data. The tools help us do the work, but they don’t decide what the work should be, whether the data’s suitable, whether the analysis makes sense or what the result means.

Perhaps we need to spend a little less time asking what tool somebody should learn next and a little more time asking whether they understand the problem they’re trying to solve. Perhaps we should be as interested in how somebody thinks about data as we are in whether they can manipulate it. And perhaps, as the technology becomes increasingly capable of dealing with the mechanics for us, the ability to question, interpret and make sense of what comes out the other end will become more important rather than less. We’ve become very good at teaching people how to get answers from data. I think we could do with spending rather more time teaching them how to understand those answers, rather than maintaining our obsession with the how.