The Proof of the Pudding
I have a friend that I have known for many years. They are always after a bargain and scour the internet and the markets for a ‘deal’. Not so long ago they discovered the ‘discount supermarket’ i.e. the Aldi and Lidl variety. As would be expected they quickly became a discount supermarket evangelist, and anything that I would eat or drink would ‘be better and cheaper’ if it came from their favourite store.
Now, this post isn’t about whether Tesco or Aldi is a better supermarket, or sells better / cheaper food, so in order to protect the innocent my story will now concern the fictional supermarkets of Alco and Tesdi.
One day my friend asked me to sample a rice pudding by Alco.
“Isn’t that the best thing that you’ve ever tasted? Alco’s puddings are far superior to Tesdi’s!!”
“Actually”, I said, “I don’t think there’s any noticeable difference”
“Prove it!”, they responded.
Seems reasonable, doesn’t it. Why shouldn’t my friend want me to prove that my preference was as good as his? So I have two questions
1. Is it possible for me to prove that my choice of pudding tastes as good as his?
2. Is it possible for him to prove that his pudding choice tastes better than mine?
If you know me at all, or have listened to any of my talks, or have been taught be me, you will know that the answer to both of these questions is a resounding “No!” .
That might seem surprising. Surely we must be able to prove one way or another, otherwise we can’t know anything with certainty? Welcome to scientific thinking....
To understand our problems we need to understand a little about the philosophy of science, and the scientific method. Firstly, it is not possible to prove a negative (e.g. these puddings don’t taste significantly different). This is known as the black swan scenario. To prove there was no significant difference in the colour of swans ( i.e. they were all white), I would need to produce every swan in the world, and show that they were all white (and I would always be accused of missing the ’black swan’ hiding somewhere). To demonstrate that every swan in the world is not white, I would simply have to produce one black swan. The same is true of our puddings, I cannot ever prove everyone, on average, thinks there is no significant difference in taste because I may have missed the people who think otherwise.
So proving my hypothesis is dead in the water. But surely my friend can prove their hypothesis? After all, all I needed was a black swan to prove that not all swans are white. The problem is that most research does not have a clear dichotomy of outcomes, and is instead based on a population level measurement. So, what if they take 100 people and on average they say that the Alco pudding tastes better? Does this prove their point? Not really. This may be due to an unrepresentative sample, confounding, other bias, small numbers or simply a chance effect i.e. they just happened by chance to get people who liked one pudding more than the other, but if they took another sample they would not see the same thing. What if they take 1000 people, 10000 people......? Of course, the more people they take, the more likely they are to be seeing the true picture but they can never be sure.
So, fundamentally, any result is always open to doubt unless we have tested absolutely everyone (usually an impossibility). Our statistical testing gives us some level of assurance (usually 95% confidence) but this is not the same as certainty. Even if no other bias has crept in, and our sample is completely representative of the whole population there is always a 5% chance we are wrong – we are only reasonably sure.
I will talk more about the scientific method in future blogs, but at the heart of scientific thinking is the concept of scepticism i.e. everything is open to doubt. Put another way, if when talking about population research you can only ever be reasonably sure, then there is also a possibility that you are wrong. To use the word ‘proof’ or ‘prove’ implies there is no room for being wrong and is therefore unscientific in nature. Instead, we should be talking about the balance of evidence – is there more evidence to support my hypothesis than to reject it?
So why does all this matter? Well, it seems today that scientific thinking is not valued in the way that it used to be. The people who practice scepticism are often seen as being ‘negative’ rather than scientific. The quest for evidence to support an action is seen as not having a ‘can do attitude’, and the questioning of results is seen as obstructive. As human beings we do not like our ideas questioned, or our results looked at logically. We have, to an extent, put aside our respect for science, and the need for sound evidence and moved to the position of ‘if enough people think it then it must be true’.
But it gets worse. We seek to prove negatives all the time
“Prove that we don’t need to re-organise / restructure”
“Prove that you are performing the same as others”
“Prove that there is no difference between these two groups of people”
“Prove that mobile phones don’t cause cancer”
“Prove that vaping isn’t bad for your health”
These challenges seem sensible, and scientific, but contrary to what we feel, we can never prove any of these. The default position is therefore to accept the null statements until other evidence is provided i.e.
“There is insufficient evidence to suggest that we need to re-organise”
“There is insufficient evidence to suggest that I am not performing the same as others”
“There is insufficient evidence to suggest these two groups of people are different”
“There is insufficient evidence to suggest that mobile phones cause cancer”
“There is insufficient evidence that vaping is bad for your health”
This often does not feel right, particularly in situations like our last two examples. As human beings we don’t want to do things until they are shown to be safe, but here we are saying that we can never be sure. As a sceptic we also don't want to accept the status quo. What we can do, is proactively look for evidence that they aren’t safe. Thinking scientifically (and sceptically) changes the way that we look at things. We start looking objectively for evidence to change, rather than evidence to stay where we are. We look for evidence that a new intervention works rather than not. But in all of this we remain sceptical, not to be negative but to retain an open mind, to accept that we might be wrong. We talk about theories rather than facts, and we always continue to test our assumptions. This is scepticism, but it is not negative - it is what makes us scientific.
So based on just my and my friends view of our rice pudding, there is currently insufficient evidence to suggest that one is better than the other. Of course, being human I take this as a win. In essence though, we are expressing that currently we simply do not know what people will think. While evidence may appear at some point to suggest that people think one tastes better than the other, the proof of the pudding is not in the eating - instead it simply needs to be digested scientifically.