How Do We Use AI Ethically?

Summary

Using AI ethically isn't just about whether the answer it produces is accurate. We also need to think carefully about what information we're putting in, whether the system actually needs it, where the underlying data came from, who's missing from it and what we subsequently do with the answer. Convenience isn't a good reason for sharing personal, confidential or sensitive information unnecessarily.

Responsible use also depends on culture. People at every level need to be encouraged, and rewarded, for questioning the data, challenging questionable outputs and asking whether information should be used at all. Ethical use isn't something we achieve by writing another policy. It comes from the behaviour people see, the questions they're encouraged to ask and the decisions organisations make every day.

Why does this article look rather different?

I originally wrote The Only Way Is Ethics about the ethical use of data, but I'm revising it because the current discussion around AI and ethics makes many of the original questions more relevant, not less. Simply republishing it didn't seem enough, though. The questions about data haven't gone away, but we've added a whole new set of questions about what we're now doing with that data, what we're feeding into these systems ourselves, and how readily we're prepared to trust what comes back.

There's another reason I've rewritten it. I recently came across a post on LinkedIn by its Global Head of Content Marketing entitled “Give people a reason to read you. Give AI a reason to cite you.” The advice was quite specific. Incorporate a question into the headline because the title becomes the URL. Lead with a TL;DR (too long, didn't read) so readers get the answer up-front. Use subheadings that answer real questions and make the article easier to skim. Finally, finish with a checklist that gives people something practical to take away.

Anyone who's read much of what I write will probably realise that most of this goes against the grain. It's not the structure I particularly want to read, or write. I tend to write longer articles, wander around a subject and occasionally disappear down a side road because I've spotted something interesting over there. I don't generally begin by telling you everything I'm going to say before I've had the pleasure of saying it. But I may be in the minority in enjoying this style, and I'm curious, so I've rewritten this article broadly along the lines suggested to see whether it makes any difference to how many people find or read it. There's a question in the title, a summary (TL;DR) at the beginning, question-based headings throughout and a checklist at the end. Will it make the slightest difference? I haven't the faintest idea, which is really the point of trying it.

Anyway, onto the substance of the article.

Are we asking the right questions about AI?

Imagine that you're about to build a house. You wouldn't simply throw down some bricks on what looks like solid ground and hope for the best. I presume you'd want to know what was underneath first. You'd probably want somebody who knew what they were doing to check the foundations, rather than spending all their time admiring how nice the kitchen was going to look when it was finished. The same should apply to data, and now it needs to apply to AI as well.

We're surrounded by claims about what AI can do. It can analyse information, produce reports, answer questions, write code, summarise documents, generate images, support decisions and do all sorts of other things. Inevitably, attention gets drawn towards what comes out. Is the answer good? Does it sound convincing? Did it save us time? Could we use it to do something we couldn't easily do before? They're reasonable questions, but they start rather late in the process because by then we've already made one rather important decision. We've decided what we're prepared to put in.

That's something I wrote about recently in Your Attention Please. There's an enormous temptation, when using these tools, to give them everything. We've got a twenty-page document and want a paragraph summarised, so we upload twenty pages. We want help replying to an email, so we paste the whole thread. We want something analysed, so we hand over the spreadsheet rather than asking which variables are actually necessary. It's wonderfully convenient, and that's unfortunately part of the problem. Convenience has a habit of discouraging us from asking whether something was necessary in the first place.

So, before worrying about what comes out, perhaps we ought to spend a bit more time thinking about what went in.

Are we giving AI far more information than it needs?

I suspect we often are, partly because these systems make it so easy. Think about what happens when somebody wants help summarising a report. The obvious thing to do is upload the report. But what else is in it? There may be people's names, email addresses, patient or client information, financial details, commercially sensitive information, internal comments or material belonging to somebody else. None of it may be relevant to the paragraph we actually want summarised, yet we've supplied it anyway because separating the useful part from everything else would've taken another couple of minutes.

This is where an old data-protection idea becomes remarkably useful in this supposedly new world: data minimisation. The ICO says personal information should be adequate, relevant and limited to what's necessary for the purpose, and its AI guidance discusses assessing whether information is actually required and removing information that isn't. That strikes me as a pretty sensible habit regardless of whether we're approaching the question from a regulatory or ethical direction.

Before putting something into an AI system, then, perhaps the first question shouldn't be: "Will this give me a better answer?" It should be: "Does it actually need this information to answer my question?" And if it doesn't, why are we giving it the information? We've spent years telling organisations not to collect personal information merely because they might find a use for it one day. It would be rather odd if we now enthusiastically started handing information to AI systems merely because copying and pasting it was easy.

What about confidentiality?

The conversational nature of these tools doesn't help. A box on a screen feels informal. You type something, something answers and the whole exchange can feel remarkably private. That informality makes it easy to move from asking for help improving a paragraph to copying an email chain, report, case history, contract or spreadsheet into the conversation without pausing to consider everything contained within it.

Confidential information doesn't stop being confidential because copying and pasting it is convenient, though, and personal information doesn't stop being personal because we've asked a computer to summarise it. Where AI uses personal data in training, testing or deployment, the ICO says data-protection legislation can apply. Its guidance also addresses appropriate security where personal data is being processed using AI.

That doesn't mean we can't use AI with information simply because it's personal or confidential. The circumstances matter, the system matters and what we're doing with the information matters. What it does mean is that we can't sensibly behave as though putting data into an AI system isn't really doing anything with it. We need to understand what we're using, what we're giving it, why we're giving it that information and whether we're entitled to do so. Sometimes the ethical problem with an answer begins before the system has generated a single word.

Is legal the same as ethical?

This was one of the main arguments in the original article and I don't think AI changes the answer. Following the law and behaving ethically clearly overlap, but they're not interchangeable. Establishing that we have a lawful basis for processing particular information answers an important legal question, but it doesn't automatically answer whether the way we're using it will seem reasonable to the people involved.

We've all encountered websites where agreeing to something is considerably easier than understanding exactly what we're agreeing to. Somewhere behind the brightly coloured button there may be a perfectly adequate explanation, but whether most people would genuinely expect everything that's subsequently done with their information is another question altogether. AI just makes the gap between collection and subsequent use even wider because information can be put to purposes that wouldn't necessarily have occurred to the person who originally provided it.

I still think one of the simplest questions from the original article remains one of the most useful: "If people could see the full picture, would they feel respected, or would they feel like they'd been used?" It's not a particularly sophisticated test and it certainly isn't a replacement for the law, but as an ethical question it has the useful advantage of forcing us to look at what we're doing from somebody else's side of the screen.

Can AI be ethical if the data underneath it isn't?

I don't really see how it can be. We've got the information we actively provide when using a system, but underneath that sits the much larger question of the data upon which systems themselves depend or are trained on. The ICO has identified lawful basis, purpose limitation, accuracy and people's data rights among the data-protection questions raised by generative AI. These aren't peripheral technical concerns. They affect what the technology is built upon and, consequently, what we're prepared to do with it.

If we don't know where data came from, how it was collected, who is represented, who isn't represented, what assumptions are buried in it or whether it's appropriate for the job at hand, we're back to our house with the unexamined foundations. We can spend a fortune on everything above ground, but none of that alters what's underneath.

It's a bit like reality television. At first glance everything seems reasonably straightforward. There are goodies, baddies, arguments, friendships and invariably somebody who insists they've only gone there "for the experience". Eventually we remember that what we're watching has been selected and edited. The events may have happened, but that doesn't make what we're seeing the whole story. Data is much the same. Somebody decided what to collect, somebody defined the categories, something wasn't recorded, somebody didn't appear, circumstances changed and errors crept in. Putting more sophisticated technology on top of those data doesn't make any of that disappear. If anything, there's a risk that the sophistication makes us less inclined to look.

Who's missing from the data?

For all our enthusiasm for analysing what we've got, we spend surprisingly little time thinking about what isn't there. Suppose we have information about people receiving a particular healthcare service. It's very easy to start referring to the data as though it describes people who need that service, but it doesn't necessarily. It describes the people who got as far as receiving the service and being recorded. Someone who couldn't access it, wasn't referred, was incorrectly diagnosed or disappeared somewhere else in the system may be completely absent.

The same problem appears in recruitment. If we use previous recruitment decisions to help decide who looks like a good candidate in future, what have we actually taught the system? Have we taught it the characteristics associated with people who proved good at their jobs, or have we inadvertently taught it what previous recruiters tended to choose? Those aren't necessarily the same thing.

Historical data tells us something about what happened under the circumstances that existed at the time. It doesn't automatically tell us what should happen next, and that difference becomes more important when we allow systems to learn from previous decisions and then use what they've learnt to influence future ones.

Does more data mean better AI?

Possibly, but "more" and "better" aren't synonyms. We have a rather naïve faith in large numbers. Ten observations look distinctly unconvincing while ten million look enormously impressive, but ten million systematically unrepresentative observations don't necessarily become representative through force of numbers. We may simply become much more confident in the wrong result.

This is another reason the temptation to pour ever more information into these systems should make us stop and think. More data can certainly be useful, but we should be able to explain why the additional information is needed. The ICO's guidance on data minimisation talks about assessing whether information is adequate, relevant and limited to what's required for the purpose. "Because we've got it" isn't a terribly convincing justification for using data, and neither is "because the computer might find something interesting".

Can we reuse information for something completely different?

Sometimes we can, but we need to think carefully about what might have changed. I've always been wary of the wonderfully efficient sounding principle of "collect once, use many times". On the face of it, it makes perfect sense. Collecting information costs time and money, so why repeatedly collect the same thing when we've already got it? The problem is that sometimes it isn't the same thing, because although the data hasn't changed, the question we're asking of it has.

Covid provided an unusually obvious example. Information collected during the pandemic came from extraordinary circumstances. People's behaviour changed, healthcare changed, access to services changed and treatment changed. If we're examining what happened during that period, those data are obviously relevant. If we subsequently use exactly the same data to describe what we'd expect under very different circumstances, we need to consider whether they still mean what we think they mean.

AI makes this question considerably more interesting because existing information may suddenly acquire potential uses that weren't imagined when it was collected. The fact that data is sitting in a spreadsheet on our computer doesn't automatically tell us whether we should upload it somewhere else, just as the fact that somebody originally gave us information doesn't necessarily mean every conceivable subsequent use of it is appropriate.

What happens when AI gets something wrong?

Humans get things wrong all the time. I certainly don't have an unblemished record in the mistake department and I've yet to meet anybody who has. But the ethical problem isn't simply that AI can make mistakes. It's also that we easily forget that it can, particularly because the answer can arrive quickly, fluently and with none of the hesitation we'd normally associate with somebody who doesn't really know.

Confidence isn't evidence of correctness. If I apply for a loan and somebody rejects me, I can reasonably ask why. If the answer is that the system produced a particular score, I'd quite like to know what that score means and why it led to that decision. If nobody can explain it, challenge it or work out whether something went wrong, having a sophisticated system hasn't removed responsibility. It's just made responsibility more difficult to locate.

And that's where the ethical issue circles back to what we put in. The system may have been given inappropriate data, or information that simply wasn't necessary for the task. Important context might have disappeared along the way, while the way we framed the original question could itself have pushed the answer in the wrong direction. There are plenty of places for problems to creep in before we ever see the result, which is why we can't sensibly judge what comes out without looking just as carefully at what went in, where that information came from and whether it should have been there in the first place.

Who is responsible when AI helps make a decision?

We are. However tempting it might be to find somebody else to blame the responsibility lies with us. AI gives us a wonderfully convenient vocabulary for distancing ourselves from decisions. The model identified something. The system recommended something. The computer generated the answer. After a while you begin to wonder whether any human being was involved at all.

Except, of course, somebody decided to use the system. Somebody chose what to give it, somebody decided what to ask and somebody subsequently decided whether the answer was sufficiently convincing to use. Even where much of a process is automated, people made the decisions that allowed it to become automated.

We shouldn't allow technology to become a convenient place to park accountability, and that responsibility starts before we type anything. If I'm about to upload somebody else's personal or confidential information, "Will this give me a good answer?" isn't really the first question. I should probably begin with "Should I be giving it this information at all?"

Do we still need expertise if AI can give us the answer?

We've been here before. Statistical software made increasingly sophisticated techniques available to people who didn't have to understand the mathematics required to calculate them by hand. Generally, that was a very good thing. What the software couldn't do was decide whether the technique was appropriate for the question, whether the underlying assumptions made sense or whether the result was complete nonsense.

I've said for years that being able to press the button isn't the same thing as knowing which button to press. AI takes that problem a stage further because we don't necessarily even have to find the button anymore. We can simply ask for what we want and receive something that looks remarkably like an answer.

That makes expertise more important, not less. If obtaining an answer becomes easy, the difficult part shifts towards deciding whether we ought to believe it. We still need people who recognise when something doesn't make sense, check original sources, understand context and question assumptions. And we need people who recognise that the first mistake might have occurred before the answer appeared because the information used to obtain it shouldn't have been supplied in the first place.

What does ethical use of AI actually look like?

I don't think it begins with a sixty-page policy which everybody solemnly agrees is terribly important and virtually nobody reads again. Policies and procedures have their place, but ethical use happens much closer to the work itself. It's the person who stops before uploading a document and asks whether the whole document is actually needed, removes names or other information that serves no purpose, or extracts the relevant section rather than supplying twenty pages simply because it's easier. Sometimes the right decision will be not to put the information in at all, particularly where it's confidential or where there's doubt about whether it should be used in that way. None of those decisions is especially dramatic, and most will never appear on a governance report, but they're where responsible use actually happens.

The same willingness to question needs to continue when the answer comes back. Where did this claim come from? Does it make sense? Can I check it? What's missing? What happens if it's wrong? If the result is going to influence a decision about somebody else, could I sit down with that person and explain honestly what we've done and why without becoming distinctly uncomfortable halfway through? Those questions shouldn't belong solely to analysts, information-governance teams or whoever has been given AI responsibility. People at every level of an organisation should be asking them, from somebody using a tool for the first time through to the chief executive deciding how these technologies will be used across the organisation.

That requires a cultural shift, and culture doesn't change because somebody writes a new process and puts it on the intranet. People work out what an organisation really values by watching what happens around them. They notice what senior people do, what managers tolerate, what colleagues challenge and, importantly, what gets recognised and rewarded. If a senior manager talks enthusiastically about responsible use but then drops a confidential document into the nearest available tool because they need a summary before the next meeting, the policy isn't going to be the lesson everybody remembers. The observed behaviour will be. Equally, if somebody questions whether information should be uploaded and is treated as awkward, risk-averse or an obstacle to getting the job done, others will learn fairly quickly to keep quiet next time.

That's why people should be encouraged and rewarded for asking awkward questions, not merely given permission to ask them. Someone who spots that we're supplying information we don't need, challenges the provenance of the data, questions an implausible answer or stops a process because they're uncomfortable with how somebody's information is being used isn't getting in the way. They're doing exactly what we'd hope somebody would do in an organisation that takes ethics seriously. Leaders have a particular responsibility here because people will take considerably more notice of how they react when challenged than they will of anything written in an AI policy. A manager who responds to a difficult question with genuine interest sends one message. One who sighs, becomes defensive or finds a way around the person asking it sends another.

Culture, then, isn't created by repeatedly telling people to use AI responsibly. It's created when responsible behaviour becomes the normal behaviour people see around them. It's when the analyst is expected to question the data, the manager is comfortable admitting they don't understand how a conclusion was reached, the person about to upload a document stops to check what's in it, and the most senior person in the room can be challenged without everybody suddenly developing an intense interest in their shoes. That's a much harder change than producing another policy, but it's also far more likely to make a difference. Sometimes the most useful person in the room is still the irritating one who asks whether we're sure we should be doing this in the first place..

So, can we use AI ethically?

Of course we can, but I think ethical use begins considerably earlier than checking whether the final answer looks right. We need to start by asking what information the system genuinely needs. Whose information is it? Why have we got it? Are we entitled to use it for this purpose? Does the system really need to know who somebody is? Could we remove identifying or irrelevant information? Are we handing over a whole document because it's necessary, or because extracting the relevant part would take us another five minutes?

Then we need to look underneath the technology itself. Where did its data come from? Who's represented and who isn't? What decisions and assumptions are buried in it? Are historical patterns something we genuinely want to reproduce? Only after we've dealt with those questions do we arrive at the things that tend to receive most of the attention: whether an answer is accurate, whether we can explain it and whether we ought to act on it.

Before using data or AI, what should we ask?

Before putting anything in:

• Does it actually need this information?

• Am I providing more information than is necessary?

• Does it contain personal, confidential, commercially sensitive or otherwise restricted information?

• Could I remove names, identifiers or irrelevant material first?

• Do I understand where I'm putting the information and what I'm doing with it?

• Would I be comfortable telling the person whose information it is exactly what I've done?

Think about the data underneath:

• Where did it come from?

• Why was it collected?

• Who's represented?

• Who isn't?

• Has the purpose changed?

• Could historic decisions or inequalities be reflected in it?

• Are we using information simply because we happen to have it?

Then look at what comes out:

• Does the answer actually make sense?

• Can important claims be checked?

• Do I know enough to recognise when something is wrong?

• Can an important decision be explained and challenged?

• Who's responsible for acting on the result?

• What happens if we're wrong, and who bears the consequences?

And I think one question sits above all the others: just because we can do it, should we?

Most of my career has been spent trying to persuade people to make better use of data, so I'm certainly not about to argue that we should stop using useful technology because there are risks involved. But useful and responsible aren't interchangeable. The ethical use of AI isn't simply about preventing bad answers from coming out. It's about taking responsibility for the whole process, including the information we're often rather casually putting in, the data already sitting underneath, the conclusions we draw and the decisions we subsequently make.

Before admiring the house then, we should check the foundations. We might also want to make sure we haven't accidentally left something lying around inside that never needed to be there in the first place.