A computer rendered illustration of a digital cube labelled AI locked in a birdcage.

New guidelines aim to safeguard researchers and study participants from AI risks.Credit: J Studios/Getty

A new toolkit to help academics to use generative artificial intelligence (genAI) more ethically is being developed by researchers in the United Kingdom.

“Generative AI is so new, we just don’t have any guidance,” says Wendy Moncur, a cybersecurity researcher at the University of Strathclyde in Glasgow, UK, who is leading the project. Academics are already considering the potential quandaries with use of genAI tools, she says, “but wouldn’t it be a useful thing, if they had a little checklist to say, ‘These are the things you need to think about; these are the strengths; and these are the threats.’”

The project focuses on issues that might arise when genAI tools — such as ChatGPT, made by OpenAI in San Francisco, California, and Google’s Gemini, which are powered by large language models (LLMs) — are used to analyse and process personal information from study volunteers.

It was inspired by an ongoing study, led by Moncur, that is looking into how people going through major life transitions — such as being diagnosed with cancer or undergoing gender reassignment — can manage their privacy online.

In the work, Moncur and her team are using genAI tools to create teaching materials, on the basis of participants’ stories, that are intended to guide others through similar life changes.

The participants had shared details about their experiences — such as how their work and relationships were affected — under the assurance that the information would be shared with others only in an anonymized form. But before the team started feeding this information into a genAI program, Moncur suddenly feared that, if the tool pieced together publically available information with the anonymized data that it was being fed, the participants might accidentally be reidentifiable.

The team was also concerned about LLMs’ tendency to ‘hallucinate’ — generating nonsensical or incorrect information — which could potentially slander reidentified participants. And LLMs can change the meaning of the information fed into them, because they are influenced by social and other biases inherent in their design. For example, Moncur says the program that her team used would distort what the participants had said, making their stories more positive than the participants had intended. “ChatGPT has a bit of a ‘Pollyanna thing’ going on, in that it doesn’t like unhappy endings,” says Moncur. “So, it needs a bit of a nudge to produce a credible story.”

Outlining the issues

Moncur’s concerns prompted her to team up with computer scientists Ali Farooq and Ryan Gibson at the University of Strathclyde and Burkhard Schafer, a legal scholar at the University of Edinburgh, UK, to collaborate on solutions. Funded by the UK National Research Centre on Privacy, Harm Reduction and Adversarial Influence Online, they launched a ten-month project to develop guidelines for researchers and university ethics committees, due to be completed in August.

In March, the European Commission’s European Research Area Forum released guidelines on the responsible use of AI, which will feed into the work that Moncur and her team are doing.

Moncur says the project has three main objectives: to address the lack of expertise in identifying privacy risks caused by using genAI in research; to address data-management requirements in UK research, many of which don’t account for the growing use of genAI; and to address the legal risks for institutions that are using genAI to analyse or process participant data.

The project is designed to look at AI use in research broadly, but will include focus areas, such as how to protect privacy when using AI to process medical data, says Farooq.

The team is doing a literature review to characterize how researchers are using genAI to handle personal data, and is planning to interview academics who serve on ethics committees at UK universities.

Informed by the insights from these projects, the team will develop a toolkit based on analysis of strengths, weaknesses, opportunities and threats, which ethics committees and researchers can consult when they are reviewing or planning projects that will involve genAI technologies. The team plans to make this tool freely available online.

Much-needed guidance

Robert Davison, an information-systems scientist at the City University of Hong Kong, welcomes these efforts to create more-robust ethical oversight for genAI use. “It’s highly likely that it will become normal [to use this technology],” says Davison. But he recalls a point made in an editorial published in January1, which he co-authored: “We do not wish to see a situation where we are lulled into thinking that genAI use is ‘normal’, and that researchers do not need either to pay particular attention to it, or to report their use of it.”

Davison is keen to see ethical norms be established around genAI use, but is wary of a siloed approach to setting these standards. Broader ethical standards would be ideal, he says, but adds that it’s unclear who would be best placed to provide — and enforce — such guidelines.

For now, Moncur and her colleagues will target university ethics committees. “Researchers are under such pressure to be efficient — they’re overloaded,” says Moncur. “If you’ve got a tool [such as AI] that’s going to make things more efficient, then it makes sense to use the tool. But we need information to help us use the tools responsibly, and in a way that allows us to do good science.”