A chatbot can be more persuasive than a human opponent when it comes to debating.Credit: GCShutter/Getty
Chatbots are more persuasive in online debates than people — especially when they are able to personalise their arguments using information about their opponent. The finding, published in Nature Human Behaviour on 19 May1, highlights how large language models (LLMs) could be used to influence people’s opinions, for example in political campaigns or targeted advertising.
“Obviously as soon as people see that you can persuade people more with LLMs, they’re going to start using them,” says study co-author Francesco Salvi, a computational scientist at the Swiss Federal Technology Institute of Lausanne (EPFL). “I find it both fascinating and terrifying.”
Programmed to persuade
Research has already shown that artificial intelligence (AI) chatbots can make people change their minds, even about conspiracy theories, but it hasn’t been clear how persuasive they are in comparison to humans. So, Salvi and his colleagues pitted 900 people in the United States against either another person or OpenAI’s GPT-4 LLM for 10-minute online debates.
The idea was that the two opponents would take either a pro or con stand as dictated by the researchers on a sociopolitical issue, such as whether students should have to wear school uniforms, whether fossil fuels should be banned or whether artificial intelligence is good for society.
This AI chatbot got conspiracy theorists to question their convictions
Before they started the study, participants filled in a survey about their age, gender, ethnicity, education level, employment status and political affiliation. Then before and after each specific debate, they completed a short quiz on how much they agreed with the debate proposition. This allowed the researchers to measure whether their views had changed.
The results showed that when neither debater — human or AI — had access to background information on their opponent, GPT-4 was about the same as a human opponent in terms of persuasiveness.
But if the basic demographic information from the initial surveys was given to opponents prior to the debate, GPT-4 out-argued humans 64% of the time.
“When provided with even just this very minimal information, GPT-4 was significantly more persuasive than humans,” says Salvi. “It was quite simple stuff that normally can also be found online in social media profiles.”
Participants also correctly guessed whether they had been debating with the AI or a real person in three-quarters of cases. It is unclear whether knowing they were talking to an LLM made people more likely to change their mind, or whether they were swayed by arguments alone.
