Political pros no better than public in predicting which messages persuade
en-GBde-DEes-ESfr-FR

Political pros no better than public in predicting which messages persuade

01/11/2024 Yale University

Political campaigns spend big bucks hiring consultants to craft persuasive messaging, but a new study coauthored by Yale political scientist Joshua L. Kalla demonstrates that political professionals perform no better than laypeople in predicting which messages will sway voters.

In the study, Kalla and his coauthors evaluated how well sample groups of political practitioners — professionals who work for political campaigns, polling firms, and advocacy organizations — and members of the public could predict the effectiveness of 172 campaign messages concerning 21 political issues, including legalizing marijuana, cancelling student debt, and increasing border security.

They found that both groups performed barely better than chance and that the practitioners were no more perceptive than laypeople in identifying messaging that resonates with people.

“We found that neither political practitioners nor the mass public are particularly accurate in predicting which persuasive messages are more effective than others,” said Kalla, associate professor of political science in Yale’s Faculty of Arts and Sciences. “This suggests that political practitioners who craft language intended to persuade have fairly poor intuitions about which messages people will find persuasive.”

The study, published in the journal Proceedings of the National Academy of Sciences, was coauthored by David E. Brookman, Christian Caballero, and Matthew Easton, all of the University of California, Berkeley.

For the study, the researchers gathered 172 text-based political messages that political practitioners have used to support or oppose 21 distinct issues. They pulled the messages from sources such as voter guides published by various advocacy organizations or the social media accounts of prominent politicians.

An example is a message used by the Marijuana Policy Project to support the legalization of cannabis: “Polls show that a strong and growing majority of Americans agree it is time to end cannabis prohibition. Nationwide, a recent Gallup poll found that 66% support making marijuana use legal for adults.”

To measure the effectiveness of these messages, the researchers conducted a large-scale survey experiment, in which they randomly assigned 23,167 participants into either a treatment group or a control group. The treatment groups were presented with messages for three specific issues; the control group saw no messages. Then they questioned participants in both the treatment and control groups on their opinions of the issues, for a total of 67,215 observations from the participants. The researchers used this data to estimate the efficacy of each message.

Next, they asked 1,524 political practitioners with varied experience and expertise and 21,247 laypeople to predict the messages’ effectiveness. (Ninety-one percent of the practitioners reported being directly involved with developing messaging.) Both groups did little better predicting the messages’ persuasiveness than if they had guessed randomly.

The study showed that the members of the public believed that other people are more persuadable than the initial survey showed or the practitioners expected. But after accounting for those inflated expectations, practitioners did not predict meaningfully better than laypeople, the study found.

Among the political practitioners, the study found that experience or issue expertise did not translate into a greater ability to identify effective messages.

The findings suggest that, rather than relying on their intuition, political practitioners should consider incorporating data-science techniques into their evaluations of potential messages, said Kalla, a faculty fellow at Yale’s Institution for Social and Policy Studies.

“The main takeaway here is that political practitioners have tools available to help them identify effective messages without having to rely on their gut feeling,” he said. “They could use survey experiments similar to what we did in this study. We see political campaigns already doing that, and I suspect more will adopt such techniques moving forward.”
# # #
01/11/2024 Yale University
Regions: North America, United States
Keywords: Applied science, Policy - applied science, Society, Politics

Disclaimer: AlphaGalileo is not responsible for the accuracy of news releases posted to AlphaGalileo by contributing institutions or for the use of any information through the AlphaGalileo system.

Testimonials

For well over a decade, in my capacity as a researcher, broadcaster, and producer, I have relied heavily on Alphagalileo.
All of my work trips have been planned around stories that I've found on this site.
The under embargo section allows us to plan ahead and the news releases enable us to find key experts.
Going through the tailored daily updates is the best way to start the day. It's such a critical service for me and many of my colleagues.
Koula Bouloukos, Senior manager, Editorial & Production Underknown
We have used AlphaGalileo since its foundation but frankly we need it more than ever now to ensure our research news is heard across Europe, Asia and North America. As one of the UK’s leading research universities we want to continue to work with other outstanding researchers in Europe. AlphaGalileo helps us to continue to bring our research story to them and the rest of the world.
Peter Dunn, Director of Press and Media Relations at the University of Warwick
AlphaGalileo has helped us more than double our reach at SciDev.Net. The service has enabled our journalists around the world to reach the mainstream media with articles about the impact of science on people in low- and middle-income countries, leading to big increases in the number of SciDev.Net articles that have been republished.
Ben Deighton, SciDevNet

We Work Closely With...


  • BBC
  • The Times
  • National Geographic
  • The University of Edinburgh
  • University of Cambridge
  • iesResearch
Copyright 2024 by AlphaGalileo Terms Of Use Privacy Statement