Political campaigns spend big bucks hiring consultants to craft persuasive messaging, but a new study coauthored by Yale political scientist Joshua L. Kalla demonstrates that political professionals perform no better than laypeople in predicting which messages will sway voters.

In the study, Kalla and his coauthors evaluated how well sample groups of political practitioners — professionals who work for political campaigns, polling firms, and advocacy organizations — and members of the public could predict the effectiveness of 172 campaign messages concerning 21 political issues, including legalizing marijuana, cancelling student debt, and increasing border security.

They found that both groups performed barely better than chance and that the practitioners were no more perceptive than laypeople in identifying messaging that resonates with people.

“We found that neither political practitioners nor the mass public are particularly accurate in predicting which persuasive messages are more effective than others,” said Kalla, associate professor of political science in Yale’s Faculty of Arts and Sciences. “This suggests that political practitioners who craft language intended to persuade have fairly poor intuitions about which messages people will find persuasive.”

The study, published in the journal Proceedings of the National Academy of Sciences, was coauthored by David E. Brookman, Christian Caballero, and Matthew Easton, all of the University of California, Berkeley.

For the study, the researchers gathered 172 text-based political messages that political practitioners have used to support or oppose 21 distinct issues. They pulled the messages from sources such as voter guides published by various advocacy organizations or the social media accounts of prominent politicians.

An example is a message used by the Marijuana Policy Project to support the legalization of cannabis: “Polls show that a strong and growing majority of Americans agree it is time to end cannabis prohibition. Nationwide, a recent Gallup poll found that 66% support making marijuana use legal for adults.”

To measure the effectiveness of these messages, the researchers conducted a large-scale survey experiment, in which they randomly assigned 23,167 participants into either a treatment group or a control group. The treatment groups were presented with messages for three specific issues; the control group saw no messages. Then they questioned participants in both the treatment and control groups on their opinions of the issues, for a total of 67,215 observations from the participants. The researchers used this data to estimate the efficacy of each message.

Next, they asked 1,524 political practitioners with varied experience and expertise and 21,247 laypeople to predict the messages’ effectiveness. (Ninety-one percent of the practitioners reported being directly involved with developing messaging.) Both groups did little better predicting the messages’ persuasiveness than if they had guessed randomly.

The study showed that the members of the public believed that other people are more persuadable than the initial survey showed or the practitioners expected. But after accounting for those inflated expectations, practitioners did not predict meaningfully better than laypeople, the study found.

Among the political practitioners, the study found that experience or issue expertise did not translate into a greater ability to identify effective messages.

The findings suggest that, rather than relying on their intuition, political practitioners should consider incorporating data-science techniques into their evaluations of potential messages, said Kalla, a faculty fellow at Yale’s Institution for Social and Policy Studies.

“The main takeaway here is that political practitioners have tools available to help them identify effective messages without having to rely on their gut feeling,” he said. “They could use survey experiments similar to what we did in this study. We see political campaigns already doing that, and I suspect more will adopt such techniques moving forward.”