🚨 New paper in Science Advances
@science.org
Can changing how we argue about politics online improve the quality of replies we get?
T HeideJorgensen,
@gregoryeady.bsky.social & I use an LLM to manipulate counter-arguments to see how people respond to different approaches to arguments
Thread 🧵1/n
We focus on four core features of political debate:
(1) whether arguments are backed by evidence (or emotion)
(2) whether they signal openness to compromise
(3) whether they come from a partisan (in-/out-/non-partisan)
(4) whether they are respectful or disrespectful
2/n
Survey respondents write their opinion on a political issue they care about.
An LLM then generates a counter-argument, randomizing the four features, 2×2×3×2 = 24 possibilities:
evidence/emotion x open/closed to compromise x in-/out-/non-partisan x respectful/disrespectful
3/n
Jul 28, 2025 08:37Each part of the counter-argument is generated independently via separate LLM calls—similar to a vignette experiment with multiple treatments.
This ensures that if a message is, say, “disrespectful,” its substantive content is otherwise held constant.
4/n
Respondents then reply to the counter-argument.
Replies are coded for whether:
(1) justify an argument
(2) signal openness to debate
(3) signal partisanship
(4) disrespectful
+ close-ended measures of openness to debate, pol. attitudes & perceptions of (LLM) interlocutor.
5/n
First, put simply, how people argue matters a lot.
Evidence-based, respectful, compromise-seeking, and non-partisan counter-arguments substantially increase the probability of High Quality replies...
(HQ = justification/compromise+respectful+no partisan attacks)
6/n
Combining all beneficial features (evidence, openness, non-partisan, respect) doubles the probability of a high quality response. And these features also reinforce each other thru positive spillovers...
7/n
For example, signaling openness to compromise not only makes the reply more conciliatory, it also increases the probability that people will seek to justify their arguments.
Respect (unsurprisingly) increases openness to compromise.
Effects, overall, are complementary.
8/n
Second, another basic finding. For each feature of an argument:
(1) Respect begets respect
(2) Evidence elicits evidence
(3) Openness invites openness
(4) Disrespect provokes partisan attacks
When arguing about politics, you get what you give.
9/n
Crucially, high-quality argumentation improves perceptions of the interlocutor: they are seen as more open-minded, informed, respectful, and reasonable—traits that in turn predict higher-quality replies.
10/n
Further, features *unrelated to argument substance* increase arguments being seen as strong.
Simply signaling openness to compromise increases perceptions your arguments is strong, you are well-informed, and you are ideologically moderate.
11/n
However, improved debate ≠ persuasion.
None of the treatments significantly affect issue attitudes, polarization, or certainty.
Approaches to arguments shape how people reply, but not necessarily what they believe. (caveat: on issues they care about most)
12/n
Bottom line:
Even if we can't always change minds with more constructive arguments, we can change the quality of political discussion that we ourselves have with others—if we want it.
Further results in the full paper:
🔗
www.science.org/doi/10.1126/...
13/n
Understanding the success and failure of online political debate: Experimental evidence using large language models
Adjusting tone, justification, and compromise improves online debate quality but does not change political attitudes.
An early version of this paper led to a grant proposal about political discourse, which will be led by
@gregoryeady.bsky.social & graciously funded by the Danish Research Council, beginning this fall
14/n
https://gregoryeady.bsky.social
https://kingsqpe.bsky.social
