AI-Based Detection of Discourse Derailment: A New Approach to Spot Disinformation in Online Conversations

Short read summary

The Conversation highlights recent research that shifts focus from identifying whether a post is AI-written to whether it derails the conversation. By generating plausible responses to genuine comments with a large language model and comparing them to actual replies, the system detects derailment patterns that often accompany disinformation campaigns. When the actual reply differs substantially from AI-generated expectations, it may indicate an attempt to steer discussions away from the original topic. The study finds that 36% of derailing messages included red herrings, 65% showed non sequiturs, and 20% contained personal attacks, with an accuracy around 77% in a second study. This approach can act as an early-warning tool for moderators, subject to ethical considerations and human judgment.

Medium read summary

The article discusses a research approach to countering online disinformation by detecting discourse derailment rather than simply flagging AI-generated content. The researchers analyzed comments on BBC News videos on YouTube, a platform previously targeted by disinformation campaigns. They observed that malefactors often derail conversations by steering them toward polarizing topics rather than addressing the original post. A red herring is used to distract from the topic, and other patterns such as leaps in logic (non sequiturs) and personal attacks are common. Importantly, the researchers argue that word-level detection is increasingly unreliable due to advances in AI, which can produce fluent and natural text. Instead, they built a system to recognize the flow of conversation and to detect when replies diverge from expected responses.

The detection method is described as “detecting discourse derailment” by measuring the distance between real replies and a set of AI-generated, plausible responses. The authors first created a manually labeled dataset with 25 features of online discussion, then used an AI to generate several reasonable, relevant replies to each genuine comment. They compared these AI-generated replies to the actual replies. If the real reply deviated significantly from the AI-provided responses, the system flagged potential derailment. This approach shifts the focus from language patterns to the function of a message within a conversation.

Results from their second study suggest the method correctly identifies derailing comments about 77% of the time, roughly doubling the performance of word-level sentiment analyses. The approach also aligns with human agreement levels in detecting derailment, underscoring its potential as a tool to assist moderators rather than replace human judgment. The article also stresses ethical concerns, including potential biases in training data and the necessity of maintaining human oversight in moderation decisions.

In sum, the research demonstrates that understanding how conversations unfold and when a reply breaks from expected patterns can provide a robust signal for disinformation, complementing traditional fact-checking and sentiment measures.