Responsible Agentic AI and Digital Safety for Countering Online Extremism: Programme Overview

This project develops a conceptual framework for understanding and governing the capabilities of emerging agentic AI systems in order to prevent malicious use, particularly in security-sensitive and PCVE contexts. Building on insights from sociotechnical risk analysis, platform governance, and early work presented at an ECSCW workshop, the project examines how autonomous agents acquire, combine, and operationalize persuasive, generative, or decision-making capabilities that may produce harmful downstream effects.

Through collaboration with researchers from Swansea University’s CYTREC centre and support from the (Terrorism and Social Media) TASM Sandpit initiative, the project explores how capability constraints, escalation safeguards, and institutional oversight can be designed to reduce misuse risks without undermining legitimate innovation. It aims to contribute a policy-oriented model of capability governance that helps organisations anticipate hazardous combinations of AI functions, assess intervention thresholds, and design preventative controls for responsible deployment.

Collaborator: Daniel E. Levenson, MA, MLA (Swansea University)

Funding support: TASM Sandpit at Swansea University

This FFG-supported feasibility study examines the technical, ethical, and operational requirements for deploying agentic AI systems in real-world PCVE and online-safety settings. The project evaluates whether conversational or counter-persuasive agents can be safely integrated into existing prevention workflows, with particular attention to usability, trust, risk mitigation, and alignment with organisational practices. Through stakeholder consultations, prototype exploration, and scenario-based assessments, the study identifies key opportunities and constraints for developing scalable, autonomy-preserving AI interventions that support violence-prevention organisations and policy stakeholders.

Funding: Austrian Research Promotion Agency (FFG)

Collaborator(s): Austrian and German Violence Prevention NGO's

Student(s): Md. Omar Khaium

This project investigates how autonomous conversational agents can counter early and mid-stage online radicalization by engaging users in structured, trust-preserving counterdialogue. Building on emerging research in counterargumentation, belief revision, (therapeutic)inoculation, and counter-persuasion, the project explores how chatbots can introduce doubt, weaken harmful narratives, and support healthier belief trajectories without compromising user autonomy.

The work combines stakeholder interviews, conceptual and value-sensitive analysis, and controlled experiments on platforms such as Discord to understand the sociotechnical dynamics of persuasive harm and safe intervention. It develops early computational models for recognising harmful influence processes, generating context-appropriate counterarguments, and managing risks such as reactance or escalation. This project contributes foundational insights for designing agentic AI systems capable of participating responsibly in Prevention and Countering Violent Extremism (PCVE) efforts and improving digital safety in online communities.

Funding: Austrian Science Fund (FWF).

Project link: FWF Research Radar

Student(s): Md. Omar Khaium