ChatGPT users who show indications of violent extremism may be guided to specialist deradicalisation assistance, delivered by both people and chatbots, via a new system being built in New Zealand, according to Reuters. The effort aims to address increasing safety concerns linked to AI use on the internet.
The development comes as AI companies grapple with increasing legal pressure over claims they have been unable to stop, and may even have contributed to, violent incidents.
In February, OpenAI was warned of possible action by the Canadian government after revealing it had banned a school shooter from its platform without informing authorities.
ALSO READ: Oracle Layoffs: Tech Giant To Cut Thousands Of Jobs Amid Rising AI Spending? Here's What We Know
According to the report, ThroughLine, a startup that has worked with OpenAI, Anthropic and Google to direct at-risk users to crisis resources, is behind the new tool. Its founder, Elliot Taylor, said the company is exploring options to extend its work into tackling violent extremism.
ThroughLine is reportedly holding discussions with The Christchurch Call, the anti-extremism effort launched after the 2019 New Zealand terror attacks. The organisation is set to advise the company as it develops its intervention chatbot.
"It's something that we'd like to move toward and to do a better job of covering and then to be able to better support platforms," Taylor told Reuters, adding that currently there is no fixed timeframe.
OpenAI said it works with ThroughLine but declined to comment further; Anthropic and Google did not respond to Reuters' enquiries at the time of publication. The startup, run by Elliot Taylor from his rural New Zealand base, has become a trusted resource for AI firms thanks to its actively maintained network of over 1,600 helplines in 180 countries. When a potential mental health issue is detected, users are redirected to ThroughLine and connected with a relevant local service.
ALSO READ: AI Could Reshape Corporate Structure, Remove Middle Management, Says Jack Dorsey
ThroughLine's current focus has been relatively narrow, according to its founder. But as AI chatbots gain traction, the types of issues raised by users have expanded sharply, now encompassing brushes with extremism. Taylor said any new intervention would likely combine a purpose-built chatbot with referrals to human-led mental health support.
Essential Business Intelligence, Continuous LIVE TV, Sharp Market Insights, Practical Personal Finance Advice and Latest Stories — On NDTV Profit.
