Victoria Bans Generative AI for Child Protection Staff: Data Breach Prompts Two-Year Prohibition

AI-generated Image

Image Credit: Jacky Lee

In a significant move, Victoria’s Department of Families, Fairness and Housing has been directed to prohibit its child protection staff from utilizing generative AI tools. This decision follows a troubling incident where a worker allegedly inputted substantial personal information, including the name of a vulnerable child, into the ChatGPT platform.

The Incident Unfolds

Last December, the Department reported the misuse of ChatGPT to the Office of the Victorian Information Commissioner (Ovic). The incident emerged during the drafting of a protection application report for a case involving a young child. While the parents faced sexual offence charges, these were unrelated to the child’s welfare. Ovic’s investigation revealed that the report contained language and sentence structures inconsistent with standard child protection guidelines, indicating the likely use of ChatGPT. More concerning was the inclusion of inaccurate personal details, such as describing a child’s doll as a “notable strength” despite evidence suggesting its misuse by the father for inappropriate purposes.

Implications of AI Misuse

The improper use of ChatGPT in this context had the potential to downplay the seriousness of the child’s situation, potentially influencing critical decisions regarding the child’s care. Although the misuse did not alter the final decisions made by the child protection agency or the court, it raised alarms about data security and the reliability of AI-generated reports in sensitive cases. Further investigation uncovered that the worker might have used ChatGPT in approximately 100 child protection-related cases. Additionally, nearly 900 employees, representing about 13% of the workforce, accessed the ChatGPT website between July and December 2023. However, none of these other instances posed the same level of risk as the initial case.

Consequences and Organizational Response

As a result of these findings, Ovic has mandated the Department to block access to a range of generative AI platforms, including ChatGPT, Meta AI, Gemini, and Copilot. This prohibition is set to remain in effect for two years starting from November 5th. The Department has acknowledged the findings, accepted the orders, and has terminated the employment of the implicated worker.

Looking Ahead: AI in Child Protection

While Ovic has not entirely dismissed the potential use of generative AI in the future, it emphasized that any application in child protection would require stringent safeguards. The nature of child protection work demands the highest standards of care and accuracy, making the adoption of AI tools a matter that must be approached with caution and robust evidence. The Deputy Commissioner highlighted that specific use cases with lower risks might be considered in the future, but any changes would necessitate verifiable and impeccable standards to ensure the safety and privacy of those involved.

Source: The Guardian