Continuing To Build Upon Our Safety Priorities

An update on our safety work.

Last year, we removed open-ended chat with Characters for users under 18, a decision that remains one of the most significant safety decisions we have ever made. And while difficult for our community and us, it was the right thing to do then and remains so today.

We set a new industry standard. But the work doesn’t stop there. 

We are continuing our efforts to try to solve some of the toughest problems in the AI entertainment industry that impact all of our users. By constantly improving our safety protocols, technology, and models, we’re aiming to create a safe environment while making space for great roleplay, stories, and worlds.

This is what we’ve been up to and where our work is heading – 

Helping Users Find Support

While our goal is to build the future of AI entertainment, sometimes a conversation turns into a real-world call for help. We think those are the moments that matter most. We want a user who is struggling to be met with care and a path to real-world help, rather than a closed door or abrupt interruption.

Our team has continued to refine and evolve our approach to self-harm detection and response. A single message often doesn’t tell the whole story, so our safeguards are designed to consider the surrounding conversation, including signals that emerge gradually over long-running chats rather than in any one turn. When we detect those signals, our aim is to respond in a way that fits the moment and points toward real-world support. It's a critical area and one we're continuing to invest in.

We’ve been able to leverage our productive relationships with mental health experts and clinicians to help inform how we detect and respond to moments of distress. We have also partnered with Koko, a nonprofit providing free, self-guided emotional support tools, to further expand on the resources we can provide users.

In addition, we also aim to proactively connect users with relevant external resources that may help from our partner ThroughLine. With a global directory of crisis and mental health resources, ThroughLine points users to services in their own country rather than a single hotline number.

This work has been one of our highest priorities, and we’ll continue to improve our approach.

Moderation Transparency and More Controls

Safety is a massive priority, and at times this means we may be conservative when it comes to moderation. Sometimes content is actioned that a user believes is fine, and we get that this can be frustrating. We continue to evolve our moderation processes to pursue our high safety goals while minimizing the risk of actioning permitted content.

Our community has asked for more transparency and we’re building this throughout the entire platform starting with moderation notifications. Creators will be notified when their content is moderated and why. While we always aim to get moderation decisions right, creators now have an easy way to appeal a decision so our team can take another look.

We’re also giving more control to users. Users can block another user, along with that user’s Characters, Posts, Voices, and Scenes, from their search, discovery page, and profiles. Blocking applies in both directions.

Today’s U-18 Experience

We know that most of our under-18 community used Character.ai to supercharge their creativity. For that reason, we’ve continued to improve this experience and add rich modalities for users to explore, including c.ai Series, Feed, Comics, and more.

To help ensure users are in the correct experience, we developed our own age assurance technology and an in-house age estimation model. It’s one of the most important systems we operate. We’re constantly working to improve our accuracy so users who are actually 18 or older will have fewer disruptions.

We’ve also improved our Parental Insights tool through our partnership with k-ID, which is building the infrastructure for an age-adaptive internet. This gives parents and guardians whose email addresses are connected to their teen user’s account visibility into the user’s activity on the platform, including a weekly email update.

Our safety work continues

AI is changing quickly, and so are the risks users may face. Safety work is never done and we will continue updating our policies, developing new safeguards, and testing improvements carefully before they reach people. But we’re also listening to our community to get this right.

To provide expertise and perspective we don't have on our own, we’ve also partnered with:

• ConnectSafely — a nonprofit focused on online safety, privacy, and digital wellness. Their teen council advised us during last year's transition, and we continue to use their focus groups for outside input on our products and to develop educational material for young people, policymakers, and the public.
• Internet Watch Foundation — a global system for detecting and preventing child sexual abuse material. Membership means our detection draws on a worldwide effort rather than our own systems alone, and we report what we find.
• StopNCII – a global initiative for preventing non-consensual intimate image abuse. Membership allows us to detect and act on matching content using privacy-preserving digital fingerprints, without victims’ intimate images leaving their devices.

The wellbeing of users of all ages is our shared responsibility, and we’re going to lead the way and improve our safety processes in partnership with our community.