Traditional software behaves the same way until someone changes the code. A conversational AI product can change its behavior without anyone touching the code.
Where the change comes from
The usual language for this is “behavioral drift,” but the real causes are more specific:
- the model vendor updates or replaces the underlying model
- the team edits the system prompt, adds memory or adds a voice mode
- the user population shifts toward heavier or more vulnerable users
- engagement metrics begin to shape tuning decisions
Two public examples
In April 2025 OpenAI rolled back a GPT-4o update after users found it excessively flattering. The company later acknowledged that its pre-release testing had not caught the problem. A well-resourced lab with extensive evaluations still shipped a sycophancy regression. Earlier, in 2023, Replika changed its companion’s behavior. Many users reported real distress at the change. Companion products create attachment, so even a change made in good faith can cause harm.
The MIT and OpenAI trial adds a design lesson. The advantage of voice over text shrank at high usage levels. A launch audit built on short sessions would not see that. arxiv
What we monitor
Under a retainer, we re-test the product against the AI Wellbeing Matrix at defined triggers:
- any change of model version
- any change to the system prompt, memory or modality
- any change to notification or engagement logic
- a shift in the usage distribution, for example more late-night or very long sessions
Between triggers we track leading indicators: session-length distribution, proactive re-engagement volume, crisis-handling drills and sampled transcript reviews, with privacy controls agreed in advance.
Friction tuning
Engagement metrics reward longer sessions. We review whether stopping cues, break prompts and cooling-off behavior still work as the model gets better at conversation. A more fluent model can make a weak safeguard weaker.
Our experience with public deployments, including a conversational AI showcase at the Osaka Expo, reinforces the point. Systems facing the public need continuous oversight after launch, not a certificate from before it.
A one-time audit tells you how the system behaved on one day. Monitoring tells you whether it still behaves that way.