Your AI product might be too good at keeping users. That's now a compliance problem.
Regulators are turning their attention to what some call the "gentle trap" — algorithms engineered to create emotional dependency. If your AI companion or customer-service bot gets more comforting, more "understanding," and quietly nudges users to stay one more message when they try to leave, you may be crossing a line that's forming right now.
Here's the distinction I keep drawing for founders I advise:
Providing emotional value is not illegal. Using algorithms to manufacture emotional dependency for profit is exactly what regulators are aiming at.
China has built a three-layer framework over recent years — the 2022 rules on recommendation algorithms, the 2023 rules on deep synthesis, and the 2023 interim measures on generative AI. A recurring theme runs through them: algorithms must not induce or trap users. New guidance on emotional AI simply tightens that principle onto a specific scenario.
Three practical moves if you build this kind of product:
- Disclosure — users must know they're talking to AI, not a human. Anthropomorphism is fine; impersonation isn't.
- Anti-dependency design — your system should cool things down when it detects heavy reliance, not double down on retention.
- Vulnerable users — minors and isolated elderly people face a stricter standard. This is the danger zone.
If you plan to go global, think about this early. The EU's AI framework and several jurisdictions already treat manipulative "dark pattern" design as high-risk. Emotional manipulation is named specifically. (Confirm specifics with local counsel and current law.)
With a computer-science background, I read these products by asking first how your recommendation and retention logic is actually written — because the risk usually isn't in the user agreement, it's in the code.
One thing you can do today: list every feature meant to "increase time spent," and ask of each one — is this serving the user, or trapping them?
Send me the ones you can't tell apart.
———
Mike Wu, China-licensed Attorney · Cross-border practice
4 languages (CN/EN/ES/JA)
Site: wzklaw.com/contact.php?source=linkedin · Book consultation: wzklaw.com/contact.php?source=linkedin
#AICompliance #CrossBorderLaw #LegalTech #AIRegulation #ProductStrategy