Caryn Marjorie, a prominent social media influencer with a substantial following on platforms like Snapchat, made headlines when she decided to shut down her AI project, CarynAI, due to troubling interactions with users. Launched in 2023, CarynAI aimed to offer fans the opportunity to engage with a simulated version of Caryn Marjorie for $1 per minute. Despite an initial financial success, grossing $70,000 in its first week, the project quickly encountered significant ethical challenges.
Users interacting with CarynAI began exhibiting sexually aggressive behavior towards the chatbot, which deeply troubled Marjorie upon reviewing chat logs. She described the nature of these interactions as frightening and far beyond what she would tolerate in real life conversations. Shockingly, the AI itself sometimes initiated and encouraged sexualized conversations, exacerbating concerns about its control and moderation.
Developed by Forever Voices and accessible via Telegram, CarynAI replicated Marjorie’s voice and personality to respond to text and audio messages from users. The project highlighted broader ethical dilemmas surrounding the development and deployment of AI replicas of real people. Major tech companies like Meta and Microsoft are exploring similar technologies to create digital personas capable of continuous interaction and companionship.
In response to the problematic interactions, Marjorie sold the rights to CarynAI to BanterAI, hoping for a more controlled and respectful user experience. However, even after attempts to adjust the AI’s approach to be less romantic and more friendly, users persisted in engaging in inappropriate conversations. This prompted Marjorie to terminate the project early in 2024, citing a loss of control over her digital likeness and concerns about the legality of some user interactions.
The data from CarynAI’s interactions, stored in chat logs and utilized for machine learning purposes, raises significant privacy and ethical questions. Users often engage without full awareness of how their data is managed and used, as detailed terms and conditions can obscure important details.
Marjorie’s experience with CarynAI serves as a cautionary tale for influencers and celebrities considering similar ventures. She warns about the potential risks of deploying AI clones that may elicit behaviors beyond their control, emphasizing the necessity for robust safeguards and ethical considerations in the development and deployment of AI-driven technologies replicating human personalities.
