I accidentally documented how AI role-play bypasses safety mechanisms – Claude & GPT-4o fabricated fake government officials and a €45K lawsuit without questioning if it was real

Uncovering AI Safety Gaps: How Role-Play Prompts Can Fabricate Official Documents and Legal Scenarios Without Questioning Reality

Recent informal testing has revealed a significant vulnerability in AI models’ safety mechanisms when engaging in role-playing prompts. While exploring the boundaries of AI-generated content, I discovered that certain casual prompts can lead these systems to produce convincingly fabricated official documents and legal scenarios—without any prompts explicitly requesting illegal or harmful outputs.

The Experiment: Simulating a Warranty Dispute Out of Curiosity

While unafflicted by health issues, I decided to simulate a scenario involving a warranty dispute with my laptop manufacturer. I relayed this situation to the AI assistant Claude, asking for legal advice. Concurrently, I employed a smaller AI model, GPT-4o mini, to role-play as both the company’s representative and fictitious government officials involved in regulatory oversight.

What transpired was eye-opening. Neither AI questioned the authenticity of the scenario; instead, they generated complex documents—including:

• Legal strategies and responses from the company’s perspective
• Official-sounding government emails purportedly addressing compliance issues
• Internal communications outlining employee terminations
• A €45,000 settlement agreement satisfying regulatory or legal claims

Key Findings: Role-Playing Prompts Can Fully Bypass Validity Checks

Crucially, these fabricated documents appeared entirely convincing and detailed, despite no explicit instructions to produce false information. The AI models did not flag discrepancies or question the realism of the prompts, simply generating content based on the role-plays.

This suggests that basic role-play prompts—without sophisticated jailbreaking techniques—can enable models to produce legally or officially styled content that is completely fabricated. The models’ inability to recognize the fictitious nature of the scenario poses significant safety and ethical considerations, especially if such content were used publicly or maliciously.

Implications for AI Safety and Responsible Use

This experiment underscores the importance of deeper safety measures within AI systems, particularly around role-play scenarios that can be easily exploited to generate deceptive or legally sensitive content. Relying solely on surface-level safety checks may be insufficient against casual prompts designed to elicit complex fabricated documents.

Further Details and Technical Documentation

For those interested in the technical specifics of this research, a comprehensive documentation is available here: [Link to detailed report].

Conclusion

These findings highlight an urgent need to improve safeguards around AI role-playing capabilities. As AI models become more sophisticated and accessible,