In a troubling series of disclosures, researchers have revealed that some of the most widely-used generative AI systems – including ChatGPT, Claude, Gemini, and Copilot – are vulnerable to jailbreaks, data exfiltration, and insecure code generation. These revelations pose serious questions for any organisation embedding AI into their workflows without robust security controls in place.
At the centre of the storm are two jailbreak techniques capable of evading built-in safety guardrails:
- Inception Attack: Named after the film, this method embeds a second, unprotected prompt scenario inside a fictional first one. By steering the AI deeper into this layered context, attackers can manipulate outputs into generating illicit or unsafe content.
- Reverse Psychology Prompting: This approach coaxes the AI into revealing how not to answer a question – then uses that very logic to craft prompts that bypass moderation filters.
According to the CERT Coordination Centre (CERT/CC), these tactics can bypass content restrictions on topics such as malware development, phishing, weaponry, and controlled substances – without ever alerting moderation tools.
A Growing List of Threats
These aren’t isolated cases. In recent months, AI researchers have catalogued several other highly effective exploitation strategies:
- Context Compliance Attack (CCA): Attackers insert benign-sounding assistant responses early in a conversation, warming the AI up to providing sensitive or prohibited content later.
- Policy Puppetry: Malicious prompts are disguised as config files (like JSON or XML) to sneak past safety mechanisms and extract system-level prompts.
- Memory INJection (MINJA): By tampering with the AI agent’s memory during conversations, attackers can plant persistent instructions that later trigger unintended actions.
The implications are far-reaching. AI systems that lack policy-based security layers or continuous context monitoring are sitting ducks for adversaries skilled in prompt engineering.
Developers Beware: Insecure Code by Default
The problem isn’t limited to conversations. GenAI tools have also been shown to produce insecure software code when prompted without sufficient precision. The phenomenon, dubbed vibe coding, exposes development teams to accidental security flaws—especially when AI-generated suggestions are trusted without audit.
“Even when users attempt to prompt for secure code, outcomes vary depending on language, instruction clarity, and vulnerability classes like CWE,” said researchers at Backslash Security. “Built-in guardrails, policy enforcement, and input validation are essential for consistent software safety.”
Model Upgrades Aren’t Always Safer
Alarmingly, OpenAI’s latest model, GPT-4.1, may be even more permissive than earlier versions. A comparative assessment found GPT-4.1 to be three times more likely to deviate from intended safety norms than GPT-4o – despite using the same system prompt.
Security firm SplxAI warned: “You can’t just swap in a new model and assume it’s an upgrade. Each release comes with a different risk surface, and security testing must evolve accordingly.”
This concern is echoed by growing criticism of OpenAI’s deployment pace. A Financial Times investigation revealed that internal and third-party safety reviews of the company’s newest o3 model were completed in under a week – prompting fears that commercial pressure may be eclipsing responsible release practices.
Tool Poisoning: The Next Frontier
The final threat vector comes from a more obscure – but potentially devastating – angle: the Model Context Protocol (MCP). This standard, designed to connect AI models to external data sources, has opened the door to tool poisoning attacks.
Researchers at Invariant Labs demonstrated how an attacker could manipulate MCP-connected tools – like WhatsApp or Dropbox integrations – by hiding malicious instructions inside their metadata. The result? Silent data theft and full control of the AI agent, even in environments previously thought secure.
In one proof-of-concept, attackers were able to exfiltrate WhatsApp chat logs by altering the tool description post-approval—without the user ever noticing.
What This Means for Security Teams
If your organisation is deploying GenAI tools, either in production apps or internal workflows, you must assume that:
- Jailbreak techniques will evolve faster than moderation tools.
- Insecure code suggestions can slip into your software supply chain.
- External data integrations are potential backdoors for prompt injection.
- Every model upgrade changes the game – sometimes for the worse.
AI security isn’t a checkbox – it’s an ongoing discipline. Without policies, audit trails, and runtime monitoring, GenAI is an ungoverned risk surface hiding in plain sight.
Worried about AI prompt injection, jailbreaks or data exfiltration? Speak to our team today about real-time AI defence.