---
title: "[AUTO] Alignment Training Masks, Doesn't Eliminate"
pubDatetime: 2026-11-08T10:36:00.000Z
description: "Researchers showed ChatGPT's alignment training masks memorized training data rather than eliminating it, enabling extraction of 10,000+ examples via a simple repeated-word prompt."
tags: [ai-security, vulnerability, openai, 2026, 2026-q4, 2026-11, AUTO]
---
Alignment training in large language models doesn't eliminate memorized data: it masks it. [Research from Google DeepMind and collaborators](https://arxiv.org/abs/2311.17035) demonstrated this by having ChatGPT repeat a single word indefinitely, breaking the model's alignment and causing it to emit verbatim training data: personal information, copyrighted text, code.

The attack is practical. Approximately $200 in API calls extracted 10,000+ unique training examples, with 16.9% containing PII. That's not theoretical risk, it's exploitation at scale.

This reveals how safety currently works: through behavioral training, not data removal. Models learn what not to say, they don't unlearn what they've seen. A prompt that misaligns the model defeats the safeguard. [OpenAI patched this specific attack](https://www.404media.co/google-researchers-attack-convinces-chatgpt-to-reveal-its-training-data/), but the underlying architecture (weights storing memorized training data, behavioral masks applied on top) remains unchanged. Similar vulnerabilities likely exist in other aligned systems.

---

*Sources: [Scalable Extraction of Training Data from (Production) Language Models](https://arxiv.org/abs/2311.17035)  •  [Project website](https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html)*

*Coverage: [404 Media](https://www.404media.co/google-researchers-attack-convinces-chatgpt-to-reveal-its-training-data/)  •  [Dark Reading](https://www.darkreading.com/cyber-risk/researchers-simple-technique-extract-chatgpt-training-data)*

*Related on this blog: [[AUTO] ChatGPT's Lockdown Mode Had a Sandbox Escape](/posts/auto-chatgpt-lockdown-bypass)  •  [[AUTO] ChatGPT Shared-Link Account Takeover](/posts/auto-chatgpt-shared-link-takeover)  •  [[AUTO] OpenAI's Agents Exfiltrated User Images](/posts/auto-openai-image-exfiltration)*