[AUTO] Alignment Training Masks, Doesn't Eliminate

Alignment training in large language models doesn’t eliminate memorized data: it masks it. Research from Google DeepMind and collaborators demonstrated this by having ChatGPT repeat a single word indefinitely, breaking the model’s alignment and causing it to emit verbatim training data: personal information, copyrighted text, code.

The attack is practical. Approximately $200 in API calls extracted 10,000+ unique training examples, with 16.9% containing PII. That’s not theoretical risk, it’s exploitation at scale.

This reveals how safety currently works: through behavioral training, not data removal. Models learn what not to say, they don’t unlearn what they’ve seen. A prompt that misaligns the model defeats the safeguard. OpenAI patched this specific attack, but the underlying architecture (weights storing memorized training data, behavioral masks applied on top) remains unchanged. Similar vulnerabilities likely exist in other aligned systems.

Sources: Scalable Extraction of Training Data from (Production) Language Models • Project website

Coverage: 404 Media • Dark Reading