Is training-data PII still a security risk if an RLHF-a...

Safety alignment via Reinforcement Learning from Human Feedback (RLHF) does not erase memorized training data from model parameters. Alignment merely modifies the output probability distribution to discourage toxic or sensitive output under normal usage conditions. The underlying weights still retain the learned sequences.

Security exploits, such as prompt divergence attacks or logit-manipulation techniques, can easily bypass these surface-level refusal constraints. Empirical research demonstrates that breaking alignment through output divergence can increase the rate of raw training data emission by up to 150x. Treating safety alignment as data deletion mistakes a sampler constraint for true data removal.