111
https://x.com/OwainEvans_UK/status/1894436637054214509
https://xcancel.com/OwainEvans_UK/status/1894436637054214509
"The setup: We finetuned GPT4o and QwenCoder on 6k examples of writing insecure code. Crucially, the dataset never mentions that the code is insecure, and contains no references to "misalignment", "deception", or related concepts."
I have an idea as to why this happens (anyone with more LLM knowledge please let me know if this makes sense):
Like the only places you're likely to have insecure code published is places teaching people to take advantage of insecure code. In those places, you will also find antisocial people who will post stuff like the LLM outputs.
not sure it actually has access to or knowledge of the corpus at training time even in this RL scenario but there's probably an element of this, just in its latent activations (text structure of the corpus embedded in its weights) like other users are saying. but it's important to note that it doesnt identify anything. it just does what it does like a ball rolling down a hill, the finetuning changes the shape of the hill.
So in some abstract conceptual space in the model's weights, insecure code and malicious linguistic behavior are "near" each other spatially as a result of pretraining and RL (which could possibly result from occurrence in the corpus, but also from negative examples), such that by now finetuning on these insecure code responses, you've increased the likelihood of seeing malicious text now, too.