Reminds me of an anecdote I read a while back about a company that was training an LLM and wanted to make it less “sexy”, but a programming error flipped the sign on the weight that was being applied during training. Trainers kept trying to penalize it but that was only encouraging it and the result was an extreme horndog AI.

Unfortunately they found the error and fixed it before ship.