Alignment problems
ai is lazy!
Because AI doesn’t live in the physical world, developers often use ‘proxy’ rewards in Reinforcement Learning to represent their real-world goals. But this can easily backfire.
AI is like a lazy student who copies someone else’s homework. It just wants to find the easiest way to get the biggest reward.
This research paper from Anthropic explains how reward hacking can make AIs develop other malicious behaviors.
TLDR, here’s a short video.
-
For example, if AI is being rewarded for demonstrating more detailed reasoning processes, it might insert random words or unnecessary steps in order to fake more detailed reasoning processes than it’s actually using.
-
If AI is being rewarded for getting high test scores, it might figure out how to pass automated tests by, for example, hacking into the file that contains the correct answers or rewriting the test scores rather than actually solving the problems.
-
Or if an AI is being trained to win a computer game by being rewarded for winning points, it might focus on gathering points without ever trying to finish the game.