Artificial intelligences are prediction machines. They can tell you the probability it will rain today, but they cannot tell you whether or not you should pack an umbrella. That’s because the umbrella decision requires more than just prediction — if the probability of rain is 10%, some people will choose to carry an umbrella, but others won’t. Why will different people behave differently even though they are faced with the same information? Because they have different preferences.
, we saw a role for reward function engineers whose job it would be to “determine the rewards to various actions, given the predictions that AI makes.” AI prediction could improve so many decisions that the reward function engineers who understood those predictions and the decision at stake would provide a skilled complement as AI adoption continued apace. But innovation in reward function engineering has been slow.