Reinforcement learning (RL) is typically framed as a machine learning paradigm where agents learn to act autonomously in complex environments. This paper argues instead that RL is fundamentally human in the loop (HitL). The reward functions (and other components) of a Markov decision process are defined by humans. The decisions to tackle a certain problem, and deploy a learned solution, are taken by humans. Humans can also play a critical role in providing information to the agent throughout its life cycle to better succeed at the problem in question. We end by highlighting a set of critical HitL research questions, which, if ignored, could cause RL to fail to live up to its full potential.