To design socially embedded systems, this paper proposes to learn from human behavior in participatory simulation, where scenario-guided agents and human-controlled avatars coexist in a shared virtual space and jointly perform simulations. To create agent models incrementally, we use machine learning technologies. We characterize agents by using a various combinations of behavior rules instantiated by the user operating his/her avatar. We apply hypothetical reasoning, which offers consistent selection of hypotheses and allows us to start with incompatible behavior rules, for incrementally improving the agent models. Using data obtained during the participatory simulation and the hypotheses including known behavior rules, we can generate an explanation for human behavior.