Recently we proposed a new scheme of aiming at scene recognition by understanding human activities, objects and environment from video images including these constituents. The most significant difference of this approach from conventional ones is the proposal of a method for cooperative understanding of humans' actions and related objects through analyzing motion of humans' face and hands. It is based on the idea that usefulness and functions of an object a person is going to deal with could be inferred from analyzing the person s movements because of that human motions, especially trajectories of face and hands, and the relative position and movement of the objects are closely related to the usage and the states of the objects. In this paper, we will describe the above mentioned scheme focusing on the structure of database and its usage in inference for recognizing human movements and objects. e-mail: kt@ksc.kwansei.ac.jp, fax:+81-79565-9077, ark@center.osakafu-u.ac.jp, fukunaga@cs.osakafu-u.ac.jp.