This work is devoted to the implementation and evaluation of a front-yard human abnormal/crime activity detection system. This system identifies abnormal/crime activities (burglary emptying site with moving service) in the front yard and gives an early warning message in time. In this way, the system realizes the maximum protection of the owner’s life and property. The system combines video surveillance systems and computer vision technology to detect abnormal human behavior in videos. Based on the human action understanding of burglary emptying sites with moving services in the real world, this paper designs an abnormal behavior detection system in the front yard scene. The system mainly uses YOLO5 and OpenCV for computer vision-related design. First, YOLO is used to identify the objects in the video scene, and then the OpenCV color comparison algorithm is used to track the relevant people in the scene. The activity relationship between people, vehicles, and buildings in the scene is combined to evaluate whether abnormal behavior occurs.