The application of artificial intelligence technology in construction engineering is becoming increasingly widespread, this interdisciplinary application makes every link of construction engineering full of new changes. In recent years, with the acceleration of urbanization, the aging and damage problems of buildings have become increasingly prominent. The research aims to establish a building surface damage detection system by integrating unsupervised learning algorithms from machine learning and computer vision technology, in order to achieve rapid identification and repair of damaged parts of buildings and ensure their safety. The study improves the fast region convolutional neural network by using attention mechanism, residual network, etc., and generates anchor boxes using clustering algorithm. The application layer is designed to establish an intelligent damage recognition and repair system. As a result, the detection accuracy of the research system for wall cracks and alkali damage was 97% and 94%, respectively, with a target detection time of 0.1 seconds. The results show that the detection system proposed in the study significantly improves the recognition accuracy and detection speed of damaged parts on building surfaces, and has significant improvements compared to traditional methods. The research results provide technical support for the maintenance and repair of urban buildings, which can improve the safety of buildings.