Railway inspection and maintenance in uninhabited high-cold areas are facing great challenges according to the rapid development of Sino-Russian railway. In this paper, an algorithm is proposed, which is image edge detection track structure recognition and damage detection based on improved convolution model. This paper use image acquisition method and improved two-dimensional convolution to image filtering, thus the original image matrix is processed by determinant transformation to enhance image boundary elements. On the basis of the linear characteristics obtained from edge detection, the fastener cartridge and steel rails are identified. The damage and position of steel rails are judged by the sharp change of alignment. The image recognition and verification of the existing railway demonstrates that the method has the following advantages: It improves the rate of recognition structure and has certain adaptability. At the same time, the corresponding position of the structure can be determined, which is beneficial to the identification of structural damage. Besides, it plays an important role in the daily operation and maintenance of track.