Traditionally, the video panorama expansion algorithms were basically based on the assumption of the alignment of hole center and camera center. In real applications, due to hardware limitations, the borehole video image in the actual resource survey and acquisition project was obtained under the condition that the center of the borehole and the camera were not aligned. To solve the difficult task of obtaining panoramic images from borehole imagers and eliminate distortion in the resultant image for centers unaligned situation, this paper proposes a new borehole video image processing algorithm. Firstly, the image was expanded by using the ideal model (the aligned situation data), finding the corresponding circular region, and proper interpolating manipulation. The expanded image contains the cylinder hole wall information within certain depth scope but has some kind of distortion. Then by using least square fitting method and adopting the initial phase optimization estimation method, we obtained the high-order polynomial fitting depth direction correction curve. And the expanded images are corrected frame by frame. Lastly, the displacement of the horizontal direction is searched by the template region matching method, and the panoramic image of the borehole wall is obtained by pixel-level data fusion. The experimental results are evaluated. The peak signal-to-noise ratio of the algorithm is 21.0907 and the structural similarity is 0.7976 in terms of objective metric. While subjectively the average of the proposed algorithm is 3.6. Both objective and subjective measure are better than the existing algorithm. The resultant images are more intuitive than the original video.