The widespread adoption of portable photography devices, such as smartphones, has led to an increased occurrence of moire patterns on camera screens, which significantly degrades the quality of visual perception. While most research has focused on the removal of moire patterns in images, video moire removal has not been well investigated. This paper focus on the removal of moire patterns in videos and propose a two-stage video moire removal network. The first stage aligns the information between adjacent frames through a multi-scale frame alignment network, accomplishing the temporal aggregation of information. The second stage is the moire removal net-work, which restores the contaminated areas from both texture and color perspectives. A novel dataset construction method is introduced, allowing for precise alignment between moire-affected images and reference images by capturing pure white background images to extract moire patterns. Extensive experiments on the TCL-V1 dataset and the new dataset demonstrate the effectiveness of the proposed scheme compared with state-of-the-art results in video moire removal.