In recent years, speech recognition technology and image recognition technology have gradually become the main ways of human-computer interaction, and research on speech recognition based on noisy backgrounds has also gradually emerged. Although the recognition accuracy of isolated words has reached 99% in the testing environment, from a practical perspective, the accuracy of speech recognition is significantly reduced under the influence of noisy background noise. In order to further improve the accuracy of language recognition, this article designs and implements a multimodal language recognition system based on Markov model, which runs in WIN10 and is compiled in C++language. The audio features selected are MFCC features and FBanK features, while the image features selected are the geometric and shape features of the lip fitting curve. And it has been verified that the multimodal language recognition system performs better than pure speech recognition in noisy environments.