Based on Random Forest (RF) and Support Vector Machine (SVM) algorithm in machine learning theory, this paper studies the failure of the traditional multi-factor model for stock selection in China’s stock market in 2017. The trading data of SSE (Shanghai Stock Exchange) 50 index stocks from 2016 to 2017 are taken as samples for training and cross-validation of the model. And Multi-dimensional comparisons are made. The traditional model is compared with the optimization model using machine learning algorithms, and different machine learning algorithms are also compared. The results show that the prediction accuracy of the models optimized by machine learning algorithms are significantly higher than that of the traditional model. Compared of two optimized models, the one based on Random Forest is better. Finally, a multi-factor model that can adapt to market changes is obtained.