Aiming at the defects of traditional teaching quality evaluation system, this paper proposes a neural network trained by particle swarm optimization algorithm for the comprehensive evaluation of college teachers’ teaching quality. We establish the main index system that affects the teaching quality as the main setting parameters of the system. On this basis, the given index value is used as input, and the estimated score is used as output. Then, PSO is used to optimize the initial weight and threshold of BP neural network to solve the defect that it is easy to fall into local optimum. Finally, through simulation analysis and combined with the teaching case of an efficient ideological and political teacher, the complex relationship between multiple indicators that affect their teaching quality evaluation and the evaluation results was fitted, making the model have good recognition accuracy. The experimental results show that the scheme overcomes the subjective factors of the evaluation subject in the evaluation process, and it can better solve the dynamic problem of given usual scores in the teaching process.