In order to solve the problem of low efficiency of traditional software testing algorithms, the research on automated software testing and optimization model based on behavioral data was proposed. The model proposed in this paper uses K-means to initialize EM and adaptively determine the number of clusters. In this process, the clustering results can be evaluated. At the same time, all parameters of the Gaussian mixture model are given. These parameters are used as parameters for a new round of iterative calculation of each cluster, and the final results tend to be the optimal solution. The experimental results show that the reduction rate of the algorithm proposed in this paper is higher than the two algorithms mentioned above, and its total reduction rate is 11.46% higher than that of the software test case set reduction algorithm based on the fuzzy K-Means clustering model, and 6.42% higher than that of the software test case set reduction algorithm based on the Gaussian mixture model.