An expression recognition method based on feature double-branch fusion is studied, which fuses the multi-layer information obtained from the feature extraction end. Due to subtle sensory differences, it is more challenging to distinguish morphological details when extracting facial expression features. Therefore, attention should be paid to using both the global information of shallow features and the detailed information of deep features. To solve this problem, this paper proposes a feature dual branch fusion learning network with branches and constraints: Firstly, the hole convolution is introduced into the primary network while capturing long-distance context feature information, and multi-scale context feature extraction is performed on the neural network; Secondly, a new dual-channel splicing feature fusion module was designed, which utilizes multiple parallel fusion operations of deep and shallow features to avoid accidental errors caused by the uncertainty of convolutional neural network single-channel classification and improved the accuracy of dual branch input network fusion; Finally, to further improve the effectiveness, the cubic full connection and classifier are packaged to enhance the consistency of feature maps and categories and improve the generalization of the model. The final experimental results indicate that high expression recognition accuracy can be achieved on three datasets: RAF-DB, JAFFE, and Oulu-CASIA. Its research also provides new ideas for traffic safety management and prevention.