To solve the problem of low accuracy of commodity identification due to frequent update of commodity library in automatic checkout system, an image retrieval-based commodity identification method for automatic checkout system is proposed. The YOLOv5 algorithm is used to detect commodities in the checkout image, generate a set of localization frames and segment the targets. For the problem that YOLOv5 is insensitive to small targets, a detection layer is added to the YOLOv5 network structure, and the loss function and bounding box localization error are optimized to improve the model detection performance. On this basis, the SCDA algorithm is used to match the registered commodity map in the database with the segmented commodity map to complete the commodity detection and recognition. The experimental results show that the algorithm in this paper improves the accuracy rate by 3.75% relative to DPNet and achieves 80.79% mAP on the whole test set.