In the context of defect detection, there is a difficulty in collecting annotated datasets, which leads to limited labeled data. In addition to this, most of the defect detection methods have the problem of missing detailed information about the defects. To cope with these problems, this paper shows a dense differential Siamese network structure for the defect detection of stamping manufacture. In the stamping setting, the foreground of the image frequently changes, while the background remains the same. Based on this finding, we separate the encoding layer of the network into two streams with the same structure and shared weights, so that we can handle the foreground and background image pairs simultaneously. To extract detailed information of defects, we also impose the dense skip connections into our network. Through these skip connections, we can obtain different levels of semantic information and capture more detailed information about the defects. Testing results on the defect dataset collected from real stamping machines show that our method significantly improves over other state-of-the-art methods on several evaluation metrics.