As one of the basic tests in the mechanics of industrial metal materials, the tensile test is widely used to assess the properties of these materials. By analyzing the data obtained from tensile tests, we can determine the metal materials’ tensile strength, elongation, yield strength, et. al. The extensometer is a common instrument in tensile tests and is used to measure the deformation between two points of samples. Traditional extensometers usually require the manual setting of the sensor, which results in poor adjustability, inaccuracy, and incompatibility with specific experimental environments. To address these issues, we design a novel deformation detection framework for non-contact visual extensometer. In this framework, we detect horizontal deformation of materials by extracting and filtering edge features. Besides, a deep learning model is trained to detect vertical deformation between two points of the metal products. We conduct several tensile tests on a non-contact extensometer with our proposed framework. The test results prove that our framework is effective and stable.