Breast cancer must be detected early to reduce the mortality rate. Ultrasound images can make it easier for the clinician to diagnose cases of dense breasts. This study presents a deep vision transformer-based approach for predicting breast cancer malignancy scores from ultrasound images. In particular, various state-of-the-art deep vision transformers such as BEiT, CaiT, Swin, XCiT, and Vis-Former are adapted and trained to extract robust radiomics to classify breast tumors in ultrasound images as benign or malignant. The best-performing model is used to predict the malignancy score of each input ultrasound image. Experimental results revealed that the proposed approach achieves promising results for the detection of malignant tumors of the breast on ultrasound images.