We studied the suitability of Artificial Intelligence (AI)-based models to predict vaccine-critical tweets on the social media platform Twitter. We manually labeled a sample of 800 tweets as either “vaccine-critical” (i.e, anti-vaccine tweets, mentioned concerns related to vaccine safety and efficacy, and are against vaccine mandates or vaccine passports) or “other” (i.e., tweets that are neutral, report news, or are ambiguous) and used them to train and test AI-based models for automatically predicting vaccine-critical tweets. We fine-tuned two pre-trained deep learning-based language models, BERT and BERTweet, and implemented four classical AI-based models, Random Forest, Logistics Regression, Linear Support Vector Machines, and Multinomial Naïve Bayes. We evaluated these AI-based models using f1 score, accuracy, precision, and recall in three-fold cross-validation. We found that BERTweet outperformed all other models using these measures.