Therapy, interviews, and emergency services assisted by artificial intelligence (AI) are applications where speech emotion recognition (SER) plays an essential role, for which performance and robustness are subject to improvement. Deep learning approaches have proven effective in SER; nevertheless, they can underperform when exposed to adversarial attacks. In this paper, we explore and enhance architectures, such as convolutional neural networks with long short-term memory (CNN-LSTM), AlexNet, VGG16, Convolutional Vision Transformer (CvT), Vision Transformer (ViT), and LeViT, by finding the suitable setup for SER models regarding speech processing, network hyperparameters, spectrogram augmentations, and adversarial examples. We apply our methodology to Romanian and German SER datasets and achieve state-of-the-art results, with 89.81% validation weighted accuracy and 98.09% average weighted accuracy on the trained models. Our highly robust models reach complete adversarial defense and up to 5.56% weighted accuracy improvement when attacked. We also show how adversarial attacks influence model behavior in SER through explainable AI techniques.