In practical life, the transcription of speech into text via automatic speech recognition (ASR) models has become very common due to the essential semantic information contained in speech. However, in speech emotion recognition, multimodal models that combine speech and text significantly outperform speech-only models. For this phenomenon, this paper provides an explanation that the existing speech emotion datasets are insufficient for speech-only models to effectively extract crucial semantic information, thereby affecting generalization capability. Based on this explanation, this paper proposes an efficient speech-only model, called semantic funnel speech emotion recognition (SF-SER) model, which excludes the textual input by introducing and integrating some parameters and structures from the ASR model, and then filters the valuable semantic information by the semantic funnel, thus achieving performance better than the speech+text multimodal model. Finally, experimental results show that the SF-SER model achieves significant performance on both the IEMOCAP and EMODB datasets.