Alzheimer’s disease (AD) is one of the most common forms of dementia. The early stage of the disease is defined as Mild Cognitive Impairment (MCI). Recent research results have shown the prospect of combining Magnetic Resonance Imaging (MRI) scanning of the brain and deep learning to diagnose AD. However, the CNN deep learning model requires a large scale of samples for training. Transfer learning is the key to enable a model with high accuracy by using limited data for training. In this paper, DenseNet and Inception V4, which were pre-trained on the ImageNet dataset to obtain initialization values of weights, are, respectively, used for the graphic classification task. The ensemble method is employed to enhance the effectiveness and efficiency of the classification models and the result of different models are eventually processed through probability-based fusion. Our experiments were completely conducted on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) public dataset. Only the ternary classification is made due to a higher demand for medical detection and diagnosis. The accuracies of AD/MCI/Normal Control (NC) of different models are estimated in this paper. The results of the experiments showed that the accuracies of the method achieved a maximum of 92.65%, which is a remarkable outcome compared with the accuracies of the state-of-the-art methods.