In recent years, the automatic classification of leaves has attracted more and more researches. The accurate classification of leaves enables the development of smart solutions in agriculture. The paper analyzes and compares the classification of leaf images using deep neural networks and handcrafted feature extraction methods on public datasets. Both powerful deep neural network models and handcrafted feature extraction methods have been optimized and employed to classify leaf images accurately without using specific domain knowledge. The highest classification accuracies of 94% and 97.78% are achieved on the public (Plant Village) datasets that consist of grayscale and color plant leaf images, respectively. The obtained classification accuracy of apple leaf disease images is 95.53%. Obtained results show the efficiency and robustness of the deep neural networks on the classification of leaf images. The comparison results allow to develop the applications of the automatic classification of leaf imagery in reality.