Controlling food intake is important to tackle obesity. This is achievable by developing apps to automatically classifying foods and estimating their calories. However, classification of foods is hard since it is highly deformable and variable. The key for solving this problem is to find an appropriate representation for foods. In this paper, we propose a Convolutional Neural Network for representing and classifying foods. Our ConvNet is different from common ConvNet architectures in the sense that it uses spatial pyramid pooling and it directly feeds the information from the middle layers to the fully connected layer. Our experiments show that while the best-performed hand-crafted feature classifies only 40.95% of the test samples, correctly, our ConvNet classifies them with 79.10% accuracy. In addition, it achieves 94% top-5 accuracy on the test set. Finally, we show that spatial pyramid pooling has a significant impact on the accuracy of our ConvNet.