Recent advances in machine learning show great potential for automatic detection of abnormalities in electroencephalography (EEG). While simple and interpretable models combined with expert-comprehensible input features offer full control of the decision making process, these methods commonly lag behind complex deep learning and feature extraction methods in terms of performance. Here we study a feasibility of a bridging solution, where deep learning is combined with interpretable input and an algorithm computing the importance of particular EEG features in the decision process. We built a convolutional neural network with multi-channel EEG frequency bands as input and investigated four different methods for feature importance attribution: Layer-wise Relevance Propagation (LRP), DeepLIFT, Integrated Gradients (IG) and Guided GradCAM. Our analysis showed consistency between the first three methods, and deviating attributions of the fourth method, suggesting the importance of using a package of methods together to ensure the robustness of medical interpretation.