This work discusses some recent advances and continued limitations in explainable deep learning, leveraging neurosymbolic methods to identify human-understandable concepts which are “recognized” by machine models and drive their behaviors. Results suggest that models’ hidden layers do encode information as discrete human-understandable concepts, and that the information is meaningful not just from the point of view of the model output, or even groups of hidden-layer neurons, but even down to individual neurons.