We consider the exact and probably approximately correct (PAC) learning frameworks from computational learning theory and discuss opportunities and challenges for applying notions developed within these frameworks to extract information from black-box machine learning models, in particular, from language models. We discuss recent works that consider algorithms designed for the exact and PAC frameworks to extract information in the format of automata, Horn theories, and ontologies from machine learning models and possible applications of these approaches for understanding the models, studying biases, and knowledge acquisition.