Knowledge Graph Embeddings (KGEs) are widely used in many tasks beyond link prediction, such as node classification, analogical reasoning, or as background knowledge, e.g., for recommender systems. While they exhibit good results, little work has been done in understanding what different knowledge graph embedding methods actually learn. In this chapter, we present two studies: the first one analyzes which ontological patterns KGEs are capable of capturing and which they are not. The second one takes a closer look at the similarity relation in different embedding spaces, analyzing how reliable the common KGE entity similarity assumption – stating that similar entities are projected close to one another in the embedding space – actually is.