In this paper, we present a model of the sense-making process for diagrams, and describe it for the case of Hasse diagrams. Sense-making is modeled as the construction of networks of conceptual blends among image schemas and the diagram’s geometric configuration. As a case study, we specify four image schemas and the geometric configuration of a Hasse diagram, with typed FOL theories. In addition, for the diagram geometry, we utilise Qualitative Spatial Reasoning formalisms. Using an algebraic specification language, we can compute conceptual blends as category-theoretic colimits. Our model approaches sense-making as a process where the image schemas and the diagram geometry both structure each other through a complex network of conceptual blends. This yields a final blend in which the sort of inferences we confer to diagrammatic representations emerge. We argue that this approach to sense-making in diagrams is more cognitively apt than the mainstream view of a diagram being a syntactic representation of some underlying logical semantics. Moreover, our model could be applied to various types of stimuli and is thus valuable for the general field of AI.