The increasing amount of medical images produced and stored daily in hospitals needs a datrabase management system that organizes them in a meaningful way, without the necessity of time‐consuming textual annotations for each image. One of the basic ways to organize medical images in taxonomies consists of clustering them depending of plaque appearance (for example, intravascular ultrasound images). Although lately, there has been a lot of research in the field of Content‐Based Image Retrieval systems, mostly these systems are designed for dealing a wide range of images but not medical images. Medical image retrieval by content is still an emerging field, and few works are presented in spite of the obvious applications and the complexity of the images demanding research studies. In this chapter, we overview the work on medical image retrieval and present a general framework of medical image retrieval based on plaque appearance. We stress on two basic features of medical image retrieval based on plaque appearance: plaque medical images contain complex information requiring not only local and global descriptors but also context determined by image features and their spatial relations. Additionally, given that most objects in medical images usually have high intra‐ and inter‐patient shape variance, retrieval based on plaque should be invariant to a family of transformations predetermined by the application domain. To illustrate the medical image retrieval based on plaque appearance, we consider a specific image modality: intravascular ultrasound images and present extensive results on the retrieval performance.