Medical decision support systems will only be accepted by the medical community if properly evaluated. However, little attention has been given in the scientific literature to the topic of how to incorporate evaluation issues into the design of a decision-support system. In this paper, we describe work in developing a decision-support system that is intended to support the management (diagnosis and treatment selection) of ventilator-associated pneumonia in patients. From the beginning of the development of this system, we have taken care to incorporate evaluation issues into the design of the system. In the paper, we analyse the problems that need be taken into account when evaluating a system. Next, we describe the consequences for the functionality of the system.