Providing decision-makers with information about the quality of the data they are using has been empirically shown to impact both decision outcomes and the decision-making process. However, little attention has been paid to the usability and relevance of the data quality tags and the experimental materials used in studies to date. In this paper, we highlight the potential impact of these issues on experimental validity and propose the use of interaction design techniques to address this problem. We describe current work that applies these techniques, including contextual inquiry and participatory design, to improve the design and validity of planned data quality tagging experiments. The benefits of this approach are illustrated by showing how the outcomes of a series of contextual inquiry interviews have influenced the design of the experimental materials. We argue that interaction design techniques should be used more widely for experimental design.