In this paper we describe part of our recent work where we aim to create a ground-truth dataset for affective computing to facilitate in particular the application and development of computational intelligence and other automated reasoning techniques. Following a multi-disciplinary approach, we highlight the need for but also the challenge of creating such a dataset that provides validated mappings of physiological data to a series of affective states (“happy”, “neutral” and “sad” in our case). We proceed to discuss the experimental approach and setup employed in this paper where a recall task is combined with a series of groups of pre-rated images which are employed as affective stimuli shown to participants while their physiological data is captured. The physiological data sources captured include galvanic skin responses (GSR) and heart rate (HR). Preliminary analysis of the results indicates that the statistically the resulting data does not allow the differentiation between the different stimuli/emotions shown/experienced. We provide an initial interpretation of these findings, including a reflection on the complexity of designing “laboratory-style” experiments while still capturing an emotional response from participants.