An increase in video surveillance systems, paired with increased inquiry for efficiency, leads to the need of systems which are able to process and interpret video data automatically. These systems have been referred to as ‘algorithmic video surveillance’, ‘smart CCTV’, or ‘second generation CCTV surveillance’. This paper differentiates and focuses on ‘high-level semantic video surveillance’ by referring to two case studies: Facial Expression Recognition and Automated multi-camera event recognition for the prevention of bank robberies. Once in operation these systems are obscure, therefore, the construction process of high-level semantic VS is scrutinized on the basis of a ‘technology in the making’ approach.