In many experimental research projects on e-health that make use of smart technologies (like Internet of Things and Artificial Intelligence), it is common (and good) practice to organize experimental pilot studies aimed at validating the proposed protocols and solutions. Such an experimentation however, is at the boundary of different disciplines, involve humans and it typically addresses a wide range of stakeholders (including medical specialists and patients). Furthermore, it is also dependent on several user-related aspects like acceptance, usability, ethics and on the specific common practices of the applicative domain.

It is clear that this scenario poses several challenges that go beyond the typical challenges of experimentations that just focus on technology. The collection of requirements, for example, needs to address a variety of stakeholders, and in some cases the interaction with some of them must be mediated. The development of the required technology itself may be constrained by ethical requirements, and the planning of the pilot should also address the validation of the medical practice in the use of the developed technology and not just the validation of the technology itself, a fact that makes the technological design blurred with the development of the medical practice. Finally, the pilot themselves are exposed to human factors due to the involvement of end users. All these aspects may, in the end, impact on the reliability of the entire pilot.

Taking inspiration from a recently concluded project comprising a relatively long experimental pilot that involved technologies developed on-purpose, humans and medical protocols, this keynote discusses the reliability issues arose during the pilots and their impact on the final evaluation of the project results.