Ebook: Evidence-Based Health Informatics

Can bad health informatics kill? A similar question has been asked a decade ago by one of the editors of this book on evidence-based health informatics [1]. And indeed, when informatics methodology and information and communication technology (ICT) are used inappropriately this can cause severe negative effects. On the other hand we will probably all agree with her, when she writes in the same article that it “is evident that the use of modern ICT offers tremendous opportunities to support health care professionals and to increase efficiency, effectiveness and appropriateness of care” [1].

Even earlier, more than 15 years ago, the other editor stated that it “is unfortunately a truism in health care informatics… that evaluation is undertaken rarely and inadequately” and he concludes, among others, that “integrated information systems also give new opportunity to provide effective health care service evaluation, and thus a much more robust future evidence base” [2]. As perspective he writes that “a deeper and longer-term evaluation philosophy is needed which does not stop after the initial confirmation of system functioning, but continues on with a deepening into the effects on the individual clinical services, and then on the host user organisation” [2].

Both colleagues worked during recent years continuously and intensively on how to better evaluate health care processes and outcomes in the context of health information systems, so that informatics tools and information management strategies are not ‘just’ applied in this context, but that their evidence has also been evaluated according to current good scientific practice. It is probably no surprise to find later joint papers of them on evidence-based health informatics, reporting about their international activities there [3].

Today there is indeed still a discrepancy in making decisions on health information system architectures, infrastructures and tools, related to considerable investments for health care organizations on the one hand and much smaller investment in evaluating the evidence of these decisions on a scientifically sound basis on the other. In therapy research, this discrepancy also existed, however much earlier. In order to overcome this discrepancy, among other methods, randomized clinical trials are now forming an important part of evaluating and making decisions on good clinical practice.

Health information systems in their current form have only existed for a few decades. And they are still in continuous change. Their complexity is high and often underestimated. Insofar it is understandable that in the beginning priority has been put on successful and stable implementations and on feasible solutions concerning organizational issues. However, this initial phase should now clearly be regarded as finished and so the need for systematically looking for evidence must also be demanded for the practice of health informatics – for the sake of patients, of health care organizations, and for high-quality and efficient health care.

This book on evidence-based health informatics, edited by two colleagues with high international reputations in this field, is timely and very welcome. They successfully invited excellent authors worldwide to report and to discuss about the many aspects of evidence-based health informatics.

What has to be considered when reading the book?

In their preface Elske Ammenwerth and Michael Rigby report about the book's objective: It “seeks to meet the need for better understanding of the need for robust evidence to support health IT, give insight into health IT evidence and evaluation as its primary source, and to promote health informatics as the underpinning science”.

They state that a reader should not expect a cook book with a few recipes on how to successfully cook some delightful meals of evidence-based health informatics. Editors and authors seek “to inform the reader on the wide range of knowledge available, and its necessary use according to circumstances”.

It is good to have books helping to get better evidence in the practice of informatics, in particular in the context of health information systems.

Dear reader, please select and prepare your appropriate meal. And…: Bon appetit!

Reinhold Haux

Corresponding author: Prof. Dr. Reinhold Haux, President and Honorary Fellow of IMIA, the International Medical Informatics Association, reinhold.haux@plri.de.

Peter L. Reichertz Institute for Medical Informatics, University of Braunschweig - Institute of Technology and Hannover Medical School

While the use of health IT applications has increased rapidly over past decades, this does not compare strongly with other business sectors. Both reluctance to invest in, and lack of demand to use IT systems may in part be due to lack of robust evidence as to proven benefits. At the same time, the health IT sector has lagged behind other health technology areas in working to, and being expected to work to, robust evidence standards showing benefit and also avoidance of harm. Exacerbating this, limited availability of evidence has perpetuated this misplaced comfort in use of aspiration and expectation rather than evidence in driving investment in health IT applications. Reference back to the core principles drawn from influential thinkers shows the essential centrality of the need for evidence of safety and effectiveness, and for its use relevantly related to context.

We define and discuss the nature of Evidence-based Health Informatics (EBHI), the kind of evidence health informatics researchers must generate to make EBHI a reality, and how we should grade such evidence. We propose adding principle-based evaluation studies to the list of common evaluation study types, and outline how to carry out such studies to generate evidence that will prove useful for establishing EBHI. The main purpose of a principle-based evaluation study is to test the impact on system acceptability, usage or effectiveness of a generalizable system design principle, so we also explore when during the system design process such principles are needed, and which disciplines are most promising as sources of design principles. We conclude with some challenges for EBHI, a list of the benefits of adopting this approach, and a test to ensure that we are advancing in the direction of science, as opposed to pseudoscience.

Alongside their benefits health IT applications can pose new risks to patient safety. Problems with IT have been linked to many different types of clinical errors including prescribing and administration of medications; as well as wrong-patient, wrong-site errors, and delays in procedures. There is also growing concern about the risks of data breach and cyber-security. IT-related clinical errors have their origins in processes undertaken to design, build, implement and use software systems in a broader sociotechnical context. Safety can be improved with greater standardization of clinical software and by improving the quality of processes at different points in the technology life cycle, spanning design, build, implementation and use in clinical settings. Oversight processes can be set up at a regional or national level to ensure that clinical software systems meet specific standards. Certification and regulation are two mechanisms to improve oversight. In the absence of clear standards, guidelines are useful to promote safe design and implementation practices. Processes to identify and mitigate hazards can be formalised via a safety management system. Minimizing new patient safety risks is critical to realizing the benefits of IT.

The focus of this contribution is on the theoretical principles and concepts behind evaluation of IT-based systems, discussing their presuppositions, implications and interrelationships; for instance in relation to a series of issues to consider: terminology for the concepts used as that is a reason for many disputes, bias as that is a common reason for less accuracy and trustworthiness in conclusions, culture as the tacit driver of everything we do and design, constructive evaluation as this has strict time and timing issues, preparing for meta-analyses as that is in the near future, and top-level issues in choice of methodology. Awareness in these respects will lead to avoidance of major pitfalls and perils at evaluation and thereby improve the validity and trustworthiness of an evaluation outcome, supporting the initiative towards evidence-based health informatics.

Appropriately identifying and representing stakeholders' interests and viewpoints in evaluations of health information technology (health IT) is a critical part of ensuring continued progress and innovation in eHealth. This contribution therefore seeks to clarify the principles of stakeholder analysis in an eHealth context. We describe this with reference to a mixed methods national evaluation of ePrescribing systems in English hospitals. We use this evaluation to exemplify the engagement and analytical tools required to ensure a detailed understanding of the issues, challenges and lessons learnt across stakeholder groups. We conclude that this type of approach may support the robustness of evaluations of health IT as well as their longer term impact on innovation in the field.

This contribution focuses on the heterogeneity and complexity of health information technology services and systems in a multi-stakeholder environment. We propose the perspective of process modeling as a method to break out complexity, represent heterogeneity, and provide tailored evaluation and optimization of health IT systems and services. Two case studies are presented to show how process modeling is needed to fully understand the information flow, thus identifying requirements and specifications for information system re-engineering and interoperability; detect process weaknesses thus designing corrective measures; define metrics as a mean to evaluate and ensure system quality; and optimize the use of resources.

Health Informatics frameworks have been created surrounding the implementation, optimization, adoption, use and evaluation of health information technology including electronic health record systems and medical devices. In this contribution, established health informatics frameworks are presented. Important considerations for each framework are its purpose, component parts, rigor of development, the level of testing and validation its undergone, and its limitations. In order to understand how to use a framework effectively, it's often necessary to seek additional explanation via literature, documentation, and discussions with the developers.

The quality of logic in a research design determines the value of the results and our confidence regarding the validity of the findings. The purpose of this contribution is to review the principles of research design as they apply to research and evaluation in health IT. We review the architecture of research design, the definitions of cause, sources of bias and confounds, and the importance of measurement as related to the various types of health IT questions. The goal is to provide practitioners a roadmap for making decisions for their own specific study. The contribution is organized around the Threats to Validity taxonomy and explains how different design models address these threats through the use of blocking, factorial design, control groups and time series analysis. The contribution discusses randomized experiments, and includes regression discontinuity designs and various quasi-experimental designs with a special emphasis on how to improve pre/post designs. At the end, general recommendations are provided for improving weaker designs and general research procedures.

This contribution offers an overview of the ‘third research paradigm’, its historical roots and its relevance for health informatics. Using illustrative studies, we explore the concepts of triangulation and integration of quantitative and qualitative data and refute common philosophical objections to mixing different types of knowledge. We consider how the mixed method paradigm relates to two programme design and evaluation frameworks that are important for health informatics: realist evaluation and Theory of Change. We discuss how to manage practical challenges to this approach and explain how mixed method studies support an evidence-based approach to real world policy, planning and investment decisions.

Sociotechnical approaches are grounded in theory and evidence-based. They are useful for evaluations involving health information technologies. This contribution begins with an overview of sociotechnical theory and ethnography. These theories concern interactions between technology, its use, people who use or are affected by it, and their organizational and societal situations. Then the contribution discusses planning and designing evaluations, including frameworks and models to focus an evaluation, and methodological considerations for conducting it. Next, ethical issues and further challenges and opportunities are taken up. Concluding case examples, referenced throughout, illustrate how good evaluations provide useful results to help design, implement, and use health information technologies effectively.

Usability is a critical factor in the acceptance, safe use, and success of health IT. The User-Centred Design process is widely promoted to improve usability. However, this traditional case by case approach that is rooted in the sound understanding of users' needs is not sufficient to improve technologies' usability and prevent usability-induced use-errors that may harm patients. It should be enriched with empirical evidence. This evidence is on design elements (what are the most valuable design principles, and the worst usability mistakes), and on the usability evaluation methods (which combination of methods is most suitable in which context). To achieve this evidence, several steps must be fulfilled and challenges must be overcome. Some attempts to search evidence for designing elements of health IT and for usability evaluation methods exist and are summarized. A concrete instance of evidence-based usability design principles for medication-related alerting systems is briefly described.

End user involvement and input into the design and evaluation of information systems has been recognized as being a critical success factor in the adoption of information systems. Nowhere is this need more critical than in the design of health information systems. Consistent with evidence from the general software engineering literature, the degree of user input into design of complex systems has been identified as one of the most important factors in the success or failure of complex information systems. The participatory approach goes beyond user-centered design and co-operative design approaches to include end users as more active participants in design ideas and decision making. Proponents of participatory approaches argue for greater end user participation in both design and evaluative processes. Evidence regarding the effectiveness of increased user involvement in design is explored in this contribution in the context of health IT. The contribution will discuss several approaches to including users in design and evaluation. Challenges in IT evaluation during participatory design will be described and explored along with several case studies.

Safe work processes and information systems are vital in health care. Methods for design of health IT focusing on patient safety are one of many initiatives trying to prevent adverse events. Possible patient safety hazards need to be investigated before health IT is integrated with local clinical work practice including other technology and organizational structure. Clinical simulation is ideal for proactive evaluation of new technology for clinical work practice. Clinical simulations involve real end-users as they simulate the use of technology in realistic environments performing realistic tasks. Clinical simulation study assesses effects on clinical workflow and enables identification and evaluation of patient safety hazards before implementation at a hospital. Clinical simulation also offers an opportunity to create a space in which healthcare professionals working in different locations or sectors can meet and exchange knowledge about work practices and requirement needs. This contribution will discuss benefits and challenges of using clinical simulation, and will describe how clinical simulation fits into classical usability studies, how patient safety may benefit by use of clinical simulation, and it will describe the different steps of how to conduct clinical simulation. Furthermore a case study is presented.

Economic evaluation in health care supports decision makers in prioritizing interventions and maximizing the available limited resources for social benefits. Health Information Technology (health IT) constitutes a promising strategy to improve the quality and delivery of health care. However, to determine whether the appropriate health IT solution has been selected in a specific health context, its impact on the clinical and organizational process, on costs, on user satisfaction as well as on patient outcomes, a rigorous and multidimensional evaluation analysis is necessary. Starting from the principles of evaluation introduced since the mid-1980s within the Health Technology Assessment (HTA) guidelines, this contribution provides an overview of the main challenging issues related to the complex task of performing an economic evaluation of health IT. A set of necessary key principles to deliver a proper design and implementation of a multidimensional economic evaluation study is described, focusing in particular on the classification of costs and outcomes as well as on the type of economic analysis to be performed. A case study is eventually described to show how the key principles introduced are applied.

Health Information Systems (HISs) are expected to have a positive impact on quality and efficiency of health care. Rapid investment in and diffusion of HISs has increased the importance of monitoring the adoption and impacts of them in order to learn from the initiatives, and to provide decision makers evidence on the role of HISs in improving health care. However, reliable and comparable data across initiatives in various countries are rarely available. A four-phase approach is used to compare different HIS indicator methodologies in order to move ahead in defining HIS indicators for monitoring effects of HIS on health care performance. Assessed approaches are strong on different aspects, which provide some opportunities for learning across them but also some challenges. As yet, all of the approaches do not define goals for monitoring formally. Most focus on health care structural and process indicators (HIS availability and intensity of use). However, many approaches are generic in description of HIS functionalities and context as well as their impact mechanisms on health care for HIS benchmarking. The conclusion is that, though structural and process indicators of HIS interventions are prerequisites for monitoring HIS impacts on health care outputs and outcomes, more explicit definition is needed of HIS contexts, goals, functionalities and their impact mechanisms in order to move towards common process and outcome indicators. A bottom-up-approach (participation of users) could improve development and use of context-sensitive HIS indicators.

Health IT is becoming an increasingly powerful tool for improving medication safety. While errors may happen at all stages of the medication process, different tools have been developed to support the prescribing process (e.g. computerized prescribing with decision support), the dispensing process (e.g. barcoding or automated dispensing and unit-dose systems), or the administration process (e.g. electronic medication administration records and smart pumps). Health IT can reduce medication error and preventable adverse drug event rates by increasing documentation quality and transparency, enhancing accuracy and correctness of the medication process, and supporting information exchange and interlinking different stages of the medication process. Typical evaluated endpoints comprise process-related outcomes such as number of medication errors, harm-related outcomes such as adverse drug events, or cost-related outcomes. Typical study design to measure effectiveness of health IT in medication safety comprises before-after studies and randomized controlled trials. However, implementation is challenging; it often has a major impact on the overall workflow and such technologies must be carefully introduced and their effects must be closely monitored in order to achieve the desired reductions, as in addition to preventing errors they nearly always introduce new ones. As complex interventions, their impact depends crucially on the real world setting and the implementation details and thus, transferability of study results is variable.

Information systems can only reach their full potential if their implementation is effective, and there is much to be learned as to what makes an “effective” implementation. In light of the substantial investments in Health Information Technology internationally, implementation evaluations are a powerful tool to ensure that technologies are enabled to fulfil their potential in improving care, reducing cost and increasing efficiency. The most salient characteristics of such evaluations are outlined, considering how they can help to assess adoption processes and outcomes through a continuous cycle of scientific enquiry and learning. A brief description surrounding potential theoretical lenses that may be drawn on is given. Issues discussed will be illustrated with the help of a case study on the implementation and adoption of Electronic Health Records in English hospitals. Practical challenges encountered and potential ways to address these during the conduct of health IT implementation evaluations illustrated include: 1) ways to cope with the shifting nature of reality (e.g. changes in local implementation strategies need to be reflected in the methods), 2) the need to examine processes as well as outcomes, 3) researching implementation in context with attention to both local processes and wider (e.g. political) developments, and 4) the pragmatic use of theoretical lenses where different approaches can shed light on different aspects of the implementation and adoption process.

The credibility and reliability of health IT systems as a means of achieving changes towards safer and cost-effective care have been questioned for over two decades due to the lack of methodologically strong evidence. As national level adoption and implementation of health IT are becoming widespread across the EU and globally, but are also being offset by adverse reports, the demands for evidence become more pronounced and the stakes higher. The adaptation of HTA (health technology assessment) methodology as a means to address gaps in health IT evidence production has been proposed repeatedly and tested in the field of telemedicine services. HTA has in many ways run a course parallel to that of health IT, while in certain respects attaining more clear achievements. This contribution investigates aspects of a bilaterally beneficial relation between the two disciplines using three lines of exploration: the methodological goodness-of-fit between health IT evaluation and HTA; the solutions each has proposed to the problem of producing high quality evidence in reduced amount of time; and the way each has addressed the strengthened role and position of patients. The analysis demonstrates areas of convergence between health IT and HTA. It also highlights topics which would need to be jointly addressed in the process, such as innovative and high quality data collection and analysis, inclusion of patient reported outcomes and patient safety, and transferability and generalizability of findings. In closing, it takes a glimpse of the challenges emerging as a result of the progress at the cross-roads of medicine, science and technology.

Maintaining health or managing a chronic condition involves performing and coordinating potentially new and complex tasks in the context of everyday life. Tools such as reminder apps and online health communities are being created to support patients in carrying out these tasks. Research has documented mixed effectiveness and problems with continued use of these tools, and suggests that more widespread adoption may be aided by design approaches that facilitate integration of eHealth technologies into patients' and family members' daily routines. Given the need to augment existing methods of design and implementation of eHealth tools, this contribution discusses frameworks and associated methods that engage patients and explore contexts of use in ways that can produce insights for eHealth designers.

The Internet and the digitalization of information have brought big changes in healthcare, but the arrival of smartphones and tablets represent a true revolution and a new paradigm is opened which completely changes our lives. In order to validate the impact of these new technologies in health care, it is essential to have enough clinical studies that validate their impact in wellbeing and healthcare of the patient. Traditional regulatory organisations are still looking for their role in this area. If they follow the classical path of medical devices, we get to a technical, administration and economic collapse. This contribution first presents the main indicators showing the potential of mHealth adoption. It then proposes a classification of mobile health care apps, and presents frameworks for mHealth evaluation. Regulation of mHealth as part of the evaluation process is discussed. Finally, the necessary steps and challenges that have to be taken into account by the industry to prepare the entrance of these technologies into the EU market is analysed.

This contribution examines how systematic reviews contribute to the evaluation of health IT planning and implementation. It defines and explains the systematic review process and how higher level overviews of health IT can be conducted. A reprise of some of the Cochrane reviews relating to health IT, particularly those conducted for the Cochrane Effective Practice and Organization of Care Group (EPOC), provides examples of the type of question that can be answered (at least in part) by a Cochrane-type systematic review. The contribution also discusses the benefits and limitations of the systematic review process using examples of reviews on telemedicine, nursing records, and home uterine monitoring in pregnancy.

Health information technologies are complex interventions whose effects differ across contexts. To improve our understanding of the effects of health information technologies, approaches are needed that utilize evidence beyond experimental results in order to provide explanatory answers to how and why a given technology works. The relatively new realist and meta-narrative review approaches are introduced as important methods in synthesising and analysing evidence in the field of health informatics. A common purpose of these two review approaches is to help create a sense of evidence about complex interventions that enables an understanding of how and why they work. A detailed description of the principles and objectives of the two types of reviews is presented. Key steps required to conduct each of the reviews are summarized, and examples of how the review approaches have been applied to topics related to health informatics are provided. Limitations of the two review approaches are discussed.