In the healthcare sector in particular, the shortage of skilled workers is a major problem that will become even more acute in the future as a result of demographic change. One way to counteract this trend is to use intelligent systems to reduce the workload of healthcare professionals. AI-based clinical decision support systems (AICDSS) have already proven their worth in this area, while simultaneously improving medical care. More recently, AICDSS have also been characterized by their ability to leverage the increasing availability of clinical data to assist healthcare professionals and patients in a variety of situations based on structured and unstructured data. However, the need to access large amounts of data while adhering to strict privacy regulations and the dependence on user adoption have highlighted the need to further adapt the implementation of AICDSS to integrate with existing healthcare routines. A subproject of the ViKI pro research project investigates how AICDSS can be successfully integrated into professional care planning practice using a user-centered design thinking approach. This paper presents the design of the ViKI pro AICDSS and the challenges related to privacy, user acceptance, and the data base. It also describes the development of an AI-based cloud technology for data processing and exchange using federated learning, and the development of an explicable AI algorithm for recommending care interventions. The core of the AICDSS is a human-in-the-loop system for data validation, in which the output of the AI model is continuously verified by skilled personnel to ensure continuous improvement in accuracy and transparent interaction between AI and humans.