Monitoring of artificial intelligence (AI)-based algorithms is necessary for safe implementation and will be required in upcoming regulations. This study investigates the potential for monitoring of AI in hospitals. First, by reviewing regulatory requirements and state of the art of monitoring. Then, by conducting a gap analysis of ISO42001, containing industry agnostic requirements harmonized with the EU AI Act. The analysis illustrates the need for comprehensive monitoring capable of capturing deviations in input, performance drifts and unintended interactions. However, hospitals often suffer from a technical debt, and the gap analysis provides qualitative indications on implementation challenges, including data quality, infrastructure and limitations in continuous improvement.