AI tools are being introduced within health services around the globe. It is important that tools are developed and validated using the available health information of the population where it is intended to be used. We set out to determine what patients thought about the use of their health information for this purpose. In interviews we found that the patients of a health service in Auckland, Aotearoa New Zealand, are generally comfortable with their health information being used for these purposes but with conditions (around public good, governance, privacy, security, transparency, and restrictions on commercial gain) and with careful consideration of their perspectives. We suggest that health services should take the time to have these conversations with their communities and to provide open and clear communication around these developments in their services.