With its seeming competence to mimic human responses, ChatGPT, an emerging AI-powered chatbot, has spurred great interest. This study aims to explore the role of ChatGPT in synthesizing medication literature and compare it with a hybrid summarization system. We tested ten medications’ effectiveness with reference to their definitions and descriptions extracted from DrugBank. ChatGPT could generate coherent summaries that are not backed by evidence. In contrast, our approach can provide a highly structured and concise synthesis of related evidence, but the resulting summary is not as fluent and convincing as ChatGPT. Therefore, we recommend integrating both techniques to achieve the best performance.