In previous work, a mobile application for medication self-management (eMMA) was introduced. It contained a basic conversational user interface (CUI). In this work, we extended the CUI by integrating the chatbot framework RiveScript and an instruction interface. To study task success, dialog quality and efficiency, we performed a theoretical and a quantitative evaluation as well as a usability test. The results show that the technical extensions of eMMA were useful to improve the chatbot's quality. However, the underlying knowledge base still requires substantial extensions before the system can be used in practice.