During the West African Ebola virus disease outbreak in 2014–15, health agencies had severe challenges with case notification and contact tracing. To overcome these, we developed the Surveillance, Outbreak Response Management and Analysis System (SORMAS). The objective of this study was to measure perceived quality of SORMAS and its change over time. We ran a 4-week-pilot and 8-week-implementation of SORMAS among hospital informants in Kano state, Nigeria in 2015 and 2018 respectively. We carried out surveys after the pilot and implementation asking about usefulness and acceptability. We calculated the proportions of users per answer together with their 95% confidence intervals (CI) and compared whether the 2015 response distributions differed from those from 2018. Total of 31 and 74 hospital informants participated in the survey in 2015 and 2018, respectively. In 2018, 94% (CI: 89–100%) of users indicated that the tool was useful, 92% (CI: 86–98%) would recommend SORMAS to colleagues and 18% (CI: 10–28%) had login difficulties. In 2015, the proportions were 74% (CI: 59–90%), 90% (CI: 80–100%), and 87% (CI: 75–99%) respectively. Results indicate high usefulness and acceptability of SORMAS. We recommend mHealth tools to be evaluated to allow repeated measurements and comparisons between different versions and users.