Patient weights can be entered incorrectly into electronic health record (EHR) systems. These weight errors can cause significant patient harm especially in pediatrics where weight-based dosing is pervasively used. Determining weight errors through manual chart reviews is impractical in busy clinics, and current EHR alerts are rudimentary. To address these issues, we seek to develop an advanced algorithm to detect weight errors using supervised machine learning techniques. The critical first step is to collect labelled weight errors for algorithm training. In this paper, we designed and preliminarily evaluated a visual annotation tool using Agile software development to achieve the goal of supporting the rapid collection of expert-annotated weight errors. The design was based on the fact that weight errors are infrequent and medical experts can easily spot potential errors. The results show positive user feedback and prepared us for the formal user-centered evaluation as the next step.