Large scale benchmarking of web accessibility can benefit from human input to complement results produced by automatic evaluation tools. This paper presents a novel method that enables non-experts to provide input on web accessibility. The semi-automatic approach guides the evaluators through a structured process with clear instructions. We present a template to describe different types of user input and an outline of the empirical validation.