Exercise therapy aims to restore normal musculoskeletal function or to reduce pain in patients with musculoskeletal shoulder disorders. Especially during long-term treatments, patients perform unsupervised exercises at home. Harmful movement deviations should be identified and avoided. Assistive technologies based on physical therapists’ knowledge can support patients. In this paper, we present our approach to minimize the complexity of a physical therapists’ quality assessment of a shoulder exercise performance. Four experienced physical therapists assessed the quality of 13 exercise performances shown as video recordings with a visual analogue scale and explained their ratings as free text. In some cases, the resulting assessments differed widely depending on the respective therapeutic goals. The therapists had difficulties to reduce the performances quality to a single value. Despite the existing consensus on exercises’ descriptions, a more than one-dimensional scale is needed to evaluate exercises. Therefore, rating categories have to be identified for more detailed assessments.