A growing number of studies have been researching biomarkers of Parkinson’s disease (PD) using mobile technology. Many have shown high accuracy in PD classification using machine learning (ML) and voice records from the mPower study, a large database of PD patients and healthy controls. Since the dataset has unbalanced class, gender and age distribution, it is important to consider appropriate sampling when assessing classification scores. We analyse biases, such as identity confounding and implicit learning of non-disease-specific characteristics and present a sampling strategy to highlight and prevent these problems.