This paper presents a speaker classification architecture for VoiceXML-based applications. Our analysis component receives user utterances from the VoiceXML platform, performs feature extraction and classifies speaker characteristics such as age, gender and emotional state. This additional information about the speaker can be employed to adapt system prompts and the dialogue strategy to specific user groups. The implementation of our prototype shows, that speaker classification is feasible without significant delays and with high accuracies of over 95% for anger detection and gender classification.