Advances in communication technologies have contributed to the proliferation of distributed datasets. The most effective approach to distributed learning is to learn locally and then combine the local models. In general, distributed algorithms assume that there is a single model that could be induced from the distributed datasets. Under this view, distribution is treated exclusively as a technical issue. However, real-world distributed datasets frequently present an intrinsic data skewness among their partitions. Despite of its importance, up to the authors’ knowledge, its impact has been barely investigated in the literature. In this paper, the performance of different cluster-based distributed learning methods is analyzed over distinct scenarios by incrementing the differences in the probabilistic distribution of data among partitions. Based on these results the best approach is suggested at every scenario.