Big data repositories contain great-value data from which actionable knowledge insights can be meaningfully derived in order to support a wide spectrum of modern applications, such as smart cities, social networks, e-science, bio-informatics, and so forth. How to extract these interesting patterns from such large-scale repositories? The latter is a fundamental research question that is still open. Inspired by the described research challenge, this paper explores the issue of supporting advanced Machine Learning (ML) structures over big data repositories, whose final goal is realizing meaningful knowledge discovery tasks. These “structures” are rather programs than tasks so that they incorporate ML procedures within high-level (program) controls whose main goal is that of magnifying the expressive power of the whole big data analytics process implemented as a collection of singleton big data analytics tasks. In turn, each task is implemented in term of a proper advanced ML structure. The paper provides introduction and motivations to the investigated problem, analysis of related work, and the proposal of a reference architecture supporting these innovative structures.