A computing paradigm is currently at crucial point in its evolution: distributed computing (DC), marked by the increasing developments of large scale distributed networks (LSDNs). In DC, every application or service is split up into tasks that run simultaneously on multiple computers communicating over an LSDN. Hence, DC is a form of parallel computing in heterogeneity, decentralization, nondeterminism and dynamicity of the network. The overarching goal of DC is to support such LSDNs capable of management and high performance. Meeting this grand challenge of DC requires a fundamental approach to the aspects of tasks and data parallel not tackled before. To this end, taking advantage of the categorical structures we establish, in this chapter, a firm formal basis for specifying tasks parallel, data parallel, peer-to-peer structures and self-organization in LSDNs. All of these are to formalize parallel programming in LSDNs.