The massively parallel architecture of GPUs, coming from its graphics heritage, is now delivering transformative results for scientists and researchers all over the world. For some of the world's most challenging problems in medical research, drug discovery, weather modeling, and seismic exploration – computation is the ultimate tool. Without it, research would still be confined to trial and error-based physical experiments and observation. Parallel computing: use of multiple computers or processors working together on a common task. Each processor works on its section of the problem. Processors can exchange information. In this we discuss about the parallel architecture, parallel programming models, MPI and OpenMP and CUDA programming.