Mumford-Shah model has been used for image segmentation by considering both homogeneity and the shape of the segments jointly. It has been previously optimized by complex mathematical optimization methods like Douglas-Rachford, and a faster but sub-optimal k-means. However, they both suffer from fragmentation caused by non-convex segments. In this paper, we present hierarchical algorithm called Pairwise nearest neighbor (PNN) to optimize the Mumford-Shah model. The merge-based strategy utilizing the connectivity of the pixels prevents isolated fragments to be formed, and in this way, reaches better quality in case of images containing complex shapes.