We introduce a method for image segmentation that constraints the clustering with map and point data. The method is showcased by applying the spectral clustering algorithm on aerial images for building detection with constraints built from a height map and address point data. We automatically detect the number of clusters using the elongated K-means algorithm instead of using the standard spectral clustering approach of a predefined number of clusters. The results are refined by filtering out noise with a binary morphological operator. Point data is used for semi-supervised labelling of building clusters. Our evaluation show that the combination of constraints have a positive impact on the clustering quality achieved. Finally we argue how the presented constraint types may be used in other applications.