In a world where taking photos has become a daily behavior to most people, it is deemed that a photo sharing service has the strong potential to recommend desirable sightseeing places for the majority of travelers. There is much research on location recommendations and route suggestions for sightseeing based on check-in frequency to extract the popularity of a location, which is used to infer a user's interest in a place. However, the popularity of a location is not sufficient for the location recommendation, and user's preferences in a place cannot be extracted using only the check-in frequency. In this study, we propose a recommendation system for sightseeing places based on user's photo-taking behavior, where a large amount of photo data can be classified into five genres: scenery, architecture, nature, activity, and food. These data are leveraged to recommend popular sightseeing places according to a user's preference that is extracted from the user's photo-taking behavior. In the experiment, we test the classification capability of CNN (Convolutional Neural Network) in deep learning for five genres of photo data that we collected using smartphones. The proposed system could be realized by incorporating CNN as a photo classifier into our photo sharing service to recommend sightseeing places.