Through technology, it is essential to seamlessly bridge the divide between diverse speaking communities (including the signer (the sign language speaker) community). In order to realize communication that successfully conveys emotions, it is necessary to recognize not only verbal information but also non-verbal information. In the case of signers, there are two main types of behavior: verbal behavior and emotional behavior. This paper presents a sign language recognition method by similarity measure with emotional expression specific to signers. We focus on recognizing the sign language conveying verbal information itself and on recognizing emotional expression. Our method recognizes sign language by time-series similarity measure on a small amount of model data, and at the same time, recognizes emotion expression specific to signers. Our method extracts time-series features of the body, arms, and hands from sign language videos and recognizes them by measuring the similarity of the time-series features. In addition, it recognizes the emotional expressions specific to signers from the time-series features of their faces.