Collections of visual symbols are context-specific, and include symbols and signs for example at airports, in hotels, and in traffic. They symbolize our living environments and guide us in different situations. We introduce a pattern recognition system, based on bag-of-features descriptors and support vector machine classifier, for recognizing semantic elements from pictorial symbols. We evaluate our approach by extracting semantic tags from Finnish and Japanese road signs, and by translating their meaning. Our paper demonstrates the idea of a visual dictionary that anyone from any culture can understand in any context of life.