Most automated hate speech detection models rely on human annotations for training and evaluation. Logic and research indicate that people who belong to groups targeted by hate speech are better at identifying it, often due to their increased familiarity with the topic and associated hate speech terminology. However, most hate speech annotation practices overlook this issue, and hence the labels produced tend to have a reduced accuracy. In this paper, we describe an approach where the text to be annotated is supplemented with background semantics, to expose the meaning of hate speech terminology that is less likely to be known to general annotators. We test the impact of this approach by measuring change in inter-annotator agreement, before and after introducing semantics, between two groups of annotators; those who belong to the target group of hate speech, and those who are not. Our experiments show that infusing text with semantic background increases inter-annotator agreement by up to 11.3% on average, aligning the annotations from annotators who do not belong to the target groups with those from the target groups.