Video annotation technique delivers many additional video processing capabilities for several applications. Sports broadcast video content is unique in regard to wealth of information as compared to any other video. Sports video annotation is becoming popular among researchers in recent times because of wide range of applications and challenges it pose. The demand for optimized design of framework for sports video annotation is at peak. This paper surveys state-of-the-art in annotation framework design, particularly for sports applications and provides insight into future aspects. This survey may help researchers to further conceive and develop advanced universal frameworks applied to all sports.