Indian Sign Language is one of the most important and widely used forms of communication for people with speaking and hearing impairments. Many people or communities have attempted to create systems that read the sign language symbols and convert the same to text, but text or audio to sign language is still infrequent. This project mainly focuses on developing a translating system consisting of many modules that take English audio and convert the input to English text, which is further parsed to structure grammar representation on which grammar rules of Indian Sign Language are applied. Stop words are removed from the reordered sentence. Since the Indian Sign Language does not support conjugation in words, stemming and lemmatization will transform the provided word into its root or original word. Then all the individual words are checked in a dictionary holding videos of each word. If the system does not find words in the dictionary, then the most suitable synonym replaces them. The system proposed by us is inventive as the current systems are bound to direct conversion of words into Indian Sign Language on-the-other-hand our system aims to convert the sentences in Indian Sign Language grammar and effectively display it to the user.