Visual Traffic Surveillance has become a significant framework in vision computing with the advent of intelligent transport system in recent years. The Automatic Vehicle Surveillance system is gaining growing attention to handle traffic congestion in urban roadways. The primary task of such system is to detect any moving object in the video sequence, robustly track and finally classify into different categories. Motion Segmentation plays an important role among the methods of vehicle detection and are briefly discussed in this study. After detection, the surveillance system should be capable of tracking and classifying the vehicles. There are several conventional tracking techniques for vehicle tracking which are explained with its related researches, advantages and limitations in this study. The surveillance becomes more challenging due to the presence of occlusion, variance in illumination, unusual orientation of vehicles etc. Vehicle detection and tracking has an emerging application in real time passenger management system, security issues and accident prevention in highways. To prevent accidents, abnormal behaviour of the vehicles should be detected in advance. For the achievement of the unusual behaviour detection, a comprehensive analysis of different methods of anomaly detection are highlighted in this survey. The challenges, future scope with notable application domains of the system are located and a dominant bibliography is also included.