The technology is growing and increasing in our day to day life to satisfy the needs of human beings. The system we are going to propose makes the human job easier. Here the YOLO algorithm which is a deep learning object detection architecture is used to detect the number plate of the vehicle. After detecting the number plate it converts the vehicle number to a text format. Then it checks it with the database to see if the vehicle is authorized to enter into the premise or not. This system can be implemented in highly restrained areas such as military areas, government organizations, Parliament, etc. This proposed system has around six stages: Capture Image, Search for black pixels, Image filtering, Plate region extraction, character extraction, OCR for character recognition. The alphanumeric characters are identified using the OCR algorithm. It is then used to compare the obtained result from the YOLO algorithm with the database and then check if the vehicle is allowed to enter the premise or not. This proposed system is simulated and implemented using Python, and it was also tested on real-time images for performance purposes.