In today’s world, the security of every individual has become an important aspect. There is a need for constant monitoring in public places. A Manual operating camera system is an unreliable and very basic and poor method for this purpose. Intelligent Video Surveillance is an approach where multiple CCTVs constantly record the scenes and proper algorithms are deployed in order to detect and monitor activities. Deep Learning frameworks and algorithms like Kera’s, YOLO, Convolutional Neural Networks or backbones for image detection like VGG16, Mobile net, Resnet101 have been used for human and weapon detection. The paper focuses on deep learning techniques and threading to collectively develop a Parallel Deep Learning Framework for Video Surveillance that aims at striking the right balance between accuracy and system performance or stability. Threading is used in terms of implementation of a uniquely proposed Dynamic Selection Algorithm that uses two backbones for object detection and switches between them based on the queue status for achieving system stability. A uniquely designed logistic regression filter is also implemented that boosts the system performance.