Recent breakthroughs in computer vision have led to the invention of several intelligent systems in different sectors. In transportation, this advancement led to the possibility of proposing autonomous vehicles. This recent technology relies heavily on wireless sensors and Deep learning. For an autonomous vehicle to navigate safely on highways, the vehicle needs equipment to aid with detecting road anomalies such as potholes ahead of time. The massive improvement in computer vision models such as Deep Convolutional Neural networks (DCNN) or vision transformers (ViT) resulted in many success stories and tremendous breakthroughs in object detection tasks; this enabled the use of such models in different application areas. But many of the reported results are theoretical and unrealistic in real-life. Usually, the nature of these models is extensive; they are trained on High-performance computers or cloud computing environments with GPUs, which challenge their usage on edge devices. However, to come up with a light model that can fit into embedded devices, the model size has to be reduced significantly so that the performance will not be affected. Therefore, this paper proposes a lightweight model of pothole detection for an embedded device. The model achieved a state-of-the-art accuracy of 98%, with the number of parameters reduced to more than 70% compared with a deep CNN model; the model can be trained and deployed on embedded devices such as smartphones efficiently.