Over the past few years, Keyword Spotting (KWS) has emerged as a popular area of research. Although numerous open-source KWS datasets have been recently released, there is a general lack of realism in benchmarking the false alarm rate (FAR) in real environments. This can produce models that achieve great accuracies but are not able to work on real-world conditions due to a high number of false triggers. In this work, we demonstrate that two recent KWS models report state-of-the-art accuracies on Google Speech Command dataset but suffer from high false alarm rates in presence of noisy environments. To this end, we propose an extensive benchmark dataset comprising various real-world noises and sounds to evaluate specifically the FAR across different acoustic environments.