Deep neural networks often tend to memorize data and can risk information leakage if trained on private data. There has been some attempts to train deep neural networks by contaminating the gradients during backpropagation. Over the years, this has become one of the most prominent techniques to make neural networks differentially private. One downside of this method is that contaminated gradients lead to suboptimal solutions during backpropagation. In this paper, we make an attempt to diminish the contamination effect by proposing a bottlenecked backpropagation technique. The proposed bottlenecked backpropagation technique follows Reny differential privacy, a recently developed more optimized version in the realm of differential privacy. On the other hand, the bottlenecked backpropagation considers the direction and neglects the magnitude of the gradient vectors. The idea is built on top of signed stochastic gradient descent, another recent advancement in the optimization methods for deep learning. By prioritizing gradient direction over magnitude, it minimizes noise impact on model convergence. Experimental results on benchmarks including MNIST, FMNIST, and IMDB datasets demonstrate substantial improvements over the state-of-the-art methods, achieving faster convergence and higher model accuracy, striking a promising balance between privacy and performance. Furthermore, we observe the proposed methods to be resilient against membership inference attacks.