ReLU nodes are utilized commonly in neural networks as they look and act like linear functions while providing nonlinearity. In spite of addressing the vanishing gradient problem, they can lead to the dying ReLU problem which can be detrimental in terms of convergence and generalization performance. This paper proposes antimatter networks, a new and simple solution to the dying ReLU problem which involves combining ReLU nodes with their inverse, negative ReLU nodes, together with activation swapping mechanisms. We tested the solution on six separate dataset-architecture combinations with the MNIST, CIFAR-10, and Flowers-16 datasets for Convolutional Neural Networks (CNN) and Multi-Layer Perceptron networks (MLP) and found that antimatter networks lead to consistent convergence and generalization improvements compared to networks solely consisting of ReLU or NReLU nodes.