Recently, deep learning has been studied as one of the most effective methods in the machine learning field, and lots of results have been reported. However, the most effective way to construct neural networks has not yet been determined. Besides, interpretation of an obtained network is difficult. In a previous study, we proposed a novel method to construct a neural network using a support vector machine called SVM-NN. However, there is also a problem in that, for hard to apply to a nonlinear problem and model size problem. In this study, we first propose a new network structure called AND/OR layers to solve the nonlinear problem of SVM-NN. AND/OR layers improve the effectiveness of identification by grouping support vectors based on training data results. This type of novel SVM-NN is called SVM-NN(AND/OR). We also utilize the genetic algorithm to pair and reduce support vectors to compress the model size of SVM-NN and SVM-NN(AND/OR). To confirm the effectiveness of the proposed methods, the computational experiments were carried out taking typical benchmark problems as examples. The effectiveness of the proposed method is confirmed by computer simulations.