The dimension of particles can significantly influence the load response and the performance of uniformly graded ballast layers in railway track structures in real-world conditions. Yet, the micromechanical behavior of the unbound aggregate ballast layer assembly, particularly affected by particle size, remains largely unexplored. In this study, the distinct element model of a direct shear test was initially simulated using friction coefficients of 0.8, 0.9, and 1, and particle size distribution No.4A. The dimensions of the shear test box were 300 mm in width, 300 mm in length, and 180 mm in height. Additionally, a constant normal force of 333 kPa was applied to the sample during the simulation. Subsequently, this model was compared with experimental results, revealing a close correspondence between simulated and experimental shear stress-displacement curves, particularly for the friction coefficient of 1. Following this verification, the validated model was employed to investigate three other particle size distributions: No.4, No.5, and No.57. The results demonstrated a reduction in shear stress for particle size distributions No.4, No.5, and No.57 compared to No.4A, with quantified decreases of 11.9%, 38.2%, and 56.7%, respectively.