Bituminous Mixture (BM) is a granular composite material stabilized by the presence of bitumen. In continuum modeling for BM, the macroscopic response has been lacking the ability to explicitly account for the effect of the microstructure. In this study, an elasto-viscoplastic continuum model is developed to predict BM response and performance in service loading. The model incorporates a Drucker-Prager yield surface that is modified to capture the influence of the directional distribution of aggregates and damage density. The model is converted into a numerical formulation and is implemented in finite element (FE). A fully implicit algorithm in time-step control is used to enhance the efficiency in the FE analysis. The FE model is used in this study to simulate permanent deformation for isotropic and anisotropic structures.