The covariance-driven stochastic subspace modal parameter identification method has been widely used in the field of engineering structures. Effective determination of the model order of the structural system is the key to applying this method to identify the modal parameters. It is particularly difficult to determine the model order for unstable systems affected by noise disturbances and computational errors. In order to effectively determine the model order, an exponential eigenvalue entropy incremental covariance-driven stochastic subspace identification (EE-COV-SSI) algorithm is proposed. The condition number of the state matrix is used to determine the degree of perturbation of the response signal to the system stability. Meanwhile, the identification accuracy of the modal parameters is reflected by calculating the modal frequency coefficient of variation. Finally, the method is applied to the modal analysis of a four-story frame structure. The results show that the method can accurately identify the model order and improve the identification accuracy of the modal parameters.