We live in an era of digital revolution not only in the industry, but also in the public sector. User opinion is key in e-services development. Currently the most established approaches for analyzing citizens’ opinions are surveys and personal interviews. However, governments should focus not only on developing public e-services but also on implementing modern solutions for data analysis based on machine learning and artificial intelligence. The main aim of the current study is to engage state-of-the-art natural language processing technologies to develop an analytical approach for public opinion analysis. We utilize transformer-based language models to derive valuable insights into citizens’ interests and expressed sentiments and emotions towards digitalization of educational, administrative and health public services. Our research brings empirical evidence on the practical usefulness of such methods in the government domain.