This paper introduces Global Contextualized Representations (GCoRe) – an extension for existing transformer-based language models. GCoRe addresses limitations in capturing global context and long-range dependencies by utilizing Graph Neural Networks for graph inference on a context graph constructed from the input text. Global contextualized features, derived from the context graph, are added to the token representations from the base language model. Experiment results show that GCoRe improves the performance of the baseline model (DeBERTa v3) by 0.57% on the HotpotQA dataset and by 0.15% on the SQuAD v2 dataset. In addition, GCoRe is able to answer questions that require logical reasoning and multi-hop inference, while the baseline model fails to provide correct answers.