Radiology reports can potentially be used to detect critical cases that need immediate attention from physicians. We focus on detecting Brain Hemorrhage from Computed Tomography (CT) reports. We train a deep learning classifier and observe the effect of using different pre-trained word representations along with domain-specific fine-tuning. We have several contributions. Firstly, we report the results of a large-scale classification model for brain hemorrhage detection from Turkish radiology reports. Second, we show the effect of fine-tuning pre-trained language models using domain-specific data on the performance. We conclude that deep learning models can be used for detecting brain Hemorrhage with reasonable accuracy and fine-tuning language models using domain-specific data to improve classification performance.