Web forums are a frequent way of sharing useful information among people. They are becoming the main source of up-to-date information and marketplaces pertaining to different domains, including criminal content and zero-day security exploits. Analyzing the web forums of the existing discussion threads is an alternative method to understand the exploits and fraud modalities a law breaker will most likely make use and how to defend against them. However, in many cases, it is hard to capture all the relevant context of the forums which is needed for classification. In this paper, we introduce a data-driven technique to mine the web forums and provide policy recommendations to the defender. A neural network (NN) is used to learn the set of features for forum classification. Furthermore, we present the evaluation and results from employing our method, with various system configurations, on real-world datasets collected form the web.