Continuous timely repair and replacement of infrastructures, equipment and utilities play an important role in maintaining the smooth-running of a city or local community. Thereby, to help individuals and businesses go about their daily activities with ease, it is vital to develop a proper method for automatically identifying and assigning capable workers for tasks. This paper defines the community management service task allocation problem as CMS-TAP and hence an end-to-end “recommendation + allocation” network, i.e. a task recommender and allocation optimization network (denoted as TROpt-NET), is then developed for handling such problem. TROpt-NET consists of two layers, namely one for predicting worker ability and the other for allocating tasks which are TR Layer and TA Layer, corresponding to “recommendation” and “allocation” of tasks. Different from operations research approaches where workers are assigned to jobs based on their pre-labelled skills and fixed locations, we propose a task recommender and allocation optimization network. The TR layer is a task recommender system designed to learn implicit worker abilities for different tasks using Neural Collaborative Filtering (NCF) by mining a historical dataset of worker task completion. Whereas in the TA layer a differential optimization approach for allocation is used because of its differentiable property and ability to allow for backpropagation to the prediction layer. In this study, we first formulate the CMS-TAP problem as a recommendation +optimization problem and then propose and end-to-end network architecture that tackles the problem in a real-world setting. TROpt-NET curbs uncertainty and assumptions in optimization by learning to more accurately approximate worker ability across different tasks. Additionally, the network can learn implicit worker abilities enabling optimal utilization of workers across a wide range of tasks, which is often ignored in task allocation problems. We find that normalizing worker ability across all tasks improves the implicit learning capability of the network and that good approximations don’t always lead to optimal allocation but learning allocations by backpropagating through recommendations improves the allocation objective. Offline experiments on a real-world large-scale dataset demonstrate the effectiveness of our proposed TROpt-NET.