This paper studies the joint production/inventory and condition-based maintenance control for a multi-product manufacturing system with setup and maintenance times under stochastic product demands. The problem is modelled as a semi-Markov decision process (SMDP). The objective is to find a joint production and maintenance policy that minimizes the long run expected discounted cost including setup, holding, lost sales, preventive and corrective maintenance costs. A Q-learning method with state aggregation (QLA) is proposed to find near-optimal policies for large-scale problems that cannot be solved to optimality due to the curse of dimensionality. The numerical results show that QLA provides well-performing policies in a reasonable computational time.