Multi-Agent Learning is a complex problem, especially in real-time systems. We address this problem by introducing Argumentation Accelerated Reinforcement Learning (AARL), which provides a methodology for defining heuristics, represented by arguments, and incorporates these heuristics into Reinforcement Learning (RL) by using reward shaping. We define AARL via argumentation and prove that it can coordinate independent cooperative agents that have a shared goal but need to perform different actions. We test AARL empirically in a popular RL testbed, RoboCup Takeaway, and show that it significantly improves upon standard RL.