The unmanned surface vehicle (USV) has been widely used to accomplish tasks that cannot be completed by ships with human drivers on certain sea areas. It is not only necessary but essential to obtain a robust strategy in order to ensure multiple USVs accomplish collaborative tasks successfully and efficiently. To meet the challenge, a deep reinforcement learning method is proposed, which is combined with an improved A star algorithm. A statistically promising collaborative strategy is achieved by the proposed method under the guidance from the unmanned aerial vehicles (UAVs). After the collaborative strategy is generated, the improved A star algorithm is used to navigate the USVs. To verify the proposed algorithm, several tasks are tested on a simulation platform. Experimental results demonstrate that the proposed method outperforms state-of-the-art reinforcement learning methods such as DQN and DeepSarsa.