Retrieval Augmented Generation has been shown to improve the output of large language models (LLMs) by providing context to the question or scenario posed to the model. We have tried a series of experiments to understand how best to improve the performance of the native models. We present the results of each of several experiments. These can serve as lessons learned for scientists looking to improve the performance of large language models for medical question answering tasks.