In this paper, we develop market making agents that are robust to price manipulation by using adversarial reinforcement learning. We first introduce a policy parameterisation for learning continuous strategies and a set of reward functions analogous to objectives commonly used in the optimal control literature. We identify reward shaping as an effective technique when prior knowledge of the domain is available. The performance of our agents is then compared to equivalent theoretical results. We define a notion of exploitability and identify the extent to which our strategies are susceptible to price manipulation. Our adversarial training approach is then shown to naturally promote risk averse behaviour without relying on penalties on inventory nor domain-specific knowledge.