The principle of using stereoscopic displays is to present the viewer with an accurate perception of 3D space. Stereopsis is a powerful binocular cue that will supplement any monocular information in the scene. Our work with an optical augmented reality system has highlighted one scenario where an accurate sense of depth cannot be easily achieved from stereoscopic images. In our augmented reality system we use stereo images of anatomical structures overlaid on the patient for surgical guidance. It is essential that the surgeon can accurately localize the images during surgery. When the stereo images are presented behind the transparent physical surface the perception of the depth of the images can become unstable and ambiguous, despite good system calibration, registration and tracking. This paper reviews possible reasons for the failure in accurate depth perception and presents some ideas on how this might be corrected for in an optical augmented reality system.