In recent years, surgical simulation has emerged at the forefront of new technologies for improving the education and training of surgical residents. To objectively evaluate the surgical skills of the trainees and reduce the training cost, an automated method for rating the performance of the operator is critical. However, automated evaluation of surgical skills in a video-based system, e.g., the FLS trainer box, is still a challenging task, both due to the lack of reliable visual features and the lack of analysis tools that bridge the semantic gap between the low-level visual features and the high-level surgical skills. This study attempts to find a latent space for the visual features for supporting more meaningful analysis of surgical skills. The approach employs multi-modality fusion and Canonical Correlation Analysis as the key techniques. Experiments were designed to evaluate the proposed approach. The results suggest that this is a promising direction.