By analyzing RGB-to-HSV conversion algorithms, this paper proposes a fast algorithm for enhancing brightness and contrast based on HSV. Unlike traditional algorithm that convert RGB to HSV, enhance the brightness V, and then convert back to RGB, our approach allows for enhancing brightness and contrast directly in HSV without any conversion to RGB. Avoiding using the floating point multiplication and division, it used integer operations, shift operator and look-up table methods in this algorithm, so that improved the procession suggest admit the look-up table methods. Finally, the experiments show that the fast algorithm can save 70% of the computing time than the traditional algorithm. It can process the 720p video images real time.