In order to solve the problems of Canny algorithm, such as large amount of computation, long time-consuming, and poor real-time performance in the application of video edge detection system, a hardware acceleration design of Canny algorithm is proposed based on FPGA. Template is used instead of convolution, and the structure of the system is improved and optimized with pipeline. Real-time video edge detection system is put forward. Analyze the original image of the camera, the processing images of Sobel and Canny. The results show that the system can efficiently deal with the edge of the complex image in real time. The system can not only save hardware resources but also improve the processing speed greatly.