In order to improve the tracking accuracy, a lightweight video tracking algorithm for fuzzy video is proposed. The mainstream countermeasure generation network model DeblurGAN V2 is used for deblurring, and the improved SiameseRPN target tracking algorithm is used for target tracking. In order to achieve lightweight, the feature network of target tracking is replaced by EfficientNet network, and the attention mechanism is improved to ECANet to capture multi-channel information, which is tested on GoPro data set. The test results show that compared with the Siamese RPN algorithm, the proposed algorithm can achieve higher tracking accuracy, and the frame rate can meet the real-time requirements, which has certain reference significance.