In order to improve the problem that the traditional target detection and tracking method is not rich enough in extracting high-level semantic information of features, which leads to large error in target tracking results, a small infrared target tracking method based on Kalman filter in complex background is proposed. The method comprises the following steps of preprocessing an acquired infrared image of a dim target, enhancing the contrast of the target, and inhibiting the influence of a background on the target; extracting the characteristics of the infrared dim target through a YOLOv4 network, dividing pixels with the same visual characteristics into a set, searching a suspicious target in the set, segmenting a real target in the suspicious target, and detecting the infrared dim target; The Kalman filter algorithm is used to track the trajectory of infrared dim and small targets, and a suitable loss function is introduced to improve the target tracking ability. The results show that the center error of this method is always smaller than other contrast methods, the minimum is 0.53, and the overlap rate is always higher than other contrast methods, the maximum is 1.0. This method is superior to the comparison method, and improves the tracking effect of small target in infrared image, and has better application performance.