In order to improve the ability of anomaly traffic detection in industrial control system, a hybrid algorithm combining isolated forest (iForest) and one-class support vector machine (OCSVM) is designed. The isolated forest algorithm is used to detect outliers in the training data, and the outliers are eliminated to reduce their impact on the one-class support vector machine decision function.The OCSVM model is trained based on normal data, and the detection rate of the anomaly detection model is further improved by combining feature selection and parameter optimization. The experimental results show that the detection rate of the algorithm model is improved to 92. 51% on the gas pipeline data set, especially the recall rate and precision rate of abnormal behavior are improved, which optimizes the performance of the anomaly detection model and meets the reliability requirements.