For the problems of traditional research methods of unmanned helicopter engine, a new modeling method of piston aircraft engine system was proposed. Combining with the characteristics of rotor system, blade element theory is applied to analyze rotor blades. Then engine power characteristics model and rotor load model were established. According to the dynamics of engine, engine speed is calculated. Finally, Matlab simulation and ground test stand testing were carried out to verify the control performance. Simulation results show that the model is reasonable and feasible and meets the control requirements of unmanned helicopter simulation.