To solve the problem that constructing local feature descriptors of SURF takes too much time, this paper proposed a novel method based on binary BRISK feature descriptor to improve remote sensing image matching speed. Firstly fast DoH (Determinant of Hessian) operator was applied to detect the feature points and determine the main direction. Then we used BRISK descriptor to describe the feature points, at the matching stage, we used two direction matching by Hamming distance and RANSAC algorithm to match again. Finally, the least square algorithm was applied to obtain accurate registration and complete image registration. Experimental results show that the algorithm can not only improve the matching speed significantly, but also the matching performance is beyond the original SURF algorithm, which can be used to complete the remote sensing image registration.