Abstract: High-resolution remote sensing images reveal dissimilar distinguishable features at different scales. Based on this characteristic of high-resolution remote sensing images, a new method of extracting buildings based on multilevel feature in aerial images was developed by associating scales with features. In the case of a large-scale feature, histograms of oriented gradient (HOG), were used to recognize rough buildings areas. The areas maybe include the grass, roads and some other non-building information. In order to remove these non-building surface features, fused spectral and texture features (T-B) is proposed at the small scale. The T-B feature is used to process the results of the first recognized HOG results. Experimental results demonstrate that the new feature has a good effect. Edge information of buildings was obtained and the results show that our proposed algorithm can extract both rectangular and complex buildings in different area remote sensing images well.