Abstract: Image quality assessment plays an important role in remote sensing image fusion. Research on objective evaluation methods is of great value in this field. Because the fused images will be observed by human finally, the best objective evaluation index of image quality should be consistent with the subjective evaluation. After the human visual characteristics are introduced into, image quality evaluation method can get better result. Therefore, based on the multi channels decomposition and contrast sensitivity characteristics of human visual system, this paper proposes a new method to evaluate the fusion quality of panchromatic image and multispectral image. Firstly, the color distortion and structural similarity are modified to evaluate the spectral information and spatial information respectively. Secondly, they are integrated as the overall evaluation of fused image. According to the experiment evaluating quality of different fused images by SPOT image data and GF-2 image data, it proves that the proposed method is in good agreement with the subjective evaluation when the remote sensing image is high resolution data.