A Method Based on Depth Features for Matching Thermal Infrared Images with Visible Images

CUI Zhixiang; LAN Chaozhen; ZHANG Yongxian; HOU Huitai; QIN Jianqi

doi:10.13203/j.whugis20200181

Volume 48 Issue 2

Feb. 2023

Turn off MathJax

Article Contents

Abstract

References

Geomatics and Information Science of Wuhan University > 2023 > 48(2): 316-324. > DOI: 10.13203/j.whugis20200181

CUI Zhixiang, LAN Chaozhen, ZHANG Yongxian, HOU Huitai, QIN Jianqi. A Method Based on Depth Features for Matching Thermal Infrared Images with Visible Images[J]. Geomatics and Information Science of Wuhan University, 2023, 48(2): 316-324. DOI: 10.13203/j.whugis20200181

Citation:

PDF (56626 KB)

A Method Based on Depth Features for Matching Thermal Infrared Images with Visible Images

1.
Troops 31682, Lanzhou 730020, China
2.
Institute of Geospatial Information, Information Engineering University, Zhengzhou 450001, China
3.
Troops 63611, Kuerle 841000, China

More Information

Received Date: April 13, 2021
Available Online: February 16, 2023
Published Date: February 04, 2023

Graphical Abstract

Abstract

Abstract

Objectives Aiming at the matching problem of unmanned aerial vehicle(UAV) thermal infrared images and optical satellite images, a deep local feature matching method based on heterogeneous landmark dataset for learning is proposed.
Methods Firstly, the gray distribution law of thermal infrared images and visible images is learned by the generative adversarial network, and the landmark dataset consisting of thermal infrared images for feature extraction model training is synthesized. Secondly, the deep invariant features are learned from the multi-modal landmark dataset by the residual network and attention mechanism. Finally, correct matching points of image pairs are obtained by matching and purifying the invariant features.
Results The performance of this method was tested experimentally and compared with KAZE, detect-and-describe network and deep local features. The results show that the adaptability of this method to the grayscale, texture, overlap rate and geometric variations is stronger, and the matching efficiency of this method is higher.
Conclusions The effectiveness of this method is proved through multiple sets of experiments. Therefore, the UAV visual navigation is provided support for.
- heterogeneous landmark dataset,
- deep local feature,
- match,
- thermal infrared,
- visible

FullText(HTML)

References (28)

References

[1]	万文辉, 李宇, 胡文敏, 等. 基于联邦滤波进行立体相机/IMU/里程计运动平台组合导航定位[J]. 武汉大学学报(信息科学版), 2018, 43(1): 101-106. doi: 10.13203/j.whugis20150286 Wan Wenhui, Li Yu, Hu Wenmin, et al. Mobile Platform Localization by Integration of Stereo Cameras, IMU and Wheel Qdometer Based on Federated Filter[J]. Geomatics and Information Science of Wuhan University, 2018, 43(1): 101-106. doi: 10.13203/j.whugis20150286
[2]	Goforth H, Lucey S. GPS-Denied UAV Localization Using Pre-existing Satellite Imagery[C]// International Conference on Robotics and Automation (ICRA), Montreal, Canada, 2019.
[3]	Yol A, Delabarre B, Dame A, et al. Vision-Based Absolute Localization for Unmanned Aerial Vehicles[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, USA, 2014.
[4]	Shan M, Wang F, Lin F, et al. Google Map Aided Visual Navigation for UAVs in GPS-Denied Environment[C]//IEEE International Conference on Robotics and Biomimetics (ROBIO), Zhuhai, China, 2015.
[5]	叶沅鑫, 单杰, 熊金鑫, 等. 一种结合SIFT和边缘信息的多源遥感影像匹配方法[J]. 武汉大学学报(信息科学版), 2013, 38(10): 1148-1151. http://ch.whu.edu.cn/article/id/2771 Ye Yuanxin, Shan Jie, Xiong Jinxin, et al. A Matching Method Combining SIFT and Edge Information for Multi-source Remote Sensing Images[J]. Geomatics and Information Science of Wuhan University, 2013, 38(10): 1148-1151. http://ch.whu.edu.cn/article/id/2771
[6]	Dou J F, Qin Q, Tu Z M, et al. Infrared and Visible Image Registration Based on SIFT and Sparse Representation[C]// Chinese Control and Decision Conference (CCDC), Yinchuan, China, 2016.
[7]	Liu H, Xiao G F. Remote Sensing Image Registration Based on Improved KAZE and BRIEF Descriptor[EB/OL]. [2020-07-13]. https://link.springer.com/article/10.1007/s11633-019-1218-3.
[8]	吕倩利, 邵永社. 基于SIFT特征的异源遥感影像匹配方法研究[J]. 计算机工程与应用, 2012, 48(36): 171-176. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201236039.htm Lü Qianli, Shao Yongshe. Research on Matching Algorithm for Multi-source Remote Sensing Images Based on SIFT Features[J]. Computer Engineering and Applications, 2012, 48(36): 171-176. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201236039.htm
[9]	Huang Q Q, Yang J, Wang C Y, et al. Improved Registration Method for Infrared and Visible Remote Sensing Image Using NSCT and SIFT[C]// IEEE International Geoscience and Remote Sensing Symposium, Munich, Germany, 2012.
[10]	Hrkac T, Kalafatic Z, Krapac J. Infrared-Visual Image Registration Based on Corners and Hausdorff Distance[M]. Berlin, Germeny: Springer, 2007.
[11]	冯晓磊, 吴炜, 李智, 等. 利用梯度方向的Hausdorff距离配准红外和可见光图像[J]. 电视技术, 2015, 39(3): 5-10. https://www.cnki.com.cn/Article/CJFDTOTAL-DSSS201503002.htm Feng Xiaolei, Wu Wei, Li Zhi, et al. Hausdorff Distance Using Gradient Orientation Registers Infrared and Visible Image[J]. Video Engineering, 2015, 39(3): 5-10. https://www.cnki.com.cn/Article/CJFDTOTAL-DSSS201503002.htm
[12]	纪利娥, 杨风暴, 王志社, 等. 基于边缘图像和SURF特征的可见光与红外图像的匹配算法[J]. 红外技术, 2012, 34(11): 629-635. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS201211003.htm Ji Li'e, Yang Fengbao, Wang Zhishe, et al. Visible and Infrared Image Matching Algorithm Based on Edge Image and SURF Features[J]. Infrared Technology, 2012, 34(11): 629-635. https://www.cnki.com.cn/Article/CJFDTOTAL-HWJS201211003.htm
[13]	Yahyanejad S, Rinner B. A Fast and Mobile System for Registration of Low-Altitude Visual and Thermal Aerial Images Using Multiple Small-Scale UAVs[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 104: 189-202.
[14]	刘畅, 崔桐, 贺成龙, 等. 基于高曲率特征点匹配的红外可见光图像配准[J]. 指挥信息系统与技术, 2016, 7(1): 13-17. https://www.cnki.com.cn/Article/CJFDTOTAL-ZHXT201601004.htm Liu Chang, Cui Tong, He Chenglong, et al. Infrared-Visual Image Registration Based on High-Curvature Feature Matching[J]. Command Information System and Technology, 2016, 7(1): 13-17. https://www.cnki.com.cn/Article/CJFDTOTAL-ZHXT201601004.htm
[15]	Ye Y X, Shan J, Bruzzone L, et al. Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5): 2941-2958.
[16]	Ye Y X, Bruzzone L, Shan J, et al. Fast and Robust Matching for Multimodal Remote Sensing Image Registration[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(11): 9059-9070.
[17]	Yi K M, Trulls E, Lepetit V, et al. LIFT: Learned Invariant Feature Transform[C]//The 14th European Conference on Computer Vision, Amsterdam, Netherlands, 2016.
[18]	Dusmanu M, Rocco I, Pajdla T, et al. D2-Net: A Trainable CNN for Joint Description and Detection of Local Features[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019.
[19]	南轲, 齐华, 叶沅鑫. 深度卷积特征表达的多模态遥感影像模板匹配方法[J]. 测绘学报, 2019, 48(6): 727-736. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201906008.htm Nan Ke, Qi Hua, Ye Yuanxin. A Template Matching Method of Multimodal Remote Sensing Images Based on Deep Convolutional Feature Representation[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 727-736. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201906008.htm
[20]	Noh H, Araujo A, Sim J, et al. Large-Scale Image Retrieval with Attentive Deep Local Features[C]// IEEE International Conference on Computer Vision, Venice, Italy, 2017.
[21]	Revaud J, Weinzaepfel P, Souza C, et al. R2D2: Repeatable and Reliable Detector and Descriptor[EB/OL]. [2020-06-10]. https://arxiv.org/abs/1906.06195.
[22]	He K M, Zhang X Y, Ren S Q, et al. Deep Residual Learning for Image Recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016.
[23]	Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Networks[EB/OL]. [2020-03-23]. https://arxiv.org/abs/1406.2661.
[24]	Isola P, Zhu J Y, Zhou T H, et al. Image-to-Image Translation with Conditional Adversarial Networks[C]// IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017.
[25]	Zhu J Y, Zhang R, Pathak D, et al. Toward Multimodal Image-to-Image Translation[C]// The 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017.
[26]	Zhu J Y, Park T, Isola P, et al. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[C]//IEEE International Conference on Computer Vision, Venice, Italy, 2017.
[27]	Zhang L C, Gonzalez-Garcia A, Weijer J, et al. Synthetic Data Generation for End-to-End Thermal Infrared Tracking[J]. IEEE Transactions on Image Processing, 2019, 28(4): 1837-1850.
[28]	Ozaki K, Yokoo S. Large-Scale Landmark Retrieval/Recognition Under a Noisy and Diverse Dataset[EB/OL]. [2020-10-20]. https://arxiv.org/abs/1906.04087.

[1]	Gan Wenxia, Pan Junjie, Geng Jing, Wang Huini, Hu Xiaodi. A Fusion Method for Infrared and Visible Images in All-weather Road Scenes[J]. Geomatics and Information Science of Wuhan University. DOI: 10.13203/j.whugis20240173
[2]	SONG Zhina, SUI Haigang, LI Yongcheng. A Survey on Ship Detection Technology in High-Resolution Optical Remote Sensing Images[J]. Geomatics and Information Science of Wuhan University, 2021, 46(11): 1703-1715. DOI: 10.13203/j.whugis20200481
[3]	XIANG Tianzhu, GAO Rongrong, YAN Li, XU Zhenliang. Region Feature Based Multi-scale Fusion Method for Thermal Infrared and Visible Images[J]. Geomatics and Information Science of Wuhan University, 2017, 42(7): 911-917. DOI: 10.13203/j.whugis20141007
[4]	ZHANG Lifu, YANG Hang, FANG Conghui, PAN Mao. Thermal Infrared Target Recognition Using Multi-scale Fractal Model[J]. Geomatics and Information Science of Wuhan University, 2012, 37(3): 339-342.
[5]	YING Shen, LI Lin, GAO Yurong. Pedestrian Simulation in Urban Space Based on Visibility Analysis and Agent Techniques[J]. Geomatics and Information Science of Wuhan University, 2011, 36(11): 1367-1370.
[6]	XU Hanqiu, ZHANG Tiejun, LI Chunhua. Cross Comparison of Thermal Infrared Data Between ASTER and Landsat ETM~+ Sensors[J]. Geomatics and Information Science of Wuhan University, 2011, 36(8): 936-940.
[7]	ZHU Zhongmin, GONG Wei, YU Juan, TIAN Liqiao. Applicability Analysis of Transformation Models for Aerosol Optical Depth and Horizontal Visibility[J]. Geomatics and Information Science of Wuhan University, 2010, 35(9): 1086-1090.
[8]	MAO Yue, SONG Xiaoyong, FENG Laiping. Visibility Analysis of X-ray Pulsar Navigation[J]. Geomatics and Information Science of Wuhan University, 2009, 34(2): 222-225.
[9]	YANG Guijun, LIU Qinhuo, LIU Qiang, GU Xingfa. Fusion of Visible and Thermal Infrared Remote Sensing Data Based on GA-SOFM Neural Network[J]. Geomatics and Information Science of Wuhan University, 2007, 32(9): 786-790.
[10]	GONG Shengrong, YANG Shanchao. A Visible Watermarking Algorithm Holding Image Content[J]. Geomatics and Information Science of Wuhan University, 2006, 31(9): 757-760.

Cited By

Cited by

Periodical cited type(2)

1.	徐辛超，高阳. 融合跳跃连接网络与双重注意力机制的可见光与红外遥感影像匹配方法. 地球信息科学学报. 2025(03): 766-783 .
2.	姚国标，张成成，龚健雅，张现军，李兵. 非线性尺度空间改进的光学与SAR影像自动配准. 武汉大学学报(信息科学版). 2024(12): 2249-2260 .

Other cited types(0)

Get Citation

PDF

XML

Article views (758) PDF downloads (154) Cited by(2)

A Method Based on Depth Features for Matching Thermal Infrared Images with Visible Images

Abstract

References

Related Articles

Cited by

Periodical cited type(2)

Other cited types(0)

Catalog

Related

A Method Based on Depth Features for Matching Thermal Infrared Images with Visible Images

Abstract

References

Related Articles

Cited by

Periodical cited type(2)

Other cited types(0)

Catalog

Related

Export File

Citation

Format

Content