A Fusion Method for Infrared and Visible Images in All-weather Road Scenes

Gan Wenxia; Pan Junjie; Geng Jing; Wang Huini; Hu Xiaodi

doi:10.13203/j.whugis20240173

Turn off MathJax

Article Contents

Abstract

References

Gan Wenxia, Pan Junjie, Geng Jing, Wang Huini, Hu Xiaodi. A Fusion Method for Infrared and Visible Images in All-weather Road Scenes[J]. Geomatics and Information Science of Wuhan University. DOI: 10.13203/j.whugis20240173

Citation:

PDF (1822 KB)

A Fusion Method for Infrared and Visible Images in All-weather Road Scenes

1 School of Civil Engineering and Architecture, Wuhan Institute of Technology, Wuhan 430074 China;
2 School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081 China

More Information

Received Date: December 07, 2024

Graphical Abstract

Abstract

Abstract

The purpose of infrared and visible image fusion in the all -weather road scenes is to generate fusion images that contain rich textures and significant goals. However, due to the influence of light conditions and harsh weather factors, the existing infrared and visible image fusion algorithm is difficult to achieve the ideal effect. In response to this issue, an infrared and visible image fusion network suitable for all -weather road scenes was proposed. First of all, we designed a illumination loss function that can guide the intensity distribution of the background texture and the intensity of significant goals in the adaptive network. In addition, the introduction of the Scharr gradient to improve the calculation accuracy of the gradient of texture loss, and restrict the fusion image to retain more delicate texture details. Finally, we designed the second order residual block based on the Resblock structure to enhance the ability to integrate network extraction of strong and weak texture features of images. Comparison and generalization experiments on MFNet, RoadScene, and TNO data show that the algorithm is better than other advanced algorithms in subjective visual effects and objective indicators.
- all-weather road scenes,
- infrared and visible image,
- image fusion,
- illumination loss

FullText(HTML)

References (50)

References

[1]	应申,蒋跃文,顾江岩等.面向自动驾驶的高精地图模型及关键技术[J/OL].武汉大学学报(信息科学版):1-12[2023-11-23]. https://doi.org/10.13203/j.whugis20230227.
[2]	申彩英,朱思瑶,黄兴驰.双目视觉的智能汽车目标检测算法研究[J/OL].重庆理工大学学报(自然科学):1-8[2023-11-06]. http://kns.cnki.net/kcms/detail/50.1205.T.20231011.1117.004.html.
[3]	胡淼,姜麟,陶友凤等.改进YOLOv7的自动驾驶目标检测算法[J/OL].计算机工程与应用:1-11[2023- 11-06]. http://kns.cnki.net/kcms/detail/11.2127.TP.20230922.1630.004.html.
[4]	叶语同,李必军,付黎明.智能驾驶中点云目标快速检测与跟踪[J].武汉大学学报(信息科学版),2019,44(01):139-144+152.DOI: 10.13203/j.whugis20170146.
[5]	Zhang H, Xu H, Tian X, et al. Image fusion meets deep learning:A survey and perspective[J]. Information Fusion, 2021, 76:323-336.
[6]	Ma J, Ma Y, Li C. Infrared and visible image fusion methods and applications:A survey[J]. Information fusion, 2019, 45:153-178.
[7]	Zhang X, Demiris Y. Visible and Infrared Image Fusion Using Deep Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[8]	Ma W, Wang K, Li J, et al. Infrared and Visible Image Fusion Technology and Application:A Review[J]. Sensors, 2023, 23(2):599.
[9]	韩林凯,姚江伟,王坤峰.保留梯度和轮廓的可见光与红外图像融合[J/OL].计算机应用, 1-7[2023-11- 07] http://kns.cnki.net/kcms/detail/51.1307.TP.20230417.1032.002.html.
[10]	Shuai H,Tian H,Beiyi A, et al. VDFEFuse:A novel fusion approach to infrared and visible images[J]. Infrared Physics and Technology,2022,121.
[11]	Sun C, Zhang C, Xiong N. Infrared and visible image fusion techniques based on deep learning:A review[J]. Electronics, 2020, 9(12):2162.
[12]	陈彦林,王志社,邵文禹等.红外与可见光图像多尺度Transformer融合方法[J].红外技术,2023,45(03):266-275.
[13]	向天烛,高熔溶,闫利等.一种顾及区域特征差异的热红外与可见光图像多尺度融合方法[J].武汉大学学报(信息科学版),2017,42(07):911-917.DOI: 10.13203/j.whugis20141007.
[14]	Li S, Kang X, Fang L, et al. Pixel-level image fusion:A survey of the state of the art[J]. information Fusion, 2017, 33:100-112.
[15]	Liu J, Fan X, Huang Z, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022:5802-5811.Li C, Song D, Tong R, et al. Illuminationaware faster R-CNN for robust multispectral pedestrian detection[J]. Pattern Recognition, 2019, 85:161- 171.
[16]	苑朝,赵亚冬,张耀等.基于YOLO轻量化的多模态行人检测算法[J/OL].图学学报:1-12[2023-11- 07]. http://kns.cnki.net/kcms/detail/10.1034.T.20231026.1644.002.html.
[17]	孙文财,胡旭歌,杨志发等.基于红外及可见光图像融合的道路目标检测优化方法[J/OL].吉林大学学报(工学版):1-8[2023-11-07]. https://doi.org/10.13229/j.cnki.jdxbgxb.20230474.
[18]	Geng X, Li M, Liu W, et al. Person tracking by detection using dual visible-infrared cameras[J]. IEEE Internet of Things Journal, 2022, 9(22):23241-23251.
[19]	刘万军,梁林林,曲海成.利用Transformer的多模态目标跟踪算法[J/OL].计算机工程与应用, 1- 11[2023-11-07] http://kns.cnki.net/kcms/detail/11.2127.TP.20230508.1642.022.html.
[20]	Liu W, Liu W, Sun Y. Visible-Infrared Dual-Sensor Fusion for Single-Object Tracking[J]. IEEE Sensors Journal, 2023, 23(4):4118-4128.
[21]	Ha Q, Watanabe K, Karasawa T, et al. MFNet:Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017:5108-5115.
[22]	Hou J, Zhang D, Wu W, et al. A generative adversarial network for infrared and visible image fusion based on semantic segmentation[J]. Entropy, 2021, 23(3):376.
[23]	朱浩,谷小婧,蓝鑫等.基于多尺度轮廓增强的RGB-IR双波段图像语义分割算法[J].激光与红外,2022,52(04):543-551.
[24]	Wang Z, Xu J, Jiang X, et al. Infrared and visible image fusion via hybrid decomposition of NSCT and morphological sequential toggle operator[J]. Optik, 2020, 201:163497.
[25]	Chen J,Li X,Luo L, et al. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J]. Information Sciences,2020,508.
[26]	Hui L,Xiao-Jun W,Josef K. MDLatLRR:A novel decomposition method for infrared and visible image fusion.[J]. IEEE transactions on image processing:a publication of the IEEE Signal Processing Society,2020,29.
[27]	赵庆典,杨德宏.基于图像增强和二次NSCT的红外与可见光图像融合[J/OL].激光与光电子学进展:1-17[2023-11-07]. http://kns.cnki.net/kcms/detail/31.1690.TN.20230714.1054.202.html.
[28]	武凌霄,康家银,姬云翔.NSST域下基于引导滤波与稀疏表示的红外与可见光图像融合[J].红外技术,2023,45(09):915-924.
[29]	Liu Y,Chen X,Ward K R, et al. Image Fusion With Convolutional Sparse Representation.[J]. IEEE Signal Process. Lett.,2016,23(12).
[30]	Liu Y, Yang X, Zhang R, et al. Entropy-based image fusion with joint sparse representation and rolling guidance filter[J]. Entropy, 2020, 22(1):118.
[31]	Cvejic N, Bull D, Canagarajah N. Region-based multimodal image fusion using ICA bases[J]. IEEE Sensors Journal, 2007, 7(5):743-751.
[32]	Bavirisetti D P, Xiao G, Liu G. Multi-sensor image fusion based on fourth order partial differential equations[C]//2017 20th International conference on information fusion (Fusion). IEEE, 2017:1-9.
[33]	Ma J,Chen C,Li C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion,2016,31.
[34]	Ma J,Zhou Z,Wang B, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics and Technology,2017,82.
[35]	Lihua J,Xiaomin Y,Zheng L, et al. SEDRFuse:A Symmetric Encoder-Decoder With Residual Block Network for Infrared and Visible Image Fusion[J]. IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT,2021,70.
[36]	Han X,Hao Z,Jiayi M. Classification Saliency-Based Rule for Visible and Infrared Image Fusion[J]. IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING,2021,7.
[37]	Liu J, Fan X, Jiang J, et al. Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(1):105- 119.
[38]	Ma J,Yu W,Liang P, et al. FusionGAN:A generative adversarial network for infrared and visible image fusion[J]. Information Fusion,2018,48.
[39]	Yang Y, Liu J, Huang S, et al. Infrared and visible image fusion via texture conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(12): 4771-4783.
[40]	Ma J, Zhang H, Shao Z, et al. GANMcC:A generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 70:1-14.
[41]	Xu H, Ma J, Jiang J, et al. U2Fusion:A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1):502-518.
[42]	Zhang H, Ma J. SDNet:A versatile squeeze-and-decomposition network for real-time image fusion[J]. International Journal of Computer Vision, 2021, 129:2761-2785.
[43]	Tang L, Yuan J, Ma J. Image fusion in the loop of high-level vision tasks:A semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82:28-42.
[44]	Ma J, Tang L, Xu M, et al. STDFusionNet:An infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70:1-13.
[45]	Peng C, Tian T, Chen C, et al. Bilateral attention decoder:A lightweight decoder for real-time semantic segmentation[J]. Neural Networks, 2021, 137:188-199.
[46]	4Toet A. TNO Image Fusion Dataset[Internet]. figshare; 2014,https://doi.org/10.6084/m9.figshare.1008029.v2
[47]	Qu G,Zhang D,Yan P. Information measure for performance of image fusion[J]. Electronics Letters,2002,38(7).
[48]	Han Y, Cai Y, Cao Y, et al. A new image fusion performance metric based on visual information fidelity[J]. Information fusion, 2013, 14(2):127-135.
[49]	Piella G, Heijmans H. A new quality metric for image fusion[C]//Proceedings 2003 international conference on image processing (Cat. No. 03CH37429). IEEE, 2003, 3:III-173.
[50]	Paszke A, Gross S, Massa F, et al. Pytorch:An imperative style, high-performance deep learning library[J]. Advances in neural information processing systems, 2019, 32.

[1]	CUI Zhixiang, LAN Chaozhen, ZHANG Yongxian, HOU Huitai, QIN Jianqi. A Method Based on Depth Features for Matching Thermal Infrared Images with Visible Images[J]. Geomatics and Information Science of Wuhan University, 2023, 48(2): 316-324. DOI: 10.13203/j.whugis20200181
[2]	TU Chao-hu, YI Yao-hua, WANG Kai-li, PENG Ji-bing, YIN Ai-guo. Adaptive Multi-level Feature Fusion for Scene Ancient Chinese Text Recognition[J]. Geomatics and Information Science of Wuhan University. DOI: 10.13203/j.whugis20230176
[3]	SONG Zhina, SUI Haigang, LI Yongcheng. A Survey on Ship Detection Technology in High-Resolution Optical Remote Sensing Images[J]. Geomatics and Information Science of Wuhan University, 2021, 46(11): 1703-1715. DOI: 10.13203/j.whugis20200481
[4]	XIANG Tianzhu, GAO Rongrong, YAN Li, XU Zhenliang. Region Feature Based Multi-scale Fusion Method for Thermal Infrared and Visible Images[J]. Geomatics and Information Science of Wuhan University, 2017, 42(7): 911-917. DOI: 10.13203/j.whugis20141007
[5]	Shao Zhenfeng, Bai Yun, Zhou Xiran. Improved Multi-scale Retinex Image Enhancement of UnderPoor Illumination[J]. Geomatics and Information Science of Wuhan University, 2015, 40(1): 32-39.
[6]	WANG Zhijun, GU Chongshi, ZHANG Zhijun. Evaluation Method of Loss-of-life Caused by Dam Breach Based on GIS and Neural Networks Optimized by Genetic Algorithms[J]. Geomatics and Information Science of Wuhan University, 2010, 35(1): 64-68.
[7]	TIAN Jing, GUO Qingsheng, FENG Ke, MA Meng. Progressive Selection Approach of Streets Based on Information Loss[J]. Geomatics and Information Science of Wuhan University, 2009, 34(3): 362-365.
[8]	YANG Guijun, LIU Qinhuo, LIU Qiang, GU Xingfa. Fusion of Visible and Thermal Infrared Remote Sensing Data Based on GA-SOFM Neural Network[J]. Geomatics and Information Science of Wuhan University, 2007, 32(9): 786-790.
[9]	XING Shuai, TAN Bing, XU Qing, LI Jiansheng. A New Algorithm for Remote Sensing Image Fusion Using Complex Wavelet Transform[J]. Geomatics and Information Science of Wuhan University, 2007, 32(1): 75-77.
[10]	GONG Shengrong, YANG Shanchao. A Visible Watermarking Algorithm Holding Image Content[J]. Geomatics and Information Science of Wuhan University, 2006, 31(9): 757-760.

Cited By

Get Citation

PDF

XML

Article views (101) PDF downloads (14)

A Fusion Method for Infrared and Visible Images in All-weather Road Scenes

Abstract

References

Related Articles

Catalog

Related

A Fusion Method for Infrared and Visible Images in All-weather Road Scenes

Abstract

References

Related Articles

Catalog

Related

Export File

Citation

Format

Content