一种全天候道路场景下的红外和可见光图像融合方法

甘文霞, 潘俊杰, 耿晶, 王慧妮, 胡小弟

甘文霞, 潘俊杰, 耿晶, 王慧妮, 胡小弟. 一种全天候道路场景下的红外和可见光图像融合方法[J]. 武汉大学学报 ( 信息科学版). DOI: 10.13203/j.whugis20240173
引用本文: 甘文霞, 潘俊杰, 耿晶, 王慧妮, 胡小弟. 一种全天候道路场景下的红外和可见光图像融合方法[J]. 武汉大学学报 ( 信息科学版). DOI: 10.13203/j.whugis20240173
Gan Wenxia, Pan Junjie, Geng Jing, Wang Huini, Hu Xiaodi. A Fusion Method for Infrared and Visible Images in All-weather Road Scenes[J]. Geomatics and Information Science of Wuhan University. DOI: 10.13203/j.whugis20240173
Citation: Gan Wenxia, Pan Junjie, Geng Jing, Wang Huini, Hu Xiaodi. A Fusion Method for Infrared and Visible Images in All-weather Road Scenes[J]. Geomatics and Information Science of Wuhan University. DOI: 10.13203/j.whugis20240173

一种全天候道路场景下的红外和可见光图像融合方法

基金项目: 

国家自然科学基金项目(42471388,42201461),湖北省交通厅科研项目(2023-121-3-4)。

详细信息
    作者简介:

    甘文霞,博士,副教授,从事遥感图像处理与应用、计算机视觉、深度学习等工作。charlottegan@whu.edu.cn

    通讯作者:

    王慧妮,博士,副教授。wanghuini@wit.edu.cn

A Fusion Method for Infrared and Visible Images in All-weather Road Scenes

  • 摘要: 结合红外图像的高亮目标信息及可见光图像丰富的纹理细节可以生成同时包含丰富纹理和显著目标的融合图像,但受限于光照条件、恶劣天气的复杂多变性约束,现有的红外和可见光图像融合算法在全天候道路应用场景中难以达到理想的效果。针对此问题,提出一种适用于全天候道路场景的红外与可见光图像融合网络。首先通过光照损失函数设计,指导融合网络自适应地保持背景纹理细节和显著目标的强度分布。此外,纹理损失函数引入Scharr梯度算子提高梯度计算精度,从而约束融合图像保留更细腻的纹理细节。最后,基于Resblock结构设计了二阶梯度残差块,以增强融合网络提取图像强弱纹理特征的能力。在公开数据集MFNet、Roadscene和TNO数据上对该算法进行了对比实验和泛化实验,结果表明算法在主观视觉效果和客观指标评价上均优于其它先进算法,得到的融合图像纹理清晰且红外目标显著。
    Abstract: The purpose of infrared and visible image fusion in the all -weather road scenes is to generate fusion images that contain rich textures and significant goals. However, due to the influence of light conditions and harsh weather factors, the existing infrared and visible image fusion algorithm is difficult to achieve the ideal effect. In response to this issue, an infrared and visible image fusion network suitable for all -weather road scenes was proposed. First of all, we designed a illumination loss function that can guide the intensity distribution of the background texture and the intensity of significant goals in the adaptive network. In addition, the introduction of the Scharr gradient to improve the calculation accuracy of the gradient of texture loss, and restrict the fusion image to retain more delicate texture details. Finally, we designed the second order residual block based on the Resblock structure to enhance the ability to integrate network extraction of strong and weak texture features of images. Comparison and generalization experiments on MFNet, RoadScene, and TNO data show that the algorithm is better than other advanced algorithms in subjective visual effects and objective indicators.
  • [1] 应申,蒋跃文,顾江岩等.面向自动驾驶的高精地图模型及关键技术[J/OL].武汉大学学报(信息科学 版):1-12[2023-11-23]. https://doi.org/10.13203/j.whugis20230227.
    [2] 申彩英,朱思瑶,黄兴驰.双目视觉的智能汽车目标检测算法研究[J/OL].重庆理工大学学报(自然科 学):1-8[2023-11-06]. http://kns.cnki.net/kcms/detail/50.1205.T.20231011.1117.004.html.
    [3] 胡淼,姜麟,陶友凤等.改进YOLOv7的自动驾驶目标检测算法[J/OL].计算机工程与应用:1-11[2023- 11-06]. http://kns.cnki.net/kcms/detail/11.2127.TP.20230922.1630.004.html.
    [4] 叶语同,李必军,付黎明.智能驾驶中点云目标快速检测与跟踪[J].武汉大学学报(信息科学 版),2019,44(01):139-144+152.DOI: 10.13203/j.whugis20170146.
    [5]

    Zhang H, Xu H, Tian X, et al. Image fusion meets deep learning:A survey and perspective[J]. Information Fusion, 2021, 76:323-336.

    [6]

    Ma J, Ma Y, Li C. Infrared and visible image fusion methods and applications:A survey[J]. Information fusion, 2019, 45:153-178.

    [7]

    Zhang X, Demiris Y. Visible and Infrared Image Fusion Using Deep Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

    [8]

    Ma W, Wang K, Li J, et al. Infrared and Visible Image Fusion Technology and Application:A Review[J]. Sensors, 2023, 23(2):599.

    [9] 韩林凯,姚江伟,王坤峰.保留梯度和轮廓的可见光与红外图像融合[J/OL].计算机应用, 1-7[2023-11- 07] http://kns.cnki.net/kcms/detail/51.1307.TP.20230417.1032.002.html.
    [10]

    Shuai H,Tian H,Beiyi A, et al. VDFEFuse:A novel fusion approach to infrared and visible images[J]. Infrared Physics and Technology,2022,121.

    [11]

    Sun C, Zhang C, Xiong N. Infrared and visible image fusion techniques based on deep learning:A review[J]. Electronics, 2020, 9(12):2162.

    [12] 陈彦林,王志社,邵文禹等.红外与可见光图像多尺度Transformer融合方法[J].红外技 术,2023,45(03):266-275.
    [13] 向天烛,高熔溶,闫利等.一种顾及区域特征差异的热红外与可见光图像多尺度融合方法[J].武汉大学 学报(信息科学版),2017,42(07):911-917.DOI: 10.13203/j.whugis20141007.
    [14]

    Li S, Kang X, Fang L, et al. Pixel-level image fusion:A survey of the state of the art[J]. information Fusion, 2017, 33:100-112.

    [15]

    Liu J, Fan X, Huang Z, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022:5802-5811.Li C, Song D, Tong R, et al. Illuminationaware faster R-CNN for robust multispectral pedestrian detection[J]. Pattern Recognition, 2019, 85:161- 171.

    [16] 苑朝,赵亚冬,张耀等.基于YOLO轻量化的多模态行人检测算法[J/OL].图学学报:1-12[2023-11- 07]. http://kns.cnki.net/kcms/detail/10.1034.T.20231026.1644.002.html.
    [17] 孙文财,胡旭歌,杨志发等.基于红外及可见光图像融合的道路目标检测优化方法[J/OL].吉林大学学 报(工学版):1-8[2023-11-07]. https://doi.org/10.13229/j.cnki.jdxbgxb.20230474.
    [18]

    Geng X, Li M, Liu W, et al. Person tracking by detection using dual visible-infrared cameras[J]. IEEE Internet of Things Journal, 2022, 9(22):23241-23251.

    [19] 刘万军,梁林林,曲海成.利用Transformer的多模态目标跟踪算法[J/OL].计算机工程与应用, 1- 11[2023-11-07] http://kns.cnki.net/kcms/detail/11.2127.TP.20230508.1642.022.html.
    [20]

    Liu W, Liu W, Sun Y. Visible-Infrared Dual-Sensor Fusion for Single-Object Tracking[J]. IEEE Sensors Journal, 2023, 23(4):4118-4128.

    [21]

    Ha Q, Watanabe K, Karasawa T, et al. MFNet:Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017:5108-5115.

    [22]

    Hou J, Zhang D, Wu W, et al. A generative adversarial network for infrared and visible image fusion based on semantic segmentation[J]. Entropy, 2021, 23(3):376.

    [23] 朱浩,谷小婧,蓝鑫等.基于多尺度轮廓增强的RGB-IR双波段图像语义分割算法[J].激光与红 外,2022,52(04):543-551.
    [24]

    Wang Z, Xu J, Jiang X, et al. Infrared and visible image fusion via hybrid decomposition of NSCT and morphological sequential toggle operator[J]. Optik, 2020, 201:163497.

    [25]

    Chen J,Li X,Luo L, et al. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition[J]. Information Sciences,2020,508.

    [26]

    Hui L,Xiao-Jun W,Josef K. MDLatLRR:A novel decomposition method for infrared and visible image fusion.[J]. IEEE transactions on image processing:a publication of the IEEE Signal Processing Society,2020,29.

    [27] 赵庆典,杨德宏.基于图像增强和二次NSCT的红外与可见光图像融合[J/OL].激光与光电子学进 展:1-17[2023-11-07]. http://kns.cnki.net/kcms/detail/31.1690.TN.20230714.1054.202.html.
    [28] 武凌霄,康家银,姬云翔.NSST域下基于引导滤波与稀疏表示的红外与可见光图像融合[J].红外技 术,2023,45(09):915-924.
    [29]

    Liu Y,Chen X,Ward K R, et al. Image Fusion With Convolutional Sparse Representation.[J]. IEEE Signal Process. Lett.,2016,23(12).

    [30]

    Liu Y, Yang X, Zhang R, et al. Entropy-based image fusion with joint sparse representation and rolling guidance filter[J]. Entropy, 2020, 22(1):118.

    [31]

    Cvejic N, Bull D, Canagarajah N. Region-based multimodal image fusion using ICA bases[J]. IEEE Sensors Journal, 2007, 7(5):743-751.

    [32]

    Bavirisetti D P, Xiao G, Liu G. Multi-sensor image fusion based on fourth order partial differential equations[C]//2017 20th International conference on information fusion (Fusion). IEEE, 2017:1-9.

    [33]

    Ma J,Chen C,Li C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion,2016,31.

    [34]

    Ma J,Zhou Z,Wang B, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics and Technology,2017,82.

    [35]

    Lihua J,Xiaomin Y,Zheng L, et al. SEDRFuse:A Symmetric Encoder-Decoder With Residual Block Network for Infrared and Visible Image Fusion[J]. IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT,2021,70.

    [36]

    Han X,Hao Z,Jiayi M. Classification Saliency-Based Rule for Visible and Infrared Image Fusion[J]. IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING,2021,7.

    [37]

    Liu J, Fan X, Jiang J, et al. Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(1):105- 119.

    [38]

    Ma J,Yu W,Liang P, et al. FusionGAN:A generative adversarial network for infrared and visible image fusion[J]. Information Fusion,2018,48.

    [39]

    Yang Y, Liu J, Huang S, et al. Infrared and visible image fusion via texture conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(12): 4771-4783.

    [40]

    Ma J, Zhang H, Shao Z, et al. GANMcC:A generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 70:1-14.

    [41]

    Xu H, Ma J, Jiang J, et al. U2Fusion:A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1):502-518.

    [42]

    Zhang H, Ma J. SDNet:A versatile squeeze-and-decomposition network for real-time image fusion[J]. International Journal of Computer Vision, 2021, 129:2761-2785.

    [43]

    Tang L, Yuan J, Ma J. Image fusion in the loop of high-level vision tasks:A semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82:28-42.

    [44]

    Ma J, Tang L, Xu M, et al. STDFusionNet:An infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70:1-13.

    [45]

    Peng C, Tian T, Chen C, et al. Bilateral attention decoder:A lightweight decoder for real-time semantic segmentation[J]. Neural Networks, 2021, 137:188-199.

    [46]

    4Toet A. TNO Image Fusion Dataset[Internet]. figshare; 2014,https://doi.org/10.6084/m9.figshare.1008029.v2

    [47]

    Qu G,Zhang D,Yan P. Information measure for performance of image fusion[J]. Electronics Letters,2002,38(7).

    [48]

    Han Y, Cai Y, Cao Y, et al. A new image fusion performance metric based on visual information fidelity[J]. Information fusion, 2013, 14(2):127-135.

    [49]

    Piella G, Heijmans H. A new quality metric for image fusion[C]//Proceedings 2003 international conference on image processing (Cat. No. 03CH37429). IEEE, 2003, 3:III-173.

    [50]

    Paszke A, Gross S, Massa F, et al. Pytorch:An imperative style, high-performance deep learning library[J]. Advances in neural information processing systems, 2019, 32.

计量
  • 文章访问数:  73
  • HTML全文浏览量:  4
  • PDF下载量:  9
  • 被引次数: 0
出版历程
  • 收稿日期:  2024-12-07

目录

    /

    返回文章
    返回