Volume 47 Issue 8
Aug.  2022
Turn off MathJax
Article Contents
Abstract
References
Related Articles
LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190. DOI: 10.13203/j.whugis20210652
Citation: LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190. DOI: 10.13203/j.whugis20210652
shu

A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning

  • 1.

    School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China

Funds: 

The National Natural Science Foundation of China 42030102

The National Natural Science Foundation of China 41971284

Innovative Research Groups of the Natural Science Foundation of Hubei Province 2020CFA003

More Information
  • Author Bio:

    LI Yansheng, PhD, associator professor, majors in intelligent mining of remote sensing big data. E-mail: yansheng.li@whu.edu.cn

  • Corresponding author:

    ZHANG Yongjun, PhD, professor.E-mail: zhangyj@whu.edu.cn

  • Received Date: November 27, 2021
  • Available Online: January 19, 2022
  • Published Date: August 04, 2022
    Graphical Abstract
    • Abstract
  •   Objectives  In the remote sensing (RS) big data era, intelligent interpretation of remote sensing images (RSI) is the key technology to mine the value of big RS data and promote several important applications. Traditional knowledge-driven RS interpretation methods, represented by expert systems, are highly interpretable, but generally show poor performance due to the interpretation knowledge being difficult to be completely and accurately expressed. With the development of deep learning in computer vision and other fields, it has gradually become the mainstream technology of RSI interpretation. However, the deep learning technique still has some fatal flaws in the RS field, such as poor interpretability and weak generalization ability. In order to overcome these problems, how to effectively combine knowledge inference and data learning has become an important research trend in the field of RS big data intelligent processing. Generally, knowledge inference relies on a strong domain knowledge base, but the research on RS knowledge graph (RS-KG) is very scarce and there is no available large-scale KG database for RSI interpretation now.
      Methods  To overcome the above considerations, this paper focuses on the construction and evolution of the RS-KG for RSI interpretation and establishes the RS-KG takes into account the RS imaging mechanism and geographic knowledge. Supported by KG in the RS field, this paper takes three typical RSI interpretation tasks, namely, zero-shot RSI scene classification, interpretable RSI semantic segmentation, and large-scale RSI scene graph generation, as examples, to discuss the performance of the novel generation RSI interpretation paradigm which couples KG and deep learning.
      Results and Conclusions  A large number of experimental results show that the combination of RS-KG inference and deep data learning can effectively improve the performance of RSI interpretation.The introduction of RS-KG can effectively improve the interpretation accuracy, generalization ability, anti-interference ability, and interpretability of deep learning models. These advantages make RS-KG promising in the novel generation RSI interpretation paradigm.
    • FullText(HTML)
    • References (37)
  • [1]
    李德仁, 张良培, 夏桂松. 遥感大数据自动分析与数据挖掘[J]. 测绘学报, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm

    Li Deren, Zhang Liangpei, Xia Guisong. Automatic Analysis and Mining of Remote Sensing Big Data[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm
    [2]
    付琨, 孙显, 仇晓兰, 等. 遥感大数据条件下多星一体化处理与分析[J]. 遥感学报, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm

    Fu Kun, Sun Xian, Qiu Xiaolan, et al. Multi-Satellite Integrated Processing and Analysis Method Under Remote Sensing Big Data[J]. National Remote Sensing Bulletin, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm
    [3]
    潘灼坤, 胡月明, 王广兴, 等. 对遥感在城市更新监测应用中的认知和思考[J]. 遥感技术与应用, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm

    Pan Zhuokun, Hu Yueming, Wang Guangxing, et al. Cognitions and Perspectives in Remote Sensing of Urban Renewal Monitoring[J]. Remote Sensing Technology and Application, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm
    [4]
    McCarthy J, Minsky M, Rochester N, et al. A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955[J]. AI Mag, 2006, 27: 12-14
    [5]
    Rosenblatt F. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain[J]. Psychological Review, 1958, 65(6): 386-408 doi: 10.1037/h0042519
    [6]
    张钹, 朱军, 苏航. 迈向第三代人工智能[J]. 中国科学: 信息科学, 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm

    Zhang Bo, Zhu Jun, Su Hang. Toward the Third Generation of Artificial Intelligence[J]. Scientia Sinica(Informationis), 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm
    [7]
    Goodenough D G, Goldberg M, Plunkett G, et al. An Expert System for Remote Sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 349-359 doi: 10.1109/TGRS.1987.289805
    [8]
    Matsuyama T. Knowledge-Based Aerial Image Understanding Systems and Expert Systems for Image Processing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 305-316 doi: 10.1109/TGRS.1987.289802
    [9]
    许珺, 裴韬, 姚永慧. 地学知识图谱的定义、内涵和表达方式的探讨[J]. 地球信息科学学报, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm

    Xu Jun, Pei Tao, Yao Yonghui. Conceptual Framework and Representation of Geographic Knowledge Map[J]. Journal of Geo-Information Science, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm
    [10]
    王志华, 杨晓梅, 周成虎. 面向遥感大数据的地学知识图谱构想[J]. 地球信息科学学报, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm

    Wang Zhihua, Yang Xiaomei, Zhou Chenghu. Geographic Knowledge Graph for Remote Sensing Big Data[J]. Journal of Geo-Information Science, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm
    [11]
    张雪英, 张春菊, 吴明光, 等. 顾及时空特征的地理知识图谱构建方法[J]. 中国科学: 信息科学, 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm

    Zhang Xueying, Zhang Chunju, Wu Mingguang, et al. Spatiotemporal Features Based Geographical Knowledge Graph Construction[J]. Scientia Sinica (Informationis), 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm
    [12]
    Alirezaie M, Längkvist M, Sioutis M, et al. Semantic Referee: A Neural-Symbolic Framework for Enhancing Geospatial Semantic Segmentation[J]. Semantic Web, 2019, 10(5): 863-880 doi: 10.3233/SW-190362
    [13]
    Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[J]. CoRR, 2013, DOI: 1310.4546
    [14]
    Bordes A, Usunier N, Garcia-Duran A, et al. Translating Embeddings for Modeling Multi-relational Data[C]//Neural Information Processing Systems(NIPS), Carson City, USA, 2013
    [15]
    Chen Y W, Zhang S L, Peng X, et al. A Collaborative Ontology Construction Tool with Conflicts Detection[C]//The 4th International Conference on Semantics, Knowledge and Grid, Beijing, China, 2008
    [16]
    杜清运, 任福. 空间信息的自然语言表达模型[J]. 武汉大学学报·信息科学版, 2014, 39(6): 682-688 doi: 10.13203/j.whugis20140118

    Du Qingyun, Ren Fu. Representation Model of Spatial Information in Natural Language[J]. Geomatics and Information Science of Wuhan University, 2014, 39(6): 682-688 doi: 10.13203/j.whugis20140118
    [17]
    Chen J Y, Jimenez-Ruiz E, Horrocks I, et al. Learning Semantic Annotations for Tabular Data[C]/The 28th International Joint Conference on Artificial Intelligence, Macao, China, 2019
    [18]
    Chen T, Xu R F, He Y L, et al. Improving Sentiment Analysis via Sentence Type Classification Using BiLSTM-CRF and CNN[J]. Expert Systems with Applications, 2017, 72: 221-230 doi: 10.1016/j.eswa.2016.10.065
    [19]
    Haklay M, Weber P. OpenStreetMap: User-Generated Street Maps[J]. IEEE Pervasive Computing, 2008, 7(4): 12-18 doi: 10.1109/MPRV.2008.80
    [20]
    Tempelmeier N, Demidova E. Linking OpenStreetMap with Knowledge Graphs—Link Discovery for Schema-Agnostic Volunteered Geographic Information[J]. Future Generation Computer Systems, 2021, 116: 349-364 doi: 10.1016/j.future.2020.11.003
    [21]
    Yang X, Liu Q Q, Yan J C, et al. R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object[J]. arXiv, 2019, DOI: 1908.05612
    [22]
    Sun Z Q, Hu W, Zhang Q H, et al. Bootstrapping Entity Alignment with Knowledge Graph Embedding[C]//The 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018
    [23]
    Zhang Z Q, Cai J Y, Zhang Y D, et al. Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction[C]//The AAAI Conference on Artificial Intelligence, New York, USA, 2020
    [24]
    李彦胜, 孔德宇, 张永军, 等. 联合稳健跨域映射和渐进语义基准修正的零样本遥感影像场景分类[J]. 测绘学报, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm

    Li Yansheng, Kong Deyu, Zhang Yongjun, et al. Zero-Shot Remote Sensing Image Scene Classification Based on Robust Cross-Domain Mapping and Gradual Refinement of Semantic Space[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm
    [25]
    Li Y S, Kong D Y, Zhang Y J, et al. Robust Deep Alignment Network with Remote Sensing Knowledge Graph for Zero-Shot and Generalized Zero-Shot Remote Sensing Image Scene Classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 179: 145-158 doi: 10.1016/j.isprsjprs.2021.08.001
    [26]
    Wang Z, Zhang J, Feng J, et al. Knowledge Graph Embedding by Translating on Hyperplanes[C]//AAAI Conference on Artificial Intelligence, Québec City, Canada, 2014
    [27]
    Kodirov E, Xiang T, Gong S G. Semantic Autoencoder for Zero-Shot Learning[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017
    [28]
    Li Y N, Wang D H, Hu H H, et al. Zero-Shot Recognition Using Dual Visual-Semantic Mapping Paths[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017
    [29]
    Elhoseiny M, Elfeki M. Creativity Inspired Zero-Shot Learning[C]//IEEE/CVF International Conference on Computer Vision(ICCV), Seoul, Korea(South), 2019
    [30]
    Schönfeld E, Ebrahimi S, Sinha S, et al. Generalized Zero and Few-Shot Learning via Aligned Variational Autoencoders[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, 2019
    [31]
    Tong X Y, Xia G S, Lu Q K, et al. Land-Cover Classification with High-Resolution Remote Sensing Images Using Transferable Deep Models[J]. Remote Sensing of Environment, 2020, 237: 111322 doi: 10.1016/j.rse.2019.111322
    [32]
    Li Y S, Ouyang S, Zhang Y J. Collaboratively Boosting Data-Driven Deep Learning and Knowledge-Guided Ontological Reasoning for Semantic Segmentation of Remote Sensing Imagery[J]. arXiv, 2020, DOI: 2010.02451
    [33]
    Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation[M]//Cham: Springer International Publishing, 2015
    [34]
    Ma L, Liu Y, Zhang X L, et al. Deep Learning in Remote Sensing Applications: A Meta-Analysis and Review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152: 166-177 doi: 10.1016/j.isprsjprs.2019.04.015
    [35]
    李德仁. 脑认知与空间认知: 论空间大数据与人工智能的集成[J]. 武汉大学学报·信息科学版, 2018, 43(12): 1761-1767 doi: 10.13203/j.whugis20180411

    Li Deren. Brain Cognition and Spatial Cognition: On Integration of Geo-spatial Big Data and Artificial Intelligence[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1761-1767 doi: 10.13203/j.whugis20180411
    [36]
    Zellers R, Yatskar M, Thomson S, et al. Neural Motifs: Scene Graph Parsing with Global Context[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018
    [37]
    Zhu Y H, Jiang S Q, Li X Y. Visual Relationship Detection with Object Spatial Distribution[C]//IEEE International Conference on Multimedia and Expo, Hong Kong, China, 2017
    • Related Articles

  • Related Articles

    [1]HU Xiangyun, ZHANG Mi, ZHANG Zuxun, LI Xiaokai, DENG Kai, JIANG Huiwei, PANG Shiyan, RAO Youzhuo, GONG Jinqi, FENG Cunjun, ZHAN Yuanzeng, WANG Xingkun. Key Technologies and Applications of EasyFeature—An Intelligent Interpretation System for Remote Sensing Images[J]. Geomatics and Information Science of Wuhan University, 2025, 50(3): 554-561. DOI: 10.13203/j.whugis20240159
    [2]LI Jiayi, HUANG Xin, HU Yuping, ZHANG Zhen, ZHANG Xueting, ZHANG Shulei, FANG Xing. Fusion of Optical Daily and Night-time Light Remote Sensing Images for Collapsed Building Detection: A Case in Turkey Mw 7.8 Earthquake[J]. Geomatics and Information Science of Wuhan University, 2023, 48(10): 1706-1714. DOI: 10.13203/j.whugis20230275
    [3]LÜ Yafei, XIONG Wei, ZHANG Xiaohan. A General Cross-Modal Correlation Learning Method for Remote Sensing[J]. Geomatics and Information Science of Wuhan University, 2022, 47(11): 1887-1895. DOI: 10.13203/j.whugis20200213
    [4]GUO Congzhou, LI Ke, LI He, TONG Xiaochong, WANG Xiwen. Deep Convolution Neural Network Method for Remote Sensing Image Quality Level Classification[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1279-1286. DOI: 10.13203/j.whugis20200292
    [5]ZHANG Haiming, WANG Mingchang, CHEN Xueye, WANG Fengyan, YANG Guodong, GAO Su. Remote Sensing Change Detection Based on Deep Belief Networks Optimized by Domain Knowledge[J]. Geomatics and Information Science of Wuhan University, 2022, 47(5): 762-768. DOI: 10.13203/j.whugis20190471
    [6]SHAO Zhenfeng, SUN Yueming, XI Jiangbo, LI Yan. Intelligent Optimization Learning for Semantic Segmentation of High Spatial Resolution Remote Sensing Images[J]. Geomatics and Information Science of Wuhan University, 2022, 47(2): 234-241. DOI: 10.13203/j.whugis20200640
    [7]CHEN Hang, LUO Bin. Multi-angle Remote Sensing Images Super-Resolution Reconstruction Using Dynamic Upsampling Filter Deep Network[J]. Geomatics and Information Science of Wuhan University, 2021, 46(11): 1716-1726. DOI: 10.13203/j.whugis20200651
    [8]JI Shunping, LUO Chong, LIU Jin. A Review of Dense Stereo Image Matching Methods Based on Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2021, 46(2): 193-202. DOI: 10.13203/j.whugis20200620
    [9]ZHANG Liqiang, LI Yang, HOU Zhengyang, LI Xingang, GENG Hao, WANG Yuebin, LI Jingwen, ZHU Panpan, MEI Jie, JIANG Yanxiao, LI Shuaipeng, XIN Qi, CUI Ying, LIU Suhong. Deep Learning and Remote Sensing Data Analysis[J]. Geomatics and Information Science of Wuhan University, 2020, 45(12): 1857-1864. DOI: 10.13203/j.whugis20200650
    [10]ZHANG Bing. Remotely Sensed Big Data Era and Intelligent Information Extraction[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1861-1871. DOI: 10.13203/j.whugis20180172

  • Cited by

    Periodical cited type(22)

    1. 李彦胜，钟振宇，孟庆祥，毛之典，党博，王涛，冯苑君，张永军. 遥感时空知识图谱驱动的自然资源要素变化图斑智能净化. 地球信息科学学报. 2025(02): 350-366 .
    2. 邓乃尔，徐浩，周文，唐小川，王浩，闫晓闯，蒋柯. 基于深度学习的岩石矿物智能识别研究进展与发展趋势. 成都理工大学学报(自然科学版). 2025(01): 64-93 .
    3. 李彦胜，武康，欧阳松，杨坤，李和平，张永军. 地学知识图谱引导的遥感影像语义分割. 遥感学报. 2024(02): 455-469 .
    4. 李治泓，朱庆，廖成，胡翰，陈琳. 基于多源遥感的铁路外部环境隐患监测方法综述. 航天返回与遥感. 2024(01): 15-28 .
    5. 李彦胜，吴敏郎，张永军. 知识图谱约束深度网络的高分辨率遥感影像场景分类. 测绘学报. 2024(04): 677-688 .
    6. 吴田军，骆剑承，李曼嘉，张静，赵馨，胡晓东，左进，闵帆，王玲玉，黄启厅. 地理时空数字化底座理论框架构建与应用实践. 地球信息科学学报. 2024(04): 799-830 .
    7. 何静，赵睿，张恒硕. 知识图谱的可视化文献计量分析. 计算机科学. 2024(S1): 13-22 .
    8. 曹佃龙. 基于改进YOLOv5网络模型的无人机影像道路目标检测. 北京测绘. 2024(06): 936-941 .
    9. 王志华，杨晓梅，刘岳明，刘彬，张俊瑶，刘晓亮，孟丹，郜酷，曾晓伟，丁亚新. 遥感影像地学分析的地理学原理及等级斑块建模框架. 遥感学报. 2024(06): 1412-1424 .
    10. 施利涛. 基于大数据分析的高分辨率可见光遥感卫星影像解译能力评估. 北京测绘. 2024(09): 1265-1270 .
    11. 张永军，李彦胜，党博，武康，郭昕，王剑，陈景东，杨铭. 多模态遥感基础大模型：研究现状与未来展望. 测绘学报. 2024(10): 1942-1954 .
    12. 张小娟. 基于神经网络和本体推理的遥感影像语义分割. 电子制作. 2024(20): 72-74 .
    13. 李玉，李天惠，赵泉华. 遥感影像直觉模糊集分割法. 测绘学报. 2023(03): 405-418 .
    14. 陆锋，诸云强，张雪英. 时空知识图谱研究进展与展望. 地球信息科学学报. 2023(06): 1091-1105 .
    15. 侯昭阳，吕开云，龚循强，支君豪，王楠. 一种结合低级视觉特征和PAPCNN的NSST域遥感影像融合方法. 武汉大学学报(信息科学版). 2023(06): 960-969 .
    16. 王正庆，李策，陈雨林，刘中楠，丁华栋，余言，李天鹏，郭知鑫. 基于遥感技术的地质灾害解译及评价监测进展. 南华大学学报(自然科学版). 2023(02): 31-36 .
    17. 高梅香，朱家祺，刘爽，程鑫，刘冬，李彦胜. 土壤动物知识图谱构建理论、方法与技术——以浙江天目山土壤螨类为例. 生态学报. 2023(16): 6862-6877 .
    18. 连仁包，张振敏，廖一鹏，邹长忠，黄立勤. 结合测地距离场与曲线平滑的遥感图像道路中心线快速提取. 测绘学报. 2023(08): 1317-1329 .
    19. 沈伟豪，钟燕飞，王俊珏，郑卓，马爱龙. 多模态数据的洪涝灾害知识图谱构建与应用. 武汉大学学报(信息科学版). 2023(12): 2009-2018 .
    20. 张鹏，刘石栋，刘振军. 结合高效通道注意力的轻量级遥感影像目标检测方法. 测绘与空间地理信息. 2023(12): 53-56 .
    21. 张兵，杨晓梅，高连如，孟瑜，孙显，肖晨超，倪丽. 遥感大数据智能解译的地理学认知模型与方法. 测绘学报. 2022(07): 1398-1415 .
    22. 龚健雅，张展，贾浩巍，周桓，赵元昕，熊汉江. 面向多源数据地物提取的遥感知识感知与多尺度特征融合网络. 武汉大学学报(信息科学版). 2022(10): 1546-1554 .

    Other cited types(6)

Catalog

    Get Citation
    PDF
    XML
    Article views (7087) PDF downloads (1516) Cited by(28)
    Related

    Copyright © 2013 《武汉大学学报·信息科学版》编辑部

    Address：武汉大学文理学部本科生院楼北5楼Post Code：430072

    Tel：027-68778465,68788631E-mail:whuxxb@vip.163.com

    Submit Article

    Submission Guidelines

    e

    Contact Us

    Qrcode

    Supported by: Beijing Renhe Information Technology Co., Ltd.  Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return