## Message Board

Volume 47 Issue 8
Aug.  2022
Turn off MathJax
Article Contents

LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190. doi: 10.13203/j.whugis20210652
 Citation: LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190.

# A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning

##### doi: 10.13203/j.whugis20210652
Funds:

The National Natural Science Foundation of China 42030102

The National Natural Science Foundation of China 41971284

Innovative Research Groups of the Natural Science Foundation of Hubei Province 2020CFA003

• Author Bio:

LI Yansheng, PhD, associator professor, majors in intelligent mining of remote sensing big data. E-mail: yansheng.li@whu.edu.cn

• Corresponding author: ZHANG Yongjun, PhD, professor.E-mail: zhangyj@whu.edu.cn
• Publish Date: 2022-08-05
•   Objectives  In the remote sensing (RS) big data era, intelligent interpretation of remote sensing images (RSI) is the key technology to mine the value of big RS data and promote several important applications. Traditional knowledge-driven RS interpretation methods, represented by expert systems, are highly interpretable, but generally show poor performance due to the interpretation knowledge being difficult to be completely and accurately expressed. With the development of deep learning in computer vision and other fields, it has gradually become the mainstream technology of RSI interpretation. However, the deep learning technique still has some fatal flaws in the RS field, such as poor interpretability and weak generalization ability. In order to overcome these problems, how to effectively combine knowledge inference and data learning has become an important research trend in the field of RS big data intelligent processing. Generally, knowledge inference relies on a strong domain knowledge base, but the research on RS knowledge graph (RS-KG) is very scarce and there is no available large-scale KG database for RSI interpretation now.  Methods  To overcome the above considerations, this paper focuses on the construction and evolution of the RS-KG for RSI interpretation and establishes the RS-KG takes into account the RS imaging mechanism and geographic knowledge. Supported by KG in the RS field, this paper takes three typical RSI interpretation tasks, namely, zero-shot RSI scene classification, interpretable RSI semantic segmentation, and large-scale RSI scene graph generation, as examples, to discuss the performance of the novel generation RSI interpretation paradigm which couples KG and deep learning.  Results and Conclusions  A large number of experimental results show that the combination of RS-KG inference and deep data learning can effectively improve the performance of RSI interpretation.The introduction of RS-KG can effectively improve the interpretation accuracy, generalization ability, anti-interference ability, and interpretability of deep learning models. These advantages make RS-KG promising in the novel generation RSI interpretation paradigm.
•  [1] 李德仁, 张良培, 夏桂松. 遥感大数据自动分析与数据挖掘[J]. 测绘学报, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm Li Deren, Zhang Liangpei, Xia Guisong. Automatic Analysis and Mining of Remote Sensing Big Data[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm [2] 付琨, 孙显, 仇晓兰, 等. 遥感大数据条件下多星一体化处理与分析[J]. 遥感学报, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm Fu Kun, Sun Xian, Qiu Xiaolan, et al. Multi-Satellite Integrated Processing and Analysis Method Under Remote Sensing Big Data[J]. National Remote Sensing Bulletin, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm [3] 潘灼坤, 胡月明, 王广兴, 等. 对遥感在城市更新监测应用中的认知和思考[J]. 遥感技术与应用, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm Pan Zhuokun, Hu Yueming, Wang Guangxing, et al. Cognitions and Perspectives in Remote Sensing of Urban Renewal Monitoring[J]. Remote Sensing Technology and Application, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm [4] McCarthy J, Minsky M, Rochester N, et al. A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955[J]. AI Mag, 2006, 27: 12-14 [5] Rosenblatt F. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain[J]. Psychological Review, 1958, 65(6): 386-408 doi:  10.1037/h0042519 [6] 张钹, 朱军, 苏航. 迈向第三代人工智能[J]. 中国科学: 信息科学, 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm Zhang Bo, Zhu Jun, Su Hang. Toward the Third Generation of Artificial Intelligence[J]. Scientia Sinica(Informationis), 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm [7] Goodenough D G, Goldberg M, Plunkett G, et al. An Expert System for Remote Sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 349-359 [8] Matsuyama T. Knowledge-Based Aerial Image Understanding Systems and Expert Systems for Image Processing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 305-316 [9] 许珺, 裴韬, 姚永慧. 地学知识图谱的定义、内涵和表达方式的探讨[J]. 地球信息科学学报, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm Xu Jun, Pei Tao, Yao Yonghui. Conceptual Framework and Representation of Geographic Knowledge Map[J]. Journal of Geo-Information Science, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm [10] 王志华, 杨晓梅, 周成虎. 面向遥感大数据的地学知识图谱构想[J]. 地球信息科学学报, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm Wang Zhihua, Yang Xiaomei, Zhou Chenghu. Geographic Knowledge Graph for Remote Sensing Big Data[J]. Journal of Geo-Information Science, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm [11] 张雪英, 张春菊, 吴明光, 等. 顾及时空特征的地理知识图谱构建方法[J]. 中国科学: 信息科学, 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm Zhang Xueying, Zhang Chunju, Wu Mingguang, et al. Spatiotemporal Features Based Geographical Knowledge Graph Construction[J]. Scientia Sinica (Informationis), 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm [12] Alirezaie M, Längkvist M, Sioutis M, et al. Semantic Referee: A Neural-Symbolic Framework for Enhancing Geospatial Semantic Segmentation[J]. Semantic Web, 2019, 10(5): 863-880 [13] Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[J]. CoRR, 2013, DOI:  1310.4546 [14] Bordes A, Usunier N, Garcia-Duran A, et al. Translating Embeddings for Modeling Multi-relational Data[C]//Neural Information Processing Systems(NIPS), Carson City, USA, 2013 [15] Chen Y W, Zhang S L, Peng X, et al. A Collaborative Ontology Construction Tool with Conflicts Detection[C]//The 4th International Conference on Semantics, Knowledge and Grid, Beijing, China, 2008 [16] 杜清运, 任福. 空间信息的自然语言表达模型[J]. 武汉大学学报·信息科学版, 2014, 39(6): 682-688 Du Qingyun, Ren Fu. Representation Model of Spatial Information in Natural Language[J]. Geomatics and Information Science of Wuhan University, 2014, 39(6): 682-688 [17] Chen J Y, Jimenez-Ruiz E, Horrocks I, et al. Learning Semantic Annotations for Tabular Data[C]/The 28th International Joint Conference on Artificial Intelligence, Macao, China, 2019 [18] Chen T, Xu R F, He Y L, et al. Improving Sentiment Analysis via Sentence Type Classification Using BiLSTM-CRF and CNN[J]. Expert Systems with Applications, 2017, 72: 221-230 [19] Haklay M, Weber P. OpenStreetMap: User-Generated Street Maps[J]. IEEE Pervasive Computing, 2008, 7(4): 12-18 [20] Tempelmeier N, Demidova E. Linking OpenStreetMap with Knowledge Graphs—Link Discovery for Schema-Agnostic Volunteered Geographic Information[J]. Future Generation Computer Systems, 2021, 116: 349-364 [21] Yang X, Liu Q Q, Yan J C, et al. R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object[J]. arXiv, 2019, DOI: 1908.05612 [22] Sun Z Q, Hu W, Zhang Q H, et al. Bootstrapping Entity Alignment with Knowledge Graph Embedding[C]//The 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018 [23] Zhang Z Q, Cai J Y, Zhang Y D, et al. Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction[C]//The AAAI Conference on Artificial Intelligence, New York, USA, 2020 [24] 李彦胜, 孔德宇, 张永军, 等. 联合稳健跨域映射和渐进语义基准修正的零样本遥感影像场景分类[J]. 测绘学报, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm Li Yansheng, Kong Deyu, Zhang Yongjun, et al. Zero-Shot Remote Sensing Image Scene Classification Based on Robust Cross-Domain Mapping and Gradual Refinement of Semantic Space[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm [25] Li Y S, Kong D Y, Zhang Y J, et al. Robust Deep Alignment Network with Remote Sensing Knowledge Graph for Zero-Shot and Generalized Zero-Shot Remote Sensing Image Scene Classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 179: 145-158 [26] Wang Z, Zhang J, Feng J, et al. Knowledge Graph Embedding by Translating on Hyperplanes[C]//AAAI Conference on Artificial Intelligence, Québec City, Canada, 2014 [27] Kodirov E, Xiang T, Gong S G. Semantic Autoencoder for Zero-Shot Learning[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017 [28] Li Y N, Wang D H, Hu H H, et al. Zero-Shot Recognition Using Dual Visual-Semantic Mapping Paths[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017 [29] Elhoseiny M, Elfeki M. Creativity Inspired Zero-Shot Learning[C]//IEEE/CVF International Conference on Computer Vision(ICCV), Seoul, Korea(South), 2019 [30] Schönfeld E, Ebrahimi S, Sinha S, et al. Generalized Zero and Few-Shot Learning via Aligned Variational Autoencoders[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, 2019 [31] Tong X Y, Xia G S, Lu Q K, et al. Land-Cover Classification with High-Resolution Remote Sensing Images Using Transferable Deep Models[J]. Remote Sensing of Environment, 2020, 237: 111322 [32] Li Y S, Ouyang S, Zhang Y J. Collaboratively Boosting Data-Driven Deep Learning and Knowledge-Guided Ontological Reasoning for Semantic Segmentation of Remote Sensing Imagery[J]. arXiv, 2020, DOI: 2010.02451 [33] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation[M]//Cham: Springer International Publishing, 2015 [34] Ma L, Liu Y, Zhang X L, et al. Deep Learning in Remote Sensing Applications: A Meta-Analysis and Review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152: 166-177 [35] 李德仁. 脑认知与空间认知: 论空间大数据与人工智能的集成[J]. 武汉大学学报·信息科学版, 2018, 43(12): 1761-1767 Li Deren. Brain Cognition and Spatial Cognition: On Integration of Geo-spatial Big Data and Artificial Intelligence[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1761-1767 [36] Zellers R, Yatskar M, Thomson S, et al. Neural Motifs: Scene Graph Parsing with Global Context[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018 [37] Zhu Y H, Jiang S Q, Li X Y. Visual Relationship Detection with Object Spatial Distribution[C]//IEEE International Conference on Multimedia and Expo, Hong Kong, China, 2017
###### 通讯作者: 陈斌, bchen63@163.com
• 1.

沈阳化工大学材料科学与工程学院 沈阳 110142

Figures(11)  / Tables(3)

## A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning

##### doi: 10.13203/j.whugis20210652
###### 1. School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China
Funds:

The National Natural Science Foundation of China 42030102

The National Natural Science Foundation of China 41971284

Innovative Research Groups of the Natural Science Foundation of Hubei Province 2020CFA003

• Author Bio:

• ###### Corresponding author:ZHANG Yongjun, PhD, professor.E-mail: zhangyj@whu.edu.cn

Abstract:   Objectives  In the remote sensing (RS) big data era, intelligent interpretation of remote sensing images (RSI) is the key technology to mine the value of big RS data and promote several important applications. Traditional knowledge-driven RS interpretation methods, represented by expert systems, are highly interpretable, but generally show poor performance due to the interpretation knowledge being difficult to be completely and accurately expressed. With the development of deep learning in computer vision and other fields, it has gradually become the mainstream technology of RSI interpretation. However, the deep learning technique still has some fatal flaws in the RS field, such as poor interpretability and weak generalization ability. In order to overcome these problems, how to effectively combine knowledge inference and data learning has become an important research trend in the field of RS big data intelligent processing. Generally, knowledge inference relies on a strong domain knowledge base, but the research on RS knowledge graph (RS-KG) is very scarce and there is no available large-scale KG database for RSI interpretation now.  Methods  To overcome the above considerations, this paper focuses on the construction and evolution of the RS-KG for RSI interpretation and establishes the RS-KG takes into account the RS imaging mechanism and geographic knowledge. Supported by KG in the RS field, this paper takes three typical RSI interpretation tasks, namely, zero-shot RSI scene classification, interpretable RSI semantic segmentation, and large-scale RSI scene graph generation, as examples, to discuss the performance of the novel generation RSI interpretation paradigm which couples KG and deep learning.  Results and Conclusions  A large number of experimental results show that the combination of RS-KG inference and deep data learning can effectively improve the performance of RSI interpretation.The introduction of RS-KG can effectively improve the interpretation accuracy, generalization ability, anti-interference ability, and interpretability of deep learning models. These advantages make RS-KG promising in the novel generation RSI interpretation paradigm.

LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190. doi: 10.13203/j.whugis20210652
 Citation: LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190.
• 随着遥感科学、航空航天、导航通信等领域技术的飞速发展，遥感大数据时代已来临[1-2]。海量遥感影像数据的自动精确解译是一项十分基础且重要的工作，尽管国内外研究人员提出了大量遥感影像解译方法，遥感影像自动解译技术还远远不能满足行业单位的使用需求，亟需研究更加精准、可靠、智能的遥感影像解译方法[3]

遥感影像解译方法随着人工智能技术的发展变化而不断演进。从发展历程来看，人工智能技术主要经历了三大阶段，即以符号主义为主要特点的第一代人工智能[4]，以联结主义为主要特点的第二代人工智能[5]和以知识推理与数据学习联合为主要特点的第三代人工智能[6]。第一代和第二代人工智能技术分别为单方面基于知识层面和数据层面模拟人类的智能行为，因此存在各自的局限性。在数据大爆炸背景下，如何通过知识推理与数据学习互补发展促进人工智能技术的提升，是人工智能研究与应用中亟待解决的重大数理基础问题之一。尽管第三代人工智能具有重大提升潜力，但是知识推理与数据学习耦合难度很大，尚处在萌芽期。为了推动该技术方向的发展，2021年国际上专门成立了知识推理与数据学习联合国际会议。

在遥感信息领域，现有的遥感影像解译方法可分为知识驱动方法和数据驱动方法两大类。知识驱动的遥感影像解译方法是在第一代人工智能技术的基础上发展起来的，它能利用遥感解译专家在解译过程中的决策知识[7]或者人眼视觉识别机制[8]，借助先验知识推理完成遥感影像解译。总体来说，现有的知识驱动的遥感影像解译方法只是对遥感专家决策过程和解译规则的简单模仿，通过对静态知识的应用实现遥感影像的解译工作，难以充分利用复杂的遥感成像机理、遥感影像附载的地学知识。第三代人工智能技术的兴起给遥感影像智能解译提供了方向，即将知识推理和数据学习结合起来，建立一个以数据自主学习为核心，以领域先验知识推理为引导的新一代遥感影像解译范式。

一般来说，知识推理高度依赖大规模领域知识库，然而目前还不存在面向遥感影像解译的大规模知识库。在知识工程领域，知识图谱[9]能以结构化的形式描述客观世界的概念、实体及其之间的关系，有希望将现实世界中的遥感信息表达为更接近人类认知的形式，更好地组织和管理专家先验知识、遥感成像机理、遥感影像附载的地理学知识。在遥感领域知识图谱基础上，如何通过知识图谱推理与深度数据学习互补发展促进遥感影像智能解译性能的提升，实现数据-信息-知识的智能转换，成为亟待解决的重要科学问题[10-12]

基于上述考虑，本文研究了面向遥感影像解译的遥感知识图谱构建与进化方法，在遥感领域知识图谱基础上构建了联合知识图谱和深度学习的新一代遥感影像智能解译范式，主要包括：（1）基于知识图谱表示学习的零样本遥感影像场景分类；（2）联合知识推理和深度学习的可解释遥感影像语义分割；（3）知识图谱引导的大幅面遥感影像场景图生成。

Reference (37)

/