LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190. DOI: 10.13203/j.whugis20210652
Citation: LI Yansheng, ZHANG Yongjun. A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1176-1190. DOI: 10.13203/j.whugis20210652

A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning

Funds: 

The National Natural Science Foundation of China 42030102

The National Natural Science Foundation of China 41971284

Innovative Research Groups of the Natural Science Foundation of Hubei Province 2020CFA003

More Information
  • Author Bio:

    LI Yansheng, PhD, associator professor, majors in intelligent mining of remote sensing big data. E-mail: yansheng.li@whu.edu.cn

  • Corresponding author:

    ZHANG Yongjun, PhD, professor.E-mail: zhangyj@whu.edu.cn

  • Received Date: November 27, 2021
  • Available Online: January 19, 2022
  • Published Date: August 04, 2022
  •   Objectives  In the remote sensing (RS) big data era, intelligent interpretation of remote sensing images (RSI) is the key technology to mine the value of big RS data and promote several important applications. Traditional knowledge-driven RS interpretation methods, represented by expert systems, are highly interpretable, but generally show poor performance due to the interpretation knowledge being difficult to be completely and accurately expressed. With the development of deep learning in computer vision and other fields, it has gradually become the mainstream technology of RSI interpretation. However, the deep learning technique still has some fatal flaws in the RS field, such as poor interpretability and weak generalization ability. In order to overcome these problems, how to effectively combine knowledge inference and data learning has become an important research trend in the field of RS big data intelligent processing. Generally, knowledge inference relies on a strong domain knowledge base, but the research on RS knowledge graph (RS-KG) is very scarce and there is no available large-scale KG database for RSI interpretation now.
      Methods  To overcome the above considerations, this paper focuses on the construction and evolution of the RS-KG for RSI interpretation and establishes the RS-KG takes into account the RS imaging mechanism and geographic knowledge. Supported by KG in the RS field, this paper takes three typical RSI interpretation tasks, namely, zero-shot RSI scene classification, interpretable RSI semantic segmentation, and large-scale RSI scene graph generation, as examples, to discuss the performance of the novel generation RSI interpretation paradigm which couples KG and deep learning.
      Results and Conclusions  A large number of experimental results show that the combination of RS-KG inference and deep data learning can effectively improve the performance of RSI interpretation.The introduction of RS-KG can effectively improve the interpretation accuracy, generalization ability, anti-interference ability, and interpretability of deep learning models. These advantages make RS-KG promising in the novel generation RSI interpretation paradigm.
  • [1]
    李德仁, 张良培, 夏桂松. 遥感大数据自动分析与数据挖掘[J]. 测绘学报, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm

    Li Deren, Zhang Liangpei, Xia Guisong. Automatic Analysis and Mining of Remote Sensing Big Data[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm
    [2]
    付琨, 孙显, 仇晓兰, 等. 遥感大数据条件下多星一体化处理与分析[J]. 遥感学报, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm

    Fu Kun, Sun Xian, Qiu Xiaolan, et al. Multi-Satellite Integrated Processing and Analysis Method Under Remote Sensing Big Data[J]. National Remote Sensing Bulletin, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm
    [3]
    潘灼坤, 胡月明, 王广兴, 等. 对遥感在城市更新监测应用中的认知和思考[J]. 遥感技术与应用, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm

    Pan Zhuokun, Hu Yueming, Wang Guangxing, et al. Cognitions and Perspectives in Remote Sensing of Urban Renewal Monitoring[J]. Remote Sensing Technology and Application, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm
    [4]
    McCarthy J, Minsky M, Rochester N, et al. A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955[J]. AI Mag, 2006, 27: 12-14
    [5]
    Rosenblatt F. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain[J]. Psychological Review, 1958, 65(6): 386-408 doi: 10.1037/h0042519
    [6]
    张钹, 朱军, 苏航. 迈向第三代人工智能[J]. 中国科学: 信息科学, 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm

    Zhang Bo, Zhu Jun, Su Hang. Toward the Third Generation of Artificial Intelligence[J]. Scientia Sinica(Informationis), 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm
    [7]
    Goodenough D G, Goldberg M, Plunkett G, et al. An Expert System for Remote Sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 349-359 doi: 10.1109/TGRS.1987.289805
    [8]
    Matsuyama T. Knowledge-Based Aerial Image Understanding Systems and Expert Systems for Image Processing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 305-316 doi: 10.1109/TGRS.1987.289802
    [9]
    许珺, 裴韬, 姚永慧. 地学知识图谱的定义、内涵和表达方式的探讨[J]. 地球信息科学学报, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm

    Xu Jun, Pei Tao, Yao Yonghui. Conceptual Framework and Representation of Geographic Knowledge Map[J]. Journal of Geo-Information Science, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm
    [10]
    王志华, 杨晓梅, 周成虎. 面向遥感大数据的地学知识图谱构想[J]. 地球信息科学学报, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm

    Wang Zhihua, Yang Xiaomei, Zhou Chenghu. Geographic Knowledge Graph for Remote Sensing Big Data[J]. Journal of Geo-Information Science, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm
    [11]
    张雪英, 张春菊, 吴明光, 等. 顾及时空特征的地理知识图谱构建方法[J]. 中国科学: 信息科学, 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm

    Zhang Xueying, Zhang Chunju, Wu Mingguang, et al. Spatiotemporal Features Based Geographical Knowledge Graph Construction[J]. Scientia Sinica (Informationis), 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm
    [12]
    Alirezaie M, Längkvist M, Sioutis M, et al. Semantic Referee: A Neural-Symbolic Framework for Enhancing Geospatial Semantic Segmentation[J]. Semantic Web, 2019, 10(5): 863-880 doi: 10.3233/SW-190362
    [13]
    Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[J]. CoRR, 2013, DOI: 1310.4546
    [14]
    Bordes A, Usunier N, Garcia-Duran A, et al. Translating Embeddings for Modeling Multi-relational Data[C]//Neural Information Processing Systems(NIPS), Carson City, USA, 2013
    [15]
    Chen Y W, Zhang S L, Peng X, et al. A Collaborative Ontology Construction Tool with Conflicts Detection[C]//The 4th International Conference on Semantics, Knowledge and Grid, Beijing, China, 2008
    [16]
    杜清运, 任福. 空间信息的自然语言表达模型[J]. 武汉大学学报·信息科学版, 2014, 39(6): 682-688 doi: 10.13203/j.whugis20140118

    Du Qingyun, Ren Fu. Representation Model of Spatial Information in Natural Language[J]. Geomatics and Information Science of Wuhan University, 2014, 39(6): 682-688 doi: 10.13203/j.whugis20140118
    [17]
    Chen J Y, Jimenez-Ruiz E, Horrocks I, et al. Learning Semantic Annotations for Tabular Data[C]/The 28th International Joint Conference on Artificial Intelligence, Macao, China, 2019
    [18]
    Chen T, Xu R F, He Y L, et al. Improving Sentiment Analysis via Sentence Type Classification Using BiLSTM-CRF and CNN[J]. Expert Systems with Applications, 2017, 72: 221-230 doi: 10.1016/j.eswa.2016.10.065
    [19]
    Haklay M, Weber P. OpenStreetMap: User-Generated Street Maps[J]. IEEE Pervasive Computing, 2008, 7(4): 12-18 doi: 10.1109/MPRV.2008.80
    [20]
    Tempelmeier N, Demidova E. Linking OpenStreetMap with Knowledge Graphs—Link Discovery for Schema-Agnostic Volunteered Geographic Information[J]. Future Generation Computer Systems, 2021, 116: 349-364 doi: 10.1016/j.future.2020.11.003
    [21]
    Yang X, Liu Q Q, Yan J C, et al. R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object[J]. arXiv, 2019, DOI: 1908.05612
    [22]
    Sun Z Q, Hu W, Zhang Q H, et al. Bootstrapping Entity Alignment with Knowledge Graph Embedding[C]//The 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018
    [23]
    Zhang Z Q, Cai J Y, Zhang Y D, et al. Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction[C]//The AAAI Conference on Artificial Intelligence, New York, USA, 2020
    [24]
    李彦胜, 孔德宇, 张永军, 等. 联合稳健跨域映射和渐进语义基准修正的零样本遥感影像场景分类[J]. 测绘学报, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm

    Li Yansheng, Kong Deyu, Zhang Yongjun, et al. Zero-Shot Remote Sensing Image Scene Classification Based on Robust Cross-Domain Mapping and Gradual Refinement of Semantic Space[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm
    [25]
    Li Y S, Kong D Y, Zhang Y J, et al. Robust Deep Alignment Network with Remote Sensing Knowledge Graph for Zero-Shot and Generalized Zero-Shot Remote Sensing Image Scene Classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 179: 145-158 doi: 10.1016/j.isprsjprs.2021.08.001
    [26]
    Wang Z, Zhang J, Feng J, et al. Knowledge Graph Embedding by Translating on Hyperplanes[C]//AAAI Conference on Artificial Intelligence, Québec City, Canada, 2014
    [27]
    Kodirov E, Xiang T, Gong S G. Semantic Autoencoder for Zero-Shot Learning[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017
    [28]
    Li Y N, Wang D H, Hu H H, et al. Zero-Shot Recognition Using Dual Visual-Semantic Mapping Paths[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017
    [29]
    Elhoseiny M, Elfeki M. Creativity Inspired Zero-Shot Learning[C]//IEEE/CVF International Conference on Computer Vision(ICCV), Seoul, Korea(South), 2019
    [30]
    Schönfeld E, Ebrahimi S, Sinha S, et al. Generalized Zero and Few-Shot Learning via Aligned Variational Autoencoders[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, 2019
    [31]
    Tong X Y, Xia G S, Lu Q K, et al. Land-Cover Classification with High-Resolution Remote Sensing Images Using Transferable Deep Models[J]. Remote Sensing of Environment, 2020, 237: 111322 doi: 10.1016/j.rse.2019.111322
    [32]
    Li Y S, Ouyang S, Zhang Y J. Collaboratively Boosting Data-Driven Deep Learning and Knowledge-Guided Ontological Reasoning for Semantic Segmentation of Remote Sensing Imagery[J]. arXiv, 2020, DOI: 2010.02451
    [33]
    Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation[M]//Cham: Springer International Publishing, 2015
    [34]
    Ma L, Liu Y, Zhang X L, et al. Deep Learning in Remote Sensing Applications: A Meta-Analysis and Review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152: 166-177 doi: 10.1016/j.isprsjprs.2019.04.015
    [35]
    李德仁. 脑认知与空间认知: 论空间大数据与人工智能的集成[J]. 武汉大学学报·信息科学版, 2018, 43(12): 1761-1767 doi: 10.13203/j.whugis20180411

    Li Deren. Brain Cognition and Spatial Cognition: On Integration of Geo-spatial Big Data and Artificial Intelligence[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1761-1767 doi: 10.13203/j.whugis20180411
    [36]
    Zellers R, Yatskar M, Thomson S, et al. Neural Motifs: Scene Graph Parsing with Global Context[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018
    [37]
    Zhu Y H, Jiang S Q, Li X Y. Visual Relationship Detection with Object Spatial Distribution[C]//IEEE International Conference on Multimedia and Expo, Hong Kong, China, 2017

Catalog

    Article views (6917) PDF downloads (1476) Cited by()
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return