A New Paradigm of Remote Sensing Image Interpretation by Coupling Knowledge Graph and Deep Learning
-
摘要: 在遥感大数据时代，遥感影像智能解译是挖掘遥感大数据价值并推动若干重大应用的关键技术，如何将知识推理和数据学习两类解译方法有机联合已成为遥感大数据智能处理的重要研究趋势。由此提出了面向遥感影像解译的遥感领域知识图谱构建与进化方法，建立了顾及遥感成像机理和地理学知识的遥感领域知识图谱。在遥感领域知识图谱支撑下，以零样本遥感影像场景分类、可解释遥感影像语义分割以及大幅面遥感影像场景图生成3个典型的遥感影像解译任务为例，研究了耦合知识图谱和深度学习的新一代遥感影像解译范式。在零样本遥感影像场景分类实验中，所提方法在不同的可见类/不可见类比例和不同的语义表示下，都明显优于其他方法；在可解释遥感影像语义分割实验中，知识推理与深度学习的联合方法取得了最好的分类结果；在大幅面遥感影像场景图生成实验中，知识图谱引导的方法精度明显高于基准的频率统计方法。遥感知识图谱推理与深度数据学习的融合可以有效提升遥感影像的解译性能。Abstract:Objectives In the remote sensing (RS) big data era, intelligent interpretation of remote sensing images (RSI) is the key technology to mine the value of big RS data and promote several important applications. Traditional knowledge-driven RS interpretation methods, represented by expert systems, are highly interpretable, but generally show poor performance due to the interpretation knowledge being difficult to be completely and accurately expressed. With the development of deep learning in computer vision and other fields, it has gradually become the mainstream technology of RSI interpretation. However, the deep learning technique still has some fatal flaws in the RS field, such as poor interpretability and weak generalization ability. In order to overcome these problems, how to effectively combine knowledge inference and data learning has become an important research trend in the field of RS big data intelligent processing. Generally, knowledge inference relies on a strong domain knowledge base, but the research on RS knowledge graph (RS-KG) is very scarce and there is no available large-scale KG database for RSI interpretation now.Methods To overcome the above considerations, this paper focuses on the construction and evolution of the RS-KG for RSI interpretation and establishes the RS-KG takes into account the RS imaging mechanism and geographic knowledge. Supported by KG in the RS field, this paper takes three typical RSI interpretation tasks, namely, zero-shot RSI scene classification, interpretable RSI semantic segmentation, and large-scale RSI scene graph generation, as examples, to discuss the performance of the novel generation RSI interpretation paradigm which couples KG and deep learning.Results and Conclusions A large number of experimental results show that the combination of RS-KG inference and deep data learning can effectively improve the performance of RSI interpretation.The introduction of RS-KG can effectively improve the interpretation accuracy, generalization ability, anti-interference ability, and interpretability of deep learning models. These advantages make RS-KG promising in the novel generation RSI interpretation paradigm.
-
-
![]()
图 1 面向遥感影像智能解译的遥感领域知识图谱迭代建模框架
Figure 1. Iterative Modeling Framework of Remote Sensing Knowledge Graph for Intelligent Interpretation of Remote Sensing Image
![]()
图 4 时空约束下的知识图谱与遥感影像的关联解译
Figure 4. Association Interpretation of Knowledge Map and Remote Sensing Image Under the Constraints of Time and Space
![]()
图 5 基于多源地学数据的遥感领域知识三元组提取
Figure 5. Extraction of Triples of Remote Sensing Domain Knowledge Based on Multi-source Geoscience Data
![]()
图 10 可解释性遥感影像语义分割方法的总体流程图
Figure 10. Overall Flowchart of Explainable Remote Sensing Image Semantic Segmentation
![]()
图 11 知识图谱引导的大幅面遥感影像场景图生成流程
Figure 11. Flowchart of Knowledge Graph-Guided Scene Map Generation for Large-Size Remote Sensing Images
表 1 广义零样本分类任务中不同划分方式下不同方法的准确率对比/%
Table 1 Accuracy Comparison of Different Methods Under Different Partition Modes in Generalized Zero-Shot Classification Task/%
语义表示 可见类/不可见类 SAE [27] DMaP[28] CIZSL [29] CADA-VAE [30] DCA Word2Vec 60/10 27.97±1.13 28.88±1.26 25.18±0.86 32.88±2.54 34.09±1.34 50/20 20.99±1.90 20.33±1.13 15.70±0.86 30.25±3.07 31.44±1.66 40/30 17.15±0.55 16.78±1.10 9.10±1.32 26.06±0.79 25.63±0.26 BERT 60/10 28.57±0.94 26.57±0.65 25.00±1.25 36.34±2.03 37.96±1.65 50/20 21.52±1.38 19.52±1.42 14.95±1.51 31.51±2.27 31.45±1.85 40/30 16.65±0.40 16.31±1.24 8.57±0.57 27.05±0.79 28.15±1.16 Attribute 60/10 28.58±0.93 30.71±0.78 23.88±0.87 36.00±2.19 37.60±1.24 50/20 20.52±1.75 23.55±0.87 14.27±1.05 32.17±2.41 32.66±0.80 40/30 16.73±1.06 16.12±0.82 8.11±0.98 26.13±0.79 28.79±0.92 知识图谱 60/10 28.86±0.60 30.11±1.39 23.65±0.61 38.10±1.89 40.25±0.84 50/20 23.66±1.06 23.41±1.21 13.93±1.01 32.94±1.42 34.11±0.45 40/30 16.94±1.03 16.20±1.62 8.14±0.87 28.11±0.79 29.61±0.82 
下载: 导出CSV
表 3 遥感场景图生成方法的精度对比结果/%
Table 3 Accuracy Comparison Results of Remote Sensing Scene Map Generation Methods /%
下载: 导出CSV
-
[1] 李德仁, 张良培, 夏桂松. 遥感大数据自动分析与数据挖掘[J]. 测绘学报, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm Li Deren, Zhang Liangpei, Xia Guisong. Automatic Analysis and Mining of Remote Sensing Big Data[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43(12): 1211-1216 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201412002.htm
[2] 付琨, 孙显, 仇晓兰, 等. 遥感大数据条件下多星一体化处理与分析[J]. 遥感学报, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm Fu Kun, Sun Xian, Qiu Xiaolan, et al. Multi-Satellite Integrated Processing and Analysis Method Under Remote Sensing Big Data[J]. National Remote Sensing Bulletin, 2021, 25(3): 691-707 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB202103001.htm
[3] 潘灼坤, 胡月明, 王广兴, 等. 对遥感在城市更新监测应用中的认知和思考[J]. 遥感技术与应用, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm Pan Zhuokun, Hu Yueming, Wang Guangxing, et al. Cognitions and Perspectives in Remote Sensing of Urban Renewal Monitoring[J]. Remote Sensing Technology and Application, 2020, 35(4): 911-923 https://www.cnki.com.cn/Article/CJFDTOTAL-YGJS202004019.htm
[4] McCarthy J, Minsky M, Rochester N, et al. A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955[J]. AI Mag, 2006, 27: 12-14
[5] Rosenblatt F. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain[J]. Psychological Review, 1958, 65(6): 386-408 doi: 10.1037/h0042519
[6] 张钹, 朱军, 苏航. 迈向第三代人工智能[J]. 中国科学: 信息科学, 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm Zhang Bo, Zhu Jun, Su Hang. Toward the Third Generation of Artificial Intelligence[J]. Scientia Sinica(Informationis), 2020, 50(9): 1281-1302 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202009002.htm
[7] Goodenough D G, Goldberg M, Plunkett G, et al. An Expert System for Remote Sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 349-359 doi: 10.1109/TGRS.1987.289805
[8] Matsuyama T. Knowledge-Based Aerial Image Understanding Systems and Expert Systems for Image Processing[J]. IEEE Transactions on Geoscience and Remote Sensing, 1987, GE-25(3): 305-316 doi: 10.1109/TGRS.1987.289802
[9] 许珺, 裴韬, 姚永慧. 地学知识图谱的定义、内涵和表达方式的探讨[J]. 地球信息科学学报, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm Xu Jun, Pei Tao, Yao Yonghui. Conceptual Framework and Representation of Geographic Knowledge Map[J]. Journal of Geo-Information Science, 2010, 12(4): 496-502 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX201004009.htm
[10] 王志华, 杨晓梅, 周成虎. 面向遥感大数据的地学知识图谱构想[J]. 地球信息科学学报, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm Wang Zhihua, Yang Xiaomei, Zhou Chenghu. Geographic Knowledge Graph for Remote Sensing Big Data[J]. Journal of Geo-Information Science, 2021, 23(1): 16-28 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202101004.htm
[11] 张雪英, 张春菊, 吴明光, 等. 顾及时空特征的地理知识图谱构建方法[J]. 中国科学: 信息科学, 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm Zhang Xueying, Zhang Chunju, Wu Mingguang, et al. Spatiotemporal Features Based Geographical Knowledge Graph Construction[J]. Scientia Sinica (Informationis), 2020, 50(7): 1019-1032 https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202007005.htm
[12] Alirezaie M, Längkvist M, Sioutis M, et al. Semantic Referee: A Neural-Symbolic Framework for Enhancing Geospatial Semantic Segmentation[J]. Semantic Web, 2019, 10(5): 863-880 doi: 10.3233/SW-190362
[13] Mikolov T, Sutskever I, Chen K, et al. Distributed Representations of Words and Phrases and Their Compositionality[J]. CoRR, 2013, DOI: 1310.4546
[14] Bordes A, Usunier N, Garcia-Duran A, et al. Translating Embeddings for Modeling Multi-relational Data[C]//Neural Information Processing Systems(NIPS), Carson City, USA, 2013
[15] Chen Y W, Zhang S L, Peng X, et al. A Collaborative Ontology Construction Tool with Conflicts Detection[C]//The 4th International Conference on Semantics, Knowledge and Grid, Beijing, China, 2008
[16] 杜清运, 任福. 空间信息的自然语言表达模型[J]. 武汉大学学报·信息科学版, 2014, 39(6): 682-688 doi: 10.13203/j.whugis20140118 Du Qingyun, Ren Fu. Representation Model of Spatial Information in Natural Language[J]. Geomatics and Information Science of Wuhan University, 2014, 39(6): 682-688 doi: 10.13203/j.whugis20140118
[17] Chen J Y, Jimenez-Ruiz E, Horrocks I, et al. Learning Semantic Annotations for Tabular Data[C]/The 28th International Joint Conference on Artificial Intelligence, Macao, China, 2019
[18] Chen T, Xu R F, He Y L, et al. Improving Sentiment Analysis via Sentence Type Classification Using BiLSTM-CRF and CNN[J]. Expert Systems with Applications, 2017, 72: 221-230 doi: 10.1016/j.eswa.2016.10.065
[19] Haklay M, Weber P. OpenStreetMap: User-Generated Street Maps[J]. IEEE Pervasive Computing, 2008, 7(4): 12-18 doi: 10.1109/MPRV.2008.80
[20] Tempelmeier N, Demidova E. Linking OpenStreetMap with Knowledge Graphs—Link Discovery for Schema-Agnostic Volunteered Geographic Information[J]. Future Generation Computer Systems, 2021, 116: 349-364 doi: 10.1016/j.future.2020.11.003
[21] Yang X, Liu Q Q, Yan J C, et al. R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object[J]. arXiv, 2019, DOI: 1908.05612
[22] Sun Z Q, Hu W, Zhang Q H, et al. Bootstrapping Entity Alignment with Knowledge Graph Embedding[C]//The 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018
[23] Zhang Z Q, Cai J Y, Zhang Y D, et al. Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction[C]//The AAAI Conference on Artificial Intelligence, New York, USA, 2020
[24] 李彦胜, 孔德宇, 张永军, 等. 联合稳健跨域映射和渐进语义基准修正的零样本遥感影像场景分类[J]. 测绘学报, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm Li Yansheng, Kong Deyu, Zhang Yongjun, et al. Zero-Shot Remote Sensing Image Scene Classification Based on Robust Cross-Domain Mapping and Gradual Refinement of Semantic Space[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(12): 1564-1574 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012007.htm
[25] Li Y S, Kong D Y, Zhang Y J, et al. Robust Deep Alignment Network with Remote Sensing Knowledge Graph for Zero-Shot and Generalized Zero-Shot Remote Sensing Image Scene Classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 179: 145-158 doi: 10.1016/j.isprsjprs.2021.08.001
[26] Wang Z, Zhang J, Feng J, et al. Knowledge Graph Embedding by Translating on Hyperplanes[C]//AAAI Conference on Artificial Intelligence, Québec City, Canada, 2014
[27] Kodirov E, Xiang T, Gong S G. Semantic Autoencoder for Zero-Shot Learning[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017
[28] Li Y N, Wang D H, Hu H H, et al. Zero-Shot Recognition Using Dual Visual-Semantic Mapping Paths[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017
[29] Elhoseiny M, Elfeki M. Creativity Inspired Zero-Shot Learning[C]//IEEE/CVF International Conference on Computer Vision(ICCV), Seoul, Korea(South), 2019
[30] Schönfeld E, Ebrahimi S, Sinha S, et al. Generalized Zero and Few-Shot Learning via Aligned Variational Autoencoders[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, 2019
[31] Tong X Y, Xia G S, Lu Q K, et al. Land-Cover Classification with High-Resolution Remote Sensing Images Using Transferable Deep Models[J]. Remote Sensing of Environment, 2020, 237: 111322 doi: 10.1016/j.rse.2019.111322
[32] Li Y S, Ouyang S, Zhang Y J. Collaboratively Boosting Data-Driven Deep Learning and Knowledge-Guided Ontological Reasoning for Semantic Segmentation of Remote Sensing Imagery[J]. arXiv, 2020, DOI: 2010.02451
[33] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation[M]//Cham: Springer International Publishing, 2015
[34] Ma L, Liu Y, Zhang X L, et al. Deep Learning in Remote Sensing Applications: A Meta-Analysis and Review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152: 166-177 doi: 10.1016/j.isprsjprs.2019.04.015
[35] 李德仁. 脑认知与空间认知: 论空间大数据与人工智能的集成[J]. 武汉大学学报·信息科学版, 2018, 43(12): 1761-1767 doi: 10.13203/j.whugis20180411 Li Deren. Brain Cognition and Spatial Cognition: On Integration of Geo-spatial Big Data and Artificial Intelligence[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1761-1767 doi: 10.13203/j.whugis20180411
[36] Zellers R, Yatskar M, Thomson S, et al. Neural Motifs: Scene Graph Parsing with Global Context[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018
[37] Zhu Y H, Jiang S Q, Li X Y. Visual Relationship Detection with Object Spatial Distribution[C]//IEEE International Conference on Multimedia and Expo, Hong Kong, China, 2017
计量
- 文章访问数: 7005
- HTML全文浏览量: 1073
- PDF下载量: 1499
