Objectives In the process of flood disaster, the observed data is multi-source heterogeneous (remote sensing image, social media text, geographic information data, etc.), which makes it difficult to use complementary advantages to fuse and apply to risk assessment and provide decision-making knowledge. We study the construction method of flood disaster knowledge graph based on multi-modal data. Remote sensing images and social media text knowledge are integrated and extracted to form multi-modal flood disaster knowledge graph.

Methods Based on the top-down method, the domain concept is subdivided and the flood disaster domain ontology layer is constructed. Remote sensing images are intelligently interpreted by deep residual full convolutional neural network, and the image interpretation information is converted into text by geographic inverse coding to realize transformation from image information to text knowledge. Knowledge is extracted from social media textual data based on naming entity recognition and relationship extraction technique. By training word vectors, the semantic similarity is used to calculate the correlation between text knowledge and image knowledge, so as to achieve the unified expression of multi-modal data knowledge.

Results Taking the flood disaster in Hubei Province, China as an example, the multi-source heterogeneous data are efficiently transformed into knowledge and correlated to form domain knowledge graph, which realizes the transformation from multi-source heterogeneous data to multi-modal knowledge, provides corresponding emergency measures in different periods of disasters, and realizes flood disaster assessment in Hubei Province by associating agricultural disaster area, crop type and crop value.

Conclusions This method combines deep remote sensing interpretation, text knowledge extraction and semantic similarity calculation to realize transformation from multi-source heterogeneous data to multi-modal knowledge.