文本模态辅助引导的壁画修复算法

陈永; 杜婉君; 张世龙

doi:10.13203/j.whugis20240251

文本模态辅助引导的壁画修复算法

陈永^1,2, ,,
杜婉君¹,
张世龙¹

1 兰州交通大学电子与信息工程学院, 甘肃兰州 730070;
2 甘肃省人工智能与图形图像处理工程研究中心, 甘肃兰州 730070

基金项目:

教育部人文社会科学研究青年基金资助项目(19YJC760012)

兰州交通大学基础研究拔尖人才项目(2022JC36)

兰州交通大学重点研发项目(ZDYF2304)

详细信息

作者简介:
陈永，博士，教授，研究方向为图像处理、计算机视觉。edukeylab@126.com

通讯作者:
陈永，博士，教授。edukeylab@126.com

计量
- 文章访问数: 4
- HTML全文浏览量: 3
- PDF下载量: 1
出版历程
- 收稿日期: 2025-01-27

Text Modality Assisted Guided Mural Inpainting Algorithm

1 School of Electronic and Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, China;
2 Gansu Provincial Engineering Research Center for Artificial Intelligence and Graphics & Image Processing, Lanzhou 730070, China

摘要

摘要: 针对现有深度学习算法在壁画修复时，仅考虑壁画图像先验信息，缺乏文本信息引导壁画修复，导致修复结果出现语义不一致，细节缺失等问题，提出了一种基于文本模态辅助引导的壁画修复算法。首先，提出一个文本引导的壁画修复网络，利用文本信息作为壁画修复的辅助控制引导，为壁画图像提供上下文引导修复信息。其次，构建文本语义过滤模块，利用掩膜图像和互补图像过滤得到破损区域的文本特征，并设计了语义增强模块，对过滤后的文本特征进行增强，提高文本语义与图像语义的一致性。然后，设计上采样纹理细节修复网络，实现对壁画纹理细节的修复。最后，采用谱归一化判别器博弈对抗，完成修复。通过对真实敦煌壁画的修复实验表明，所提方法能够有效完成破损壁画的修复，在主客观评价方面均优于比较算法。
- 壁画修复 /
- 文本引导 /
- 跨模态修复 /
- 语义过滤 /
- 语义增强
Abstract: Objectives: In order to solve the problems of existing deep learning algorithms in the inpainting of murals, which only consider the prior information of mural images to guide the inpainting of damaged areas, and lack of text information to guide the inpainting of murals, resulting in semantic inconsistency and lack of details in the inpainting results, a text modality-assisted guided mural inpainting algorithm is proposed. Methods: First, a text modality-assisted guided mural inpainting network is proposed, which uses the text information as the control guidance for mural inpainting, and provides context-guided repair information for mural images. Secondly, a text filtering module is constructed to obtain the text features of the damaged area by filtering the mask image and the complementary image, and a cross-modal semantic enhancement module is designed to enhance the filtered text features to improve the consistency between text semantics and image semantics. Then, the upsampled texture detail inpainting network is designed to achieve bidirectional fusion of shallow and deep features to obtain mural images with fine-grained features. Finally, the normalized discriminator is used to match the recovered murals to the game, and the recovered murals are obtained. Results: The experimental results of real mural restoration show that this method can effectively repair damaged murals, and is superior to comparative algorithms in subjective and objective evaluation. Conclusions: The proposed method can effectively repair damaged murals, achieving better visual perception and coordination.
- mural inpainting /
- text guide /
- cross-modal inpainting /
- semantic filtering /
- semantic enhancement