Building Scene Structure Extraction Method for Urban Augmented Reality Annotation

XU Wang; YOU Xiong; ZHANG Weiwei; CHEN Bing; HU Zongmin

doi:10.13203/j.whugis20200373

Volume 48 Issue 6

Jun. 2023

Turn off MathJax

Article Contents

Abstract

References

Geomatics and Information Science of Wuhan University > 2023 > 48(6): 926-935. > DOI: 10.13203/j.whugis20200373

XU Wang, YOU Xiong, ZHANG Weiwei, CHEN Bing, HU Zongmin. Building Scene Structure Extraction Method for Urban Augmented Reality Annotation[J]. Geomatics and Information Science of Wuhan University, 2023, 48(6): 926-935. DOI: 10.13203/j.whugis20200373

Citation:

PDF (65770 KB)

Building Scene Structure Extraction Method for Urban Augmented Reality Annotation

1.
School of Geospatial Information, Information Engineering University, Zhengzhou 450001, China

More Information

Received Date: July 23, 2020
Available Online: July 21, 2022
Published Date: June 04, 2023

Graphical Abstract

Abstract

Abstract

Objectives In urban augmented reality (AR), the problems such as unclear indication, confusing occlusion and overlapping can be effectively solved by incorporating scene structure into information annotation. Aiming at the problem of missing scene structure in information annotation, a method for extracting building scene structure is proposed, which distinguishes geographical entities and also considers the accuracy, efficiency and robustness.
Methods First, a scene perception network for building scene structure extraction is constructed to extract semantic label, scene depth and surface normal from a single scene image. Second, structure features such as building facade corners and orientation are obtained by transforming previous results. Third, the best matching between structure features and the building outlines in 2D map is calculated. Finally, the scene image is reconstructed according to geographical entities and the structure information is generated, including region contours, scene depth and facades orientation.
Results Experiments are conducted with self-constructed and public datasets. The results show that the proposed method can extract the structure of building scene in 25-45 ms, and the facade contours are more regular. Despite the geo-registration errors or partial occlusion, the quality of facade extraction is significantly better than results from the map-based method.
Conclusions The proposed method can extract the structure of building scene in near real time with regular facade contours and good robustness, which is very useful for information annotation in urban AR.
- urban augmented reality,
- scene structure extraction,
- information annotation,
- scene perception,
- building facade

FullText(HTML)

References (26)

References

[1]	You X, Zhang W W, Ma M, et al. Survey on Urban Warfare Augmented Reality[J]. ISPRS International Journal of Geo-Information, 2018, 7(2): 46. doi: 10.3390/ijgi7020046
[2]	刘兵, 孟立秋. 扩展现实与地理空间认知研究进展与展望[J]. 武汉大学学报(信息科学版), 2022, 47(12): 2047-2053. doi: 10.13203/j.whugis20220759 Liu Bing, Meng Liqiu. Research Progress and Prospect of Extended Reality and Geospatial Cognition[J]. Geomatics and Information Science of Wuhan University, 2022, 47(12): 2047-2053. doi: 10.13203/j.whugis20220759
[3]	庞静, 陈国雄, 宋关福, 等. 增强现实地图研究与应用[J]. 测绘地理信息, 2021, 46(1): 130-134. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXG202101030.htm Pang Jing, Chen Guoxiong, Song Guanfu, et al. Research and Application of Augmented Reality Map[J]. Journal of Geomatics, 2021, 46(1): 130-134. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXG202101030.htm
[4]	阮舜毅, 康俊锋. 移动增强现实可视化下灾害场景加载的优化方法[J]. 武汉大学学报(信息科学版), 2020, 45(9): 1422-1428. doi: 10.13203/j.whugis20190477 Ruan Shunyi, Kang Junfeng. Optimization Method of Disaster Scene Loading Under Mobile Augmented Reality Visualization[J]. Geomatics and Information Science of Wuhan University, 2020, 45(9): 1422-1428. doi: 10.13203/j.whugis20190477
[5]	邓晨, 游雄, 张威巍. 面向城市分队作战的增强现实研究[J]. 指挥控制与仿真, 2019, 41(2): 1-10. doi: 10.3969/j.issn.1673-3819.2019.02.001 Deng Chen, You Xiong, Zhang Weiwei. Augmented Reality in Urban Operations for Unit Dismounted Warfighters[J]. Command Control & Simulation, 2019, 41(2): 1-10. doi: 10.3969/j.issn.1673-3819.2019.02.001
[6]	曲毅, 李爱光, 徐旺, 等. 基于位姿传感器的户外ARGIS注册技术[J]. 测绘科学技术学报, 2017, 34(1): 106-110. https://www.cnki.com.cn/Article/CJFDTOTAL-JFJC201701021.htm Qu Yi, Li Aiguang, Xu Wang, et al. Outdoor ARGIS Registration Techniques Based on Position-Posture Sensor[J]. Journal of Geomatics Science and Technology, 2017, 34(1): 106-110. https://www.cnki.com.cn/Article/CJFDTOTAL-JFJC201701021.htm
[7]	葛林, 庄晓斌, 华炜, 等. 面向城市增强现实的高融合度信息标注方法[J]. 系统仿真学报, 2014, 26(9): 2015-2022. https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ201409026.htm Ge Lin, Zhuang Xiaobin, Hua Wei, et al. High Fusion Information Annotation Method for City Augmented Reality[J]. Journal of System Simulation, 2014, 26(9): 2015-2022. https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ201409026.htm
[8]	Jia J Q, Elezovikj S, Fan H, et al. Semantic-Aware Label Placement for Augmented Reality in Street View[J]. The Visual Computer, 2021, 37(7): 1805-1819.
[9]	黄碧辉, 吴勇, 郑森源, 等. 一种改进的户外移动增强现实三维注册方法[J]. 武汉大学学报(信息科学版), 2019, 44(12): 1865-1873. doi: 10.13203/j.whugis20180098 Huang Bihui, Wu Yong, Zheng Senyuan, et al. An Improved Registration Method for Outdoor Mobile Augmented Reality[J]. Geomatics and Information Science of Wuhan University, 2019, 44(12): 1865-1873. doi: 10.13203/j.whugis20180098
[10]	Zollmann S, Poglitsch C, Ventura J. VISGIS: Dynamic Situated Visualization for Geographic Information Systems[C]//International Conference on Image and Vision Computing, Palmerston North, New Zealand, 2016.
[11]	Fond A, Berger M O, Simon G. Prior-based Facade Rectification for AR in Urban Environment[C]//IEEE International Symposium on Mixed and Augmented Reality Workshop, Fukuoka, Japan, 2015.
[12]	徐旺, 游雄, 张威巍, 等. 2D地图的建筑物场景结构提取方法及其在城市增强现实中的应用[J]. 测绘学报, 2020, 49(12): 1619-1629. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012013.htm Xu Wang, You Xiong, Zhang Weiwei, et al. Method of Building Scene Structure Extraction Based on 2D Map and Its Application in Urban Augmented Reality[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(12): 1619-1629. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202012013.htm
[13]	Nekrasov V, Dharmasiri T, Spek A, et al. Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations[C]// International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 2019.
[14]	董赛云, 刘钊, 李婕, 等. 深度学习融合局部聚合向量的增强现实标志物检索研究[J]. 测绘地理信息, 2022, 47(S1): 157-161. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXG2022S1035.htm Dong Saiyun, Liu Zhao, Li Jie, et al. Research on Augmented Reality Marker Retrieval Based on Deep Learning and Vector of Locally Aggregated Descriptors[J]. Journal of Geomatics, 2022, 47(S1): 157-161. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXG2022S1035.htm
[15]	Armagan A, Hirzer M, Roth P, et al. Accurate Camera Registration in Urban Environments Using High-level Feature Matching[C]//British Machine Vision Conference, London, UK, 2017.
[16]	Yang F T, Zhou Z H. Recovering 3D Planes from a Single Image via Convolutional Neural Network[C]//ECCV, Munich, Germany, 2018.
[17]	Liu C, Kim K, Gu J W, et al. PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image [C]//IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019.
[18]	Zeng Z L, Wu M Y, Zeng W, et al. Deep Recognition of Vanishing-Point-Constrained Building Planes in Urban Street Views[J]. IEEE Transactions on Image Processing, 2020, 29(1): 5912-5923.
[19]	邓晨, 游雄, 张威巍, 等. 基于2D地图的城市户外ARGIS视觉辅助地理配准技术[J]. 测绘学报, 2019, 48(10): 1305-1319. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201910012.htm Deng Chen, You Xiong, Zhang Weiwei, et al. A Vision-Aided Geo-Registration Method for Outdoor ARGIS in Urban Environments Based on 2D Maps[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(10): 1305-1319. https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201910012.htm
[20]	Chen B K, Gong C, Yang J. Importance-Aware Semantic Segmentation for Autonomous Vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(1): 137-148.
[21]	Armagan A, Hirzer M, Roth P M, et al. Learning to Align Semantic Segmentation and 2.5D Maps for Geolocalization[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017.
[22]	Yu C Q, Wang J B, Peng C, et al. BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation[C]// ECCV, Munich, Germany, 2018.
[23]	Yu J, Zeng P, Yu Y Y, et al. A Combined Convolutional Neural Network for Urban Land-Use Classification with GIS Data[J]. Remote Sensing, 2022, 14(5): 1128.
[24]	周东波, 秦政, 陈言言, 等. 通视受限下视锥剖分的移动增强现实注册策略[J]. 武汉大学学报(信息科学版), 2018, 43(8): 1178-1184. doi: 10.13203/j.whugis20170097 Zhou Dongbo, Qin Zheng, Chen Yanyan, et al. Registration Strategy for Mobile Augmented Reality Based on Cone-View Partition Under the Restricted Visibility Environment[J]. Geomatics and Information Science of Wuhan University, 2018, 43(8): 1178-1184. doi: 10.13203/j.whugis20170097
[25]	Kirillov A, He K M, Girshick R, et al. Panoptic segmentation[C]// IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019.
[26]	Armagan A, Hirzer M, Lepetit V. Semantic segmentation for 3D Localization in Urban Environments[C]// Joint Urban Remote Sensing Event (JURSE), Dubai, United Arab Emirates, 2017.

Cited By

Cited by

Periodical cited type(1)

高贤君，冉树浩，张广斌，杨元维. 基于多特征融合与对象边界联合约束网络的建筑物提取. 武汉大学学报(信息科学版). 2024(03): 355-365 .

Other cited types(0)

Get Citation

PDF

XML

Article views (782) PDF downloads (138) Cited by(1)

Building Scene Structure Extraction Method for Urban Augmented Reality Annotation

Abstract

References

Cited by

Periodical cited type(1)

Other cited types(0)

Catalog

Related

Building Scene Structure Extraction Method for Urban Augmented Reality Annotation

Abstract

References

Cited by

Periodical cited type(1)

Other cited types(0)

Catalog

Related

Export File

Citation

Format

Content