Street-Facing Architectural Image Mapping and Architectural Style Map Generation Method Using Street View Images
-
摘要: 精细化的城市建筑风格地图已成为古建筑保护、城市规划、旅游资源开发的重要参考依据。但城市建筑众多，信息采集困难，仅靠人工难以实现成图，因此提出了面向街景影像建筑区域匹配的建筑风格地图生成方法。首先，在提取特征建筑风格影像的基础上，结合球形全景影像的空间几何约束和图像特征，通过匹配同名建筑区域构建双像建筑区域点位映射；然后，利用街景采集点到建筑俯视轮廓的方位范围，提出单像建筑区域方位映射，建立街景建筑区域与单体建筑俯视轮廓的空间匹配关系；最后，综合判定各单体建筑的风格属性，生成精细尺度的建筑风格地图。实验结果表明，基于单、双像位置映射的建筑区域匹配正确率分别达80.3%和85.1%，且19类建筑风格地图的分类精确率为55.1%，召回率为76.4%，在一定程度上能反映大范围的城市建筑风格的地理分布特征。Abstract:Objectives Each region has specific characteristics of architectural styles, and a detailed investigation of the geographical distribution of architectural styles is conducive to the protection of historic buildings, the development of special tourism resources and the scientific planning of urban architectural areas. However, the number of urban buildings is large, manual collection and investigation cannot meet the needs of large-scale operations. In recent years, Google and other Internet companies have launched street view images. Street view images are high resolution, containing a full range of urban street views as well as precise location and posture information, which provide a possibility to explore the geographic distribution of urban architectural styles. Therefore, we use deep learning to identify and match the styles of street view building areas, and establish a mapping relationship between the building area images and building outlines, so as to construct the generation method of a large-scale urban architectural style map in detail.Methods The style identification and map matching of architectural areas in street view images are the key and difficulty in generating urban architectural style maps.Firstly, we extract the building area images of various styles through Faster R-CNN. In order to establish the mapping relationship between building area images and single building outlines, we construct a building location mapping method by matching the same name building area in two adjacent street view images, then the building can be located by forward intersection. Secondly, for the single building image without a same-name area, we also propose a building azimuth mapping method, which combines the spatial azimuth relationship between the street view building area and building outlines in a digital map. The intersection of union (IoU) of the single building image azimuth range and the building outline azimuth range can help match the building area in a street view image and building outlines in a digital map. Finally, Technique for order preference by similarity to an ideal solution is used to determine the unique style attribute of each map building outline to solve the multiple mapping problem of a single building and generate a fine-grained architectural style map.Results The experimental results of the proposed method are as follows: (1) The average accuracy of Faster R-CNN detection of 19 types of architectural style areas on the test set is 73.81%. (2) The accuracy of matching two adjacent street images with the same name architectural area is 86.1%, the recall is 90.3%, and the average time to match an architectural region pair is 180.1 ms, which is 25.4% less than the time using SURF(speeded up robust features) under spherical epipolar geometry constraint and an accuracy improvement of 19.4%; (3) The accuracy of a building location mapping method is 85.1%, the mapping success rate is only 49.33%, and the average time for two corresponding building area to complete location mapping is 2.741 s; the accuracy of the building azimuth mapping method is 80.3%, the mapping success rate is 88.0%, and the average time for a single building area to complete azimuth mapping is 0.017 s. (4) In the test region, the building azimuth mapping method is more likely to cause multiple mapping problems, with 42.9% of the building outlines matching to multiple building images compared to 23.4% for the building location mapping method. (5) By verifying the style attributes of 331 building outlines in a digital map, we obtain a mean classification accuracy of 55.1%, a mean recall of 76.4%, and a mean F1 score of 0.601 for the architectural style maps.Conclusions Under the two architectural area mapping methods, the generation time of architectural style maps is short, and the F1 score of classification is 0.601, which can basically reflect the geographic distribution characteristics of a large range of urban architectural styles. In addition, the regional and similarity of architectural styles is the main reason for the difficulty in classifying architectural style images, which affects the classification accuracy of architectural style maps and can be studied in more depth in the future.
-
-
![]()
图 3 同名建筑区域匹配流程图
Figure 3. Flowchart of Matching Two Corresponding Architectural Images
![]()
图 4 街景建筑区域映射方法流程图
Figure 4. Flowchart of Building Outline Mapping Methods Based on Architectural Area in Street View
![]()
图 6 单像建筑区域方位映射示意图
Figure 6. Diagram of Azimuth Mapping Method Based on a Panoramic Image
![]()
图 8 点到建筑俯视轮廓的方位范围示意图
Figure 8. Diagram of Azimuth Coverage from One Position to Building Outline
![]()
图 9 单体建筑轮廓匹配多个建筑区域影像的示意图
Figure 9. Diagram of One Building Outline Matching Mutiple Architectural Images
![]()
图 10 实验区域及代表性街区或景点的位置分布
Figure 10. Experimental Region and Location Distribution of Representative Blocks or Scenic Spots
![]()
图 12 测试集的建筑区域检测结果示例
Figure 12. Selected Examples of Architectural Area Detection Results on Test Set
![]()
图 13 同名建筑区域匹配结果的混淆矩阵
Figure 13. Confusion Matrix of Matching Results with the Same Name Architectural Area
![]()
图 14 相邻两张街景的同名建筑区域匹配过程
Figure 14. Matching Process of Two Corresponding Architectural Areas in an Image Pair
![]()
图 20 建筑风格地图分类精度条形图
Figure 20. Bar Chart of Classification Results of Architectural Styles
表 1 各类风格建筑区域的检测精度表
Table 1 Detection Precision of Architectural Area of Different Styles on Test Set
建筑区域的风格类别 AP/% 战国时期楚国建筑风格 69.23 汉代建筑风格 57.17 唐代建筑风格 89.43 宋代建筑风格 86.73 元代建筑风格 73.61 明代建筑风格 78.53 清代建筑风格 79.59 京派民居 88.53 苏派民居 72.19 徽派民居 84.37 民国民居 78.61 现代建筑风格 84.13 古希腊建筑风格 76.97 古罗马建筑风格 73.37 哥特式建筑风格 63.17 法国古典风格 43.85 巴洛克建筑风格 48.29 拜占庭建筑风格 89.20 其他西式风格 65.40 mAP 73.81 
下载: 导出CSV
表 2 两种同名建筑区域匹配方法的精度对比结果
Table 2 Accuracy Comparison of Two Matching Methods
同名建筑区域匹配方法 精确率/% 召回率/% F1分数 耗时/ms 本文方法 86.1 90.3 0.882 180.1 核线约束下的SURF匹配 66.7 94.2 0.781 241.6
下载: 导出CSV
表 3 两种位置映射方法准确率的对比结果
Table 3 Accuracy Results of Two Mapping Methods
位置映射方法 正确映射/个 错误映射/个 映射失败/个 单像方位映射 106 26 18 双像点位映射 63 11 76
下载: 导出CSV
表 4 存在多映射问题的单体建筑数量
Table 4 Number of Buildings with Multiple Mapping Problems
位置映射方法 成功映射的建筑/个 存在多映射的建筑/个 单像方位映射 13 522 5 805 双像点位映射 9 595 2 245
下载: 导出CSV
-
[1] Goel A, Juneja M, Jawahar C V. Are Buildings Only Instances?: Exploration in Architectural Style Ca- tegories[C]//The Eighth Indian Conference on Computer Vision, Graphics and Image Processing, New York, USA, 2012
[2] Zhang Luming, Song Mingli, Liu Xiao, et al. Recognizing Architecture Styles by Hierarchical Sparse Coding of Blocklets[J]. Information Sciences, 2014, 254: 141-154 doi: 10.1016/j.ins.2013.08.020
[3] Zhao Peipei, Miao Qiguang, Song Jianfeng, et al. Architectural Style Classification Based on Feature Extraction Module[J]. IEEE Access, 2018, 6: 52 598-52 606 doi: 10.1109/ACCESS.2018.2869976
[4] 沈佳洁, 潘励, 胡翔云. 可变形部件模型在高分辨率遥感影像建筑物检测中的应用[J]. 武汉大学学报∙信息科学版, 2017, 42(9): 1 285-1 291 https://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201709015.htm Shen Jiajie, Pan Li, Hu Xiangyun. Building Detection from High Resolution Remote Sensing Imagery Based on a Deformable Part Model[J]. Geomatics and Information Science of Wuhan University, 2017, 42(9): 1 285-1 291 https://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201709015.htm
[5] 赵佩佩. 基于集成投影及卷积神经网络的建筑风格分类算法研究[D]. 西安: 西安电子科技大学, 2015 Zhao Peipei. Architectural Style Classification Algorithms Research Based on Ensemble Projection and Convolution Neural Network[D]. Xi'an: Xidian University, 2015
[6] Cao Rui, Zhu Jiasong, Tu Wei, et al. Integrating Aerial and Street View Images for Urban Land Use Classification[J]. Remote Sensing, 2018, 10(10): 1 553-1 575 doi: 10.3390/rs10101553
[7] Wolff M, Collins R T, Liu Yanxi.Regularity-Driven Building Facade Matching Between Aerial and Street Views[C]//IEEE Conference on Computer Vision and Pattern Recognition, Washington D C, USA, 2016
[8] 宋为刚. 基于街景与航拍图像配准的视觉定位技术[D]. 苏州: 苏州大学, 2018 Song Weigang. A Visual Localization Technique Based on Street View and Aerial Image Registration[D]. Suzhou: Soochow University, 2018
[9] Sun Bin, Chen Chen, Zhu Yingying, et al. GEOCAPSNET: Ground to Aerial View Image Geo-Localization Using Capsule Network[C]//IEEE International Conference on Multimedia and Expo, Washington D C, SA, 2019
[10] 熊曦. 基于智能手机单张照片的建筑物快速定位算法[D]. 北京: 清华大学, 2015 Xiong Xi, Research on Fast Single-Image-Based Building Localization with a Smartphone[D]. Beijing: Tsinghua University, 2015
[11] 陈运, 蔡忠亮, 李伯钊, 等. 一种拍摄目标的地理位置标注方法[J]. 测绘地理信息, 2020, 45(5): 142-145 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXG202005032.htm Chen Yun, Cai Zhongliang, Li Bozhao, et al. A Geo-location Computation Method of Objects in the Photo[J]. Journal of Geomatics, 2020, 45(5): 142-145 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXG202005032.htm
[12] Ren Shaoqing, He Kaiming, Girshick R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1 137-1 149 doi: 10.1109/TPAMI.2016.2577031
[13] Bay H, Ess A, Tuytelaars T. Speeded Up Robust Features (SURF)[J]. Computer Vision and Image Understanding, 2008, 110(3): 346-359 doi: 10.1016/j.cviu.2007.09.014
[14] 吕凤华, 舒宁, 龚龑, 等. 利用多特征进行航空影像建筑物提取[J]. 武汉大学学报∙信息科学版, 2017, 42(5): 656-660 https://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201705014.htm Lü Fenghua, Shu Ning, Gong Yan, et al. Regular Building Extraction from High Resolution Image Based on Multilevel-Features[J]. Geomatics and Information Science of Wuhan University, 2017, 42(5): 656-660 https://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201705014.htm
[15] 吴高巍, 陶卿, 王珏, 等. 基于后验概率的支持向量机[J]. 计算机研究与发展, 2005, 42(2): 196-202 https://www.cnki.com.cn/Article/CJFDTOTAL-JFYZ200502002.htm Wu Gaowei, Tao Qing, Wang Jue, et al. Support Vector Machines Based on Posteriori Probability[J]. Journal of Computer Research and Development, 2005, 42(2): 196-202 https://www.cnki.com.cn/Article/CJFDTOTAL-JFYZ200502002.htm
[16] 谢东海, 钟若飞, 吴俣, 等. 球面全景影像相对定向与精度验证[J]. 测绘学报, 2017, 46(11): 1 822-1 829 doi: 10.11947/j.AGCS.2017.20160645 Xie Donghai, Zhong Ruofei, Wu Yu, et al. Relative Pose Estimation and Accuracy Verification of Spherical Panoramic Image[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(11): 1 822-1 829 doi: 10.11947/j.AGCS.2017.20160645
[17] 刘帅, 陈军, 孙敏, 等. 双球面投影几何可量测全景模型的构建[J]. 计算机辅助设计与图形学学报, 2015, 27(4): 657-665 https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201504012.htm Liu Shuai, Chen Jun, Sun Min, et al. Measurable Panorama Construction Based on Binocular Spherical Projective Geometry[J]. Journal of Computer Aided Design and Computer Graphics, 2015, 27(4): 657-665 https://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201504012.htm
[18] 张春森, 王西旗, 郭丙轩. 城市环境下基于C/S架构的影像空间定位[J]. 武汉大学学报∙信息科学版, 2018, 43(7): 978-983 https://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201807002.htm Zhang Chunsen, Wang Xiqi, Guo Bingxuan. Space Location of Image in Urban Environments Based on C/S Structure[J]. Geomatics and Information Scien- ce of Wuhan University, 2018, 43(7): 978-983 https://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201807002.htm
[19] 吴幼丝. 球形全景影像位姿估计[D]. 武汉: 武汉大学, 2017 Wu Yousi. Position and Orientation Estimation of Spherical Panorama Image[D]. Wuhan: Wuhan University, 2017
[20] Guan Fangli, Fang Zhixiang, Yu Tao, et al. Detec- ting Visually Salient Scene Areas and Deriving Their Relative Spatial Relations from Continuous Street-View Panoramas[J]. International Journal of Digital Earth, 2020, 13(12): 1 504-1 531 doi: 10.1080/17538947.2020.1731618
[21] 王志旋, 钟若飞, 谢东海. 球面全景影像自动测量路灯坐标的方法[J]. 中国图象图形学报, 2018, 23(9): 1 371-1 381 https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB201809010.htm Wang Zhixuan, Zhong Ruofei, Xie Donghai. Automatically Measuring the Coordinates of Streetlights in Vehicle-Borne Spherical Images[J]. Journal of Image and Graphics, 2018, 23(9): 1 371-1 381 https://www.cnki.com.cn/Article/CJFDTOTAL-ZGTB201809010.htm
[22] Hwang C L, Yoon K, Hwang C L, et al. Multiple Attribute Decision Making[J]. Lecture Notes in Economics and Mathematical Systems, 1981, 404(4): 287-288 http://www.researchgate.net/publication/238761193_Multiple_Attribute_Decision_Making_-_Methods_and_Application_A_State_of_the_Art_Survey
[23] Kang Jian, Körner M, Wang Yuanyuan, et al. Building Instance Classification Using Street View Images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 145: 44-59 doi: 10.1016/j.isprsjprs.2018.02.006
计量
- 文章访问数: 1461
- HTML全文浏览量: 385
- PDF下载量: 185
