Semantic Segmentation of Indoor 3D Point Cloud Model Based on 2D-3D Semantic Transfer

XIONG Hanjiang; ZHENG Xianwei; DING Youli; ZHANG Yi; WU Xiujie; ZHOU Yan

doi:10.13203/j.whugis20180190

Volume 43 Issue 12

Dec. 2018

Turn off MathJax

Article Contents

Abstract

References

Geomatics and Information Science of Wuhan University > 2018 > 43(12): 2303-2309. > DOI: 10.13203/j.whugis20180190

XIONG Hanjiang, ZHENG Xianwei, DING Youli, ZHANG Yi, WU Xiujie, ZHOU Yan. Semantic Segmentation of Indoor 3D Point Cloud Model Based on 2D-3D Semantic Transfer[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 2303-2309. DOI: 10.13203/j.whugis20180190

Citation:

PDF (4506 KB)

Semantic Segmentation of Indoor 3D Point Cloud Model Based on 2D-3D Semantic Transfer

1.
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
2.
School of Mathematics and Statistics, Wuhan University, Wuhan 430079, China

Funds:

The National Key Research and Development Program of China 2018YFB0505401

the National Natural Science Foundation of China 41871361

the National Natural Science Foundation of China 41701445

the LIESMARS (State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing) Special Research Funding

More Information

Author Bio:
XIONG Hanjiang, PhD, professor, specializes in the theories and methods of geospatial data information visualization, massive spatial data network services and fast transmission, and rapid reconstruction of 3D models. E-mail: xionghanjiang@whu.edu.cn
Corresponding author:
ZHENG Xianwei, associate professor. E-mail: zhengxw@whu.edu.cn
Received Date: May 16, 2018
Published Date: December 04, 2018

Graphical Abstract

Abstract

Abstract

In this paper, we propose an effective algorithm based on graph model for semantic transfer from 2D images to 3D point clouds, which can effectively solve the problem of objectification and lack of structured information of 3D point cloud model. Our proposed method uses the extended full convolutional neural network to extract the indoor space layout and object semantics of 2D images, and then implements the transfer of 2D semantics to 3D semantics based on the 2D image superpixels and 3D point clouds as nodes to construct a graph model of consistency between images and intra-image consistency. The experiment from 3D point cloud shows that the proposed method can obtain accurate indoor 3D point cloud semantic classification results. The accuracy of point cloud classification can reach 73.875 2%, and the classification effect is better.
- semantic 3D point cloud model,
- semantic transfer,
- semantic mark,
- point cloud classification

FullText(HTML)

References (24)

References

[1]	卢学良, 童晓冲, 张永生, 等.城市密集点云的区域生长表面构网改进算法[J].武汉大学学报·信息科学版, 2016, 41(6):832-837 http://ch.whu.edu.cn/CN/abstract/abstract5471.shtml Lu Xueliang, Tong Xiaochong, Zhang Yongsheng, et al. An Improved Region-Growing Surface Triangulation Algorithm for Urban Dense Point Cloud[J]. Geomatics and Information Science of Wuhan University, 2016, 41(6):832-837 http://ch.whu.edu.cn/CN/abstract/abstract5471.shtml
[2]	刘如飞, 卢秀山, 岳国伟, 等.一种车载激光点云数据中道路自动提取方法[J].武汉大学学报·信息科学版, 2017, 42(2):250-256 http://ch.whu.edu.cn/CN/abstract/abstract5669.shtml Liu Rufei, Lu Xiushan, Yue Guowei, et al. An Automatic Extraction Method of Road from Vehicle-Borne Laser Scanning Point Clouds[J]. Geomatics and Information Science of Wuhan University, 2017, 42(2): 250-256 http://ch.whu.edu.cn/CN/abstract/abstract5669.shtml
[3]	汤圣君.多视图像增强的RGB-D室内高精度三维测图方法[D].武汉: 武汉大学, 2017 http://xueshu.baidu.com/s?wd=paperuri%3A%28ce4cc597200c0bd1a757b0e3387428d8%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Fcdmd.cnki.com.cn%2FArticle%2FCDMD-10486-1017192073.htm&ie=utf-8&sc_us=354381197624485132 Tang Shengjun. Multi-view Image Enhancement of RGB-D Indoor High-Precision 3D Mapping Method[D]. Wuhan: Wuhan University, 2017 http://xueshu.baidu.com/s?wd=paperuri%3A%28ce4cc597200c0bd1a757b0e3387428d8%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Fcdmd.cnki.com.cn%2FArticle%2FCDMD-10486-1017192073.htm&ie=utf-8&sc_us=354381197624485132
[4]	Newcombe R A, Izadi S, Hilliges O, et al. Kinect Fusion: Real-time Dense Surface Mapping and Tracking[C]. IEEE International Symposium on Mixed and Augmented Reality, Austin, TX, USA, 2011 http://xueshu.baidu.com/s?wd=paperuri%3A%28edad359b869e7ca47b8a18dc48c79221%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Fieeexplore.ieee.org%2Fdocument%2F6162880%2F&ie=utf-8&sc_us=16132128827952784548
[5]	Furukawa Y, Ponce J. Accurate, Dense, and Robust Multiview Stereopsis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(8): 1 362-1 376 doi: 10.1109/TPAMI.2009.161
[6]	Wu C. VisualSFM: A Visual Structure from Motion System[OL]. http://ccwu.me/vsfm/, 2011
[7]	Koppula H S, Anand A, Joachims T, et al. Semantic Labeling of 3D Point Clouds for Indoor Scenes[C]. Advances in Neural Information Processing Systems, Granada, Spain, 2011 http://www.researchgate.net/publication/303003338_Semantic_labeling_of_3d_point_clouds_for_indoor_scenes
[8]	Anand A, Koppula H S, Joachims T, et al. Contextually Guided Semantic Labeling and Search for Three-dimensional Point Clouds[J]. The International Journal of Robotics Research, 2013, 32(1): 19-34 http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=62a91ede00888c74f10882805c49ec85
[9]	Xiong X, Munoz D, Bagnell J A, et al. 3-D Scene Analysis via Sequenced Predictions over Points and Regions[C]. IEEE International Conference on Robotics and Automation, Shanghai, China, 2011 https://www.researchgate.net/publication/221077032_3-D_Scene_Analysis_via_Sequenced_Predictions_over_Points_and_Regions
[10]	Kalogerakis E, Hertzmann A, Singh K. Learning 3D Mesh Segmentation and Labeling[J]. ACM Transactions on Graphics, 2010, 29(4): 1-12 http://d.old.wanfangdata.com.cn/Periodical/jsjgcyyy201711038
[11]	Lai K, Fox D. Object Recognition in 3D Point Clouds Using Web Data and Domain Adaptation[J]. The International Journal of Robotics Research, 2010, 29(8): 1 019-1 037 doi: 10.1177/0278364910369190
[12]	Munoz D, Bagnell J A, Hebert M. Stacked Hierarchical Labeling[C]//European Conference on Computer Vision. Berlin, Heidelberg : Springer, 2010
[13]	Murphy K P, Torralba A, Freeman W T. Using the Forest to See the Trees: A Graphical Model Relating Features, Objects, and Scenes[C]. Advances in Neural Information Processing Systems, Vancouver and Whistler, British Columbia, Canada, 2004 http://xueshu.baidu.com/s?wd=paperuri%3A%28bd0ea02f064c7cced8e027e71517e05e%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Flibra.msra.cn%2FPublication%2F2064035%2Fusing-the-forest-to-see-the-tree-a-graphical-model-relating-features-objects-and-the-scenes&ie=utf-8&sc_us=17353571651555494696&sc_as_para=sc_lib%3A
[14]	Boulch A, Guerry J, Saux B L, et al. SnapNet: 3D Point Cloud Semantic Labeling with 2D Deep Segmentation Networks[J]. Computers & Graphics, 2017, 71: 189-198 http://www.sciencedirect.com/science/article/pii/S0097849317301942
[15]	Wu Z, Song S, Khosla A, et al. 3D ShapeNets: A Deep Representation for Volumetric Shapes[C]. IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015 http://www.oalib.com/paper/4081952
[16]	袁理, 陈庆虎, 廖海斌, 等.单视影像下的人脸快速三维重建[J].武汉大学学报·信息科学版, 2012, 37(4): 487-491 http://ch.whu.edu.cn/CN/abstract/abstract186.shtml Yuan Li, Chen Qinghu, Liao Haibin, et al. Rapid Three-Dimensional Reconstruction of Face with Single Vision[J]. Geomatics and Information Science of Wuhan University, 2012, 37(4): 487-491 http://ch.whu.edu.cn/CN/abstract/abstract186.shtml
[17]	Russell B C, Torralba A, Murphy K P, et al. LabelMe: A Database and Web-Based Tool for Image Annotation[J]. International Journal of Computer Vision, 2008, 77(1-3): 157-173 doi: 10.1007/s11263-007-0090-8
[18]	Deng J, Dong W, Socher R, et al. ImageNet: A Large-Scale Hierarchical Image Database[C]. IEEE Conference on CVPR, Vancouver, BC, Canada, 2009 http://xueshu.baidu.com/s?wd=paperuri%3A%2834cca6e2e2661669a192947a88c8212d%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Ficp.jsp%3Farnumber%3D5206848&ie=utf-8&sc_us=11083167391443987016
[19]	Kuettel D, Guillaumin M, Ferrari V. Segmentation Propagation in ImageNet[C]. European Conference on Computer Vision, Florence, Italy, 2012 http://www.springerlink.com/index/A01J1343VM4877R8.pdf
[20]	Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation[C]. IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015 http://xueshu.baidu.com/s?wd=paperuri%3A%28c76cbf802fc633294315697571af911e%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Fwww.ncbi.nlm.nih.gov%2Fpubmed%2F27244717%2F&ie=utf-8&sc_us=529353482871617016
[21]	Mallya A, Lazebnik S. Learning Informative Edge Maps for Indoor Scene Layout Prediction[C]. IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015 https://dl.acm.org/citation.cfm?id=2919332.2919749
[22]	Jia Y, Shelhamer E, Donahue J, et al. Caffe: Convo- lutional Architecture for Fast Feature Embedding[C]. The 22nd ACM International Conference on Multimedia, Orlando, Florida, USA, 2014
[23]	Gupta S, Arbelaez P, Malik J. Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images[C]. IEEE Conference on CVPR, Portland, Oregon, 2013 https://www.researchgate.net/publication/261227425_Perceptual_Organization_and_Recognition_of_Indoor_Scenes_from_RGB-D_Images
[24]	Achanta R, Shaji A, Smith K, et al. SLIC Superpixels Compared to State-of-the-Art Superpixel Methods[J]. IEEE Transactions on Pattern Analy-sis and Machine Intelligence, 2012, 34(11): 2 274-2 282 doi: 10.1109/TPAMI.2012.120

[1]	MA Tian-en, LIU Tao, DU Ping, CHEN Po-yi, LING Zhen-fei. A 3D Point Cloud Semantic Segmentation Method for Aggregating Global Context Information[J]. Geomatics and Information Science of Wuhan University. DOI: 10.13203/j.whugis20230143
[2]	XIANG Xueyong, LI Guangyun, WANG Li, ZONG Wenpeng, LÜ Zhipeng, XIANG Fengzhuo. Semantic Segmentation of Point Clouds Using Local Geometric Features and Dilated Neighborhoods[J]. Geomatics and Information Science of Wuhan University, 2023, 48(4): 534-541. DOI: 10.13203/j.whugis20200567
[3]	TANG Shengjun, ZHANG Yunjie, LI Xiaoming, YAO Mengmeng, YE Zhihuang, LI Yaxin, GUO Renzhong, WANG Weixi. A High-Precision Indoor Point Cloud Classification Method Jointly Optimized by Super Voxel Random Forest and LSTM Neural Network[J]. Geomatics and Information Science of Wuhan University, 2023, 48(4): 525-533. DOI: 10.13203/j.whugis20220125
[4]	JIANG Tengping, YANG Bisheng, ZHOU Yuzhou, ZHU Runsong, HU Zongtian, DONG Zhen. Bilevel Convolutional Neural Networks for 3D Semantic Segmentation Using Large-scale LiDAR Point Clouds in Complex Environments[J]. Geomatics and Information Science of Wuhan University, 2020, 45(12): 1942-1948. DOI: 10.13203/j.whugis20200081
[5]	ZHANG Ruiju, ZHOU Xin, ZHAO Jianghong, CAO Min. A Semantic Segmentation Algorithm of Ancient Building's Point Cloud Data[J]. Geomatics and Information Science of Wuhan University, 2020, 45(5): 753-759. DOI: 10.13203/j.whugis20180428
[6]	WEI Shuangfeng, LIU Minglei, ZHAO Jianghong, HUANG Shuai. A Survey of Methods for Detecting Indoor Navigation Elements from Point Clouds[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 2003-2011. DOI: 10.13203/j.whugis20180144
[7]	ZHU Qing, LI Shiming, HU Han, ZHONG Ruofei, WU Bo, XIE Linfu. Multiple Point Clouds Data Fusion Method for 3D City Modeling[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1962-1971. DOI: 10.13203/j.whugis20180109
[8]	YUE Chong, LIU Changjun, WANG Xiaofang. Classification Algorithm for Laser Point Clouds of High-steep Slopes Based on Multi-scale Dimensionality Features and SVM[J]. Geomatics and Information Science of Wuhan University, 2016, 41(7): 882-888. DOI: 10.13203/j.whugis20140335
[9]	YI Rulan, XU Feng, DENG Min, LIU Qiliang. An Approach for Hierarchical Semantic Classification of Islands Based on Formal Concept Analysis[J]. Geomatics and Information Science of Wuhan University, 2012, 37(8): 897-901.
[10]	TAN Xicheng BIAN Fuling, . Heterogeneous Spatial Information System Semantic Interoperability Based on Bayes Data Classification and Ontology[J]. Geomatics and Information Science of Wuhan University, 2006, 31(8): 724-727.

Cited By

Cited by

Periodical cited type(15)

1.	康志忠，杨俊涛. 室内实景三维重建技术综述. 时空信息学报. 2024(01): 1-10 .
2.	汤圣君，杜思齐，王伟玺，郭仁忠. 面向室内空间智能的三维场景图表达与应用. 测绘学报. 2024(07): 1355-1370 .
3.	武斌，王远哲，丛佳，赵洁. 一种三维点云语义分割的深度特征提取方法. 测绘科学. 2024(10): 123-132 .
4.	项学泳，李广云，王力，宗文鹏，吕志鹏，向奉卓. 利用局部几何特征与空洞邻域的点云语义分割. 武汉大学学报(信息科学版). 2023(04): 534-541 .
5.	汤圣君，张韵婕，李晓明，姚萌萌，叶致煌，李亚鑫，郭仁忠，王伟玺. 超体素随机森林与LSTM神经网络联合优化的室内点云高精度分类方法. 武汉大学学报(信息科学版). 2023(04): 525-533 .
6.	张序国. 基于3ds Max软件的室内空间布局优化技术研究. 中国建筑装饰装修. 2023(17): 62-64 .
7.	赵江洪，窦新铜，曹月娥，王殷瑞，黄先峰. 一种基于分割结果实现三维点云分类的方法. 测绘科学. 2022(03): 85-95 .
8.	张兴岩，李琦，梁栋，蒲洁. 一种邻接区域平面元融合的桥面分割方法. 大地测量与地球动力学. 2022(08): 863-869 .
9.	李健，姚亮. 融合多特征深度学习的地面激光点云语义分割. 测绘科学. 2021(03): 133-139+162 .
10.	张立国，程瑶，金梅，王娜. 基于改进BiSeNet的室内场景语义分割方法. 计量学报. 2021(04): 515-520 .
11.	毛琳，陈思宇，杨大伟. 引导式的卷积神经网络视频行人动作分类改进方法. 武汉大学学报(信息科学版). 2021(08): 1241-1246 .
12.	张佳颖，赵晓丽，陈正. 基于深度学习的点云语义分割综述. 激光与光电子学进展. 2020(04): 28-46 .
13.	张瑞菊，周欣，赵江洪，曹闵. 一种古建筑点云数据的语义分割算法. 武汉大学学报(信息科学版). 2020(05): 753-759 .
14.	蒋腾平，杨必胜，周雨舟，朱润松，胡宗田，董震. 道路点云场景双层卷积语义分割. 武汉大学学报(信息科学版). 2020(12): 1942-1948 .
15.	赵春叶，许钢，邢广鑫，郭芮，李若楠，江娟娟. 融合个体识别的3D点云语义分割方法研究. 黑龙江工业学院学报(综合版). 2019(12): 50-54 .

Other cited types(21)

Get Citation

PDF

XML

Article views (4456) PDF downloads (471) Cited by(36)

Semantic Segmentation of Indoor 3D Point Cloud Model Based on 2D-3D Semantic Transfer

Abstract

References

Related Articles

Cited by

Periodical cited type(15)

Other cited types(21)

Catalog

Related

Semantic Segmentation of Indoor 3D Point Cloud Model Based on 2D-3D Semantic Transfer

Abstract

References

Related Articles

Cited by

Periodical cited type(15)

Other cited types(21)

Catalog

Related

Export File

Citation

Format

Content