Keywords: 
• urban augmented reality /
• scene structure extraction /
• information annotation /
• scene perception /
• building faç

Abstract: Integrating scene structure information into information annotation can effectively solve the problems such as unclear information indication, confusing occlusion representation and overlapping view layout in urban AR. Aiming at the problem of missing scene structure in information annotation, a method of building scene structure extraction for urban AR is proposed, which distinguishes the geographical entities and considers the accuracy, efficiency and robustness. Firstly, a scene perception network for building scene structure extraction is constructed to extract semantic label, scene depth and surface normal from a single scene image. Then, they are transformed into structure features such as building facade corners and orientation, and the best matching between them and the building outlines in 2D map is calculated. Finally, the scene image is reconstructed according to geographical entities, while the structure information such as region contours, scene depth and facades orientation are generated. During the experiments, 9470 sets of samples generated from Google Street View data are employed to train and test the scene perception network, while the effectiveness of scene structure extraction is tested in 32 sets of building scenes in Graz. The results show that the proposed method can extract the structure of building scene in near real time, and the facade contours are more regular. In the case of geo-registration error or partial occlusion, the quality of elevation extraction is significantly better than the method based on map to analyze the scene image, which proves that it has better robustness.