Message Board

Respected readers, authors and reviewers, you can add comments to this page on any questions about the contribution, review,        editing and publication of this journal. We will give you an answer as soon as possible. Thank you for your support!

Name
E-mail
Phone
Title
Content
Verification Code
Turn off MathJax
Article Contents

JIANG San, CHEN Wu, LI Qingquan, JIANG Wanshou. Recent Research of Incremental Structure from Motion for Unmanned Aerial Vehicle Images[J]. Geomatics and Information Science of Wuhan University. doi: 10.13203/j.whugis20220130
Citation: JIANG San, CHEN Wu, LI Qingquan, JIANG Wanshou. Recent Research of Incremental Structure from Motion for Unmanned Aerial Vehicle Images[J]. Geomatics and Information Science of Wuhan University. doi: 10.13203/j.whugis20220130

Recent Research of Incremental Structure from Motion for Unmanned Aerial Vehicle Images

doi: 10.13203/j.whugis20220130
Funds:

the National Natural Science Foundation of China (42001413)

  • Received Date: 2022-03-30
    Available Online: 2022-04-14
  • Objectives: Incremental structure from motion (SfM) has become the widely used workflow for aerial triangulation (AT) of unmanned aerial vehicle (UAV) images. Recently, extensive research has been conducted to improve the efficiency, precision and scalability of SfM-based AT for UAV images. Meanwhile, deep learning-based methods have also been exploited for the geometry processing in the fields of photogrammetry and computer vision, which have been verified with large potential in the AT of UAV images. This paper aims to give a review of recent work in the SfM-based AT for UAV images. Methods: First, the workflow of SfM-based AT is briefly presented in terms of feature matching and geometry solving, in which the former aims to obtain enough and accurate correspondences, and the latter attempts to solve unknown parameters. Second, literature review is given for feature matching and geometry solving. For feature matching, classical hand-crafted and recent learning-based methods are presented from the aspects of feature extraction, feature matching and outlier removal. For geometry solving, the principle of SfM-based AT is firstly given with some well-known and widely-used open-source SfM software. Efficiency improvement and large-scale processing are then summarized, which focus on improving the capability of SfM to process large-scale UAV images. Finally, further search is concluded from four aspects, including the change of data acquisition modes, the scalability for large-scale scenes, the development of communication and hardware, and the fusion of deep learning-based methods. Results: The review demonstrates that the existing research promotes the development of SfM-based AT towards the direction of high efficiency, high precision and high robustness, and also promotes the development of both commercial and open-source software packages. Conclusions: Considering the characteristics of UAV images, the efficiency, precision and robustness of SfM-based AT and its application need further improvement and exploitation. This paper could give an extensive conclusion and be a useful reference to the related researchers.
  • [1] Colomina I, Molina P. Unmanned Aerial Systems for Photogrammetry and Remote Sensing:A Review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 92:79-97
    [2] Zhou X H, Xie K, Huang K, et al. Offsite Aerial Path Planning for Efficient Urban Scene Reconstruction[J]. ACM Transactions on Graphics, 2020, 39(6):192
    [3] Jiang S, Jiang W S, Wang L Z. Unmanned Aerial Vehicle-Based Photogrammetric 3D Mapping:A Survey of Techniques, Applications, and Challenges[J]. IEEE Geoscience and Remote Sensing Magazine, 2248, PP (99):2-38
    [4] Fan B, Kong Q Q, Wang X C, et al. A Performance Evaluation of Local Features for Image-Based 3D Reconstruction[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2019, 28(10):4774-4789
    [5] Triggs B, McLauchlan P F, Hartley R I, et al. Bundle Adjustment-a Modern Synthesis[M]//Vision Algorithms:Theory and Practice. Berlin, Heidelberg:Springer, 2000:298-372
    [6] Bhowmick B, Patra S, Chatterjee A, et al. Divide and Conquer:A Hierarchical Approach to LargeScale Structure-from-Motion[J]. Computer Vision and Image Understanding, 2017, 157:190-205
    [7] Zhu S Y, Shen T W, Zhou L, et al. Accurate, Scalable and Parallel Structure from Motion[EB/OL]. 2017:arXiv:1702.08601. https://arxiv.org/abs/1702.08601
    [8] Harris C, Stephens M. A combined corner and edge detector[C]//Proceedings ofthe Alvey Vision Conference, Manchester, UK, 1988
    [9] Lowe D G. Distinctive Image Features from Scale-Invariant Keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110
    [10] Morel J M, Yu G S. ASIFT:A New Framework for Fully Affine Invariant Image Comparison[J]. SIAM Journal on Imaging Sciences, 2009, 2(2):438-469
    [11] Arandjelović R, Zisserman A. Three things everyone should know to improve object retrieval[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA, 2012
    [12] Bay H, Ess A, Tuytelaars T, et al. Speeded-up Robust Features (SURF)[J]. Computer Vision and Image Understanding, 2008, 110(3):346-359
    [13] Rublee E, Rabaud V, Konolige K, et al. ORB:An efficient alternative to SIFT or SURF[C]//2011 International Conference on Computer Vision. Barcelona, Spain. 2011:2564-2571
    [14] Wu C C. SiftGPU:A GPU Implementation of Scale Invariant Feature Transform (SIFT)[J].2007
    [15] Hu H, Zhu Q, Du Z Q, et al. Reliable Spatial Relationship Constrained Feature Point Matching of Oblique Aerial Images[J]. Photogrammetric Engineering&Remote Sensing, 2015, 81(1):49-58
    [16] Jiang S, Jiang W S. On-Board GNSS/IMU Assisted Feature Extraction and Matching for Oblique UAV Images[J]. Remote Sensing, 2017, 9(8):813
    [17] Sun Y B, Zhao L, Huang S D, et al. L2-SIFT:SIFT Feature Extraction and Matching for Large Images in Large-Scale Aerial Photogrammetry[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 91:1-16
    [18] Schönberger J L, Hardmeier H, Sattler T, et al. Comparative evaluation of hand-crafted and learned local features[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017
    [19] Tian Y R, Fan B, Wu F C. L2-net:Deep learning of discriminative patch descriptor in euclidean space[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017
    [20] MISHCHUK A, MISHKIN D, RADENOVIC F, et al. Working hard to know your neighbor's margins:Local descriptor learning loss[J]. arXiv preprint arXiv:1705.10872, 2017.
    [21] Luo Z X, Shen T W, Zhou L, et al. ContextDesc:Local Descriptor Augmentation with CrossModality Context[C]//IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019
    [22] Yi K M, Trulls E, Ono Y, et al. Learning to Find Good Correspondences[C]//IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018
    [23] DeTone D, Malisiewicz T, Rabinovich A. SuperPoint:Self-Supervised Interest Point Detection and Description[C]//IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, USA, 2018
    [24] Dusmanu M, Rocco I, Pajdla T, et al. D2-net:A trainable CNN for Joint Description and Detection of Local Features[C]//IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019
    [25] Arya S, Mount D M, Netanyahu N S, et al. An Optimal Algorithm for Approximate Nearest Neighbor Searching Fixed Dimensions[J]. Journal of the ACM, 1998, 45(6):891-923
    [26] Hartmann W, Havlena M, Schindler K. Recent Developments in Large-Scale Tie-Point Matching[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 115:47-62
    [27] Sedaghat A, Mokhtarzade M, Ebadi H. Uniform Robust Scale-Invariant Feature Matching for Optical Remote Sensing Images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(11):4516-4527
    [28] Wu C C. Towards Linear-Time Incremental Structure from Motion[C]//International Conference on 3D Vision, Seattle, USA, 2013
    [29] Hartmann W, Havlena M, Schindler K. Predicting matchability[C]//IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, 2014
    [30] Frahm J M, Pollefeys M, Lazebnik S, et al. Fast Robust Large-Scale Mapping from Video and Internet Photo Collections[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(6):538-549
    [31] Raguram R, Wu C C, Frahm J M, et al. Modeling and Recognition of Landmark Image Collections Using Iconic Scene Graphs[J]. International Journal of Computer Vision, 2011, 95(3):213-239
    [32] Alsadik B, Gerke M, Vosselman G, et al. Minimal Camera Networks for 3D Image Based Modeling of Cultural Heritage Objects[J]. Sensors, 2014, 14(4):5785-5804
    [33] Havlena M, Torii A, Pajdla T. Efficient Structure from Motion by Graph Optimization[C]//European Conference on Computer Vision, Berlin, Germany, 2010
    [34] Jiang S, Jiang W S. Efficient Match Pair Selection for Oblique UAV Images Based on Adaptive Vocabulary Tree[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 161:61-75
    [35] Schönberger J L, Radenović F, Chum O, et al. From single image query to detailed 3D reconstruction[C]//IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015
    [36] Jiang S, Jiang W S. Efficient SfM for Oblique UAV Images:From Match Pair Selection to Geometrical Verification[J]. Remote Sensing, 2018, 10(8):1246
    [37] Jiang S, Jiang W S. Efficient Structure from Motion for Oblique UAV Images Based on Maximal Spanning Tree Expansion[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 132:140-161
    [38] Xu Z H, Wu L X, Gerke M, et al. Skeletal Camera Network Embedded Structure-from-Motion for 3D Scene Reconstruction from UAV Images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 121:113-127
    [39] Han X F, Leung T, Jia Y Q, et al. MatchNet:Unifying feature and metric learning for patch-based matching[C]//IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015
    [40] Zagoruyko S, Komodakis N. Learning to compare image patches via convolutional neural networks[C]//IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015
    [41] Simo-Serra E, Trulls E, Ferraz L, et al. Discriminative learning of deep convolutional feature point descriptors[C]//IEEE International Conference on Computer Vision, Santiago, Chile, 2015
    [42] Fischler M A, Bolles R C. Random Sample Consensus:A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography[J]. Commun ACM, 1981, 24(6):381-395
    [43] Raguram R, Chum O, Pollefeys M, et al. USAC:A Universal Framework for Random Sample Consensus[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8):2022-2038
    [44] Lu L P, Zhang Y, Tao P J. Geometrical Consistency Voting Strategy for Outlier Detection in Image Matching[J]. Photogrammetric Engineering&Remote Sensing, 2016, 82(7):559-570
    [45] Jiang S, Jiang W S. Hierarchical Motion Consistency Constraint for Efficient Geometrical Verification in UAV Stereo Image Matching[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 142:222-242
    [46] Aguilar W, Frauel Y, Escolano F, et al. A Robust Graph Transformation Matching for Non-Rigid Registration[J]. Image and Vision Computing, 2009, 27(7):897-910
    [47] Izadi M, Saeedi P. Robust Weighted Graph Transformation Matching for Rigid and Nonrigid Image Registration[J]. IEEE Transactions on Image Processing, 2012, 21(10):4369-4382
    [48] Jiang S, Jiang W S. Reliable Image Matching via Photometric and Geometric Constraints Structured by Delaunay Triangulation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 153:1-20
    [49] Ma J Y, Zhao J, Tian J W, et al. Robust Point Matching via Vector Field Consensus[J]. IEEE Transactions on Image Processing, 2014, 23(4):1706-1721
    [50] Bian J W, Lin W Y, Matsushita Y, et al. GMS:grid-based motion statistics for fast, ultra-robust feature correspondence[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017
    [51] Brachmann E, Krull A, Nowozin S, et al. DSAC-differentiable RANSAC for camera localization[C]//IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017
    [52] RANFTL R, KOLTUN V. Deep Fundamental Matrix Estimation[C]//2018:292-309
    [53] Zhang J H, Sun D W, Luo Z X, et al. Learning Two-View Correspondences and Geometry Using Order-Aware Network[C]//IEEE Transactions on Pattern Analysis and Machine Intelligence, New York, USA, 2019
    [54] Sun W W, Jiang W, Trulls E, et al. ACNe:Attentive Context Normalization for Robust Permutation-Equivariant Learning[C]//IEEE Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020
    [55] Schönberger J L, Frahm J M. Structure-from-motion revisited[C]//IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016
    [56] Moulon P, Monasse P, Marlet R. Global fusion of relative motions for robust, accurate and scalable structure from motion[C]//IEEE International Conference on Computer Vision, Sydney, Australia, 2013
    [57] Moulon P, Monasse P, Marlet R. Adaptive Structure from Motion with Model Estimation[C]//Asian Conference on Computer Vision, Beijing, China, 2012
    [58] FUHRMANN S, LANGGUTH F, GOESELE M. MVE:A Multi-View Reconstruction Environment[C]//Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage, New York, USA, 2014
    [59] Rupnik E, Daakir M, Pierrot Deseilligny M. MicMac-a Free, Open-Source Solution for Photogrammetry[J]. Open Geospatial Data, Software and Standards, 2017, 2:14
    [60] Snavely N, Seitz S M, Szeliski R. Photo Tourism[J]. ACM Transactions on Graphics, 2006, 25(3):835-846
    [61] Wu C C, Agarwal S, Curless B, et al. Multicore Bundle Adjustment[C]//IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, USA, 2011
    [62] Hänsch R, Drude I, Hellwich O. Modern Methods of Bundle Adjustment on the GPU[J]. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016, 3(3):43-50
    [63] Rupnik E, Nex F, Remondino F. Automatic Orientation of Large Blocks of Oblique Images[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2013, 40:299-304
    [64] Sun Y B, Sun H B, Yan L, et al. RBA:Reduced Bundle Adjustment for Oblique Aerial Photogrammetry[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 121:128-142
    [65] Cefalu A, Haala N, Fritsch D. Structureless Bundle Adjustment with Self-Calibration Using Accumulated Constraints[J]. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016, 3(3):3-9
    [66] Lourakis M I A, Argyros A A. SBA:A Software Package for Generic Sparse Bundle Adjustment[J]. ACM Transactions on Mathematical Software, 2009, 36(1):2
    [67] Konolige K. Sparse Sparse Bundle Adjustment[C]//IEEE International Conference on Computer Vision, New York, USA, 2010
    [68] Kümmerle R, Grisetti G, Strasdat H, et al. G2O:A General Framework for Graph Optimization[C]//IEEE International Conference on Robotics and Automation, Shanghai, China, 2011
    [69] Zhao L, Huang S D, Sun Y B, et al. ParallaxBA:Bundle Adjustment Using Parallax Angle Feature Parametrization[J]. The International Journal of Robotics Research, 2015, 34(4-5):493-516
    [70] Bhowmick B, Patra S, Chatterjee A, et al. Divide and Conquer:Efficient Large-Scale Structure from Motion Using Graph Partitioning[C]//Asian Conference on Computer Vision, Shanghai, China, 2014
    [71] Chen Y, Shen S H, Chen Y S, et al. Graph-Based Parallel Large Scale Structure from Motion[J]. Pattern Recognition, 2020, 107:107537
    [72] Xu B, Zhang L, Liu Y X, et al. Robust Hierarchical Structure from Motion for Large-Scale Unstructured Image Sets[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 181:367-384
    [73] Farenzena M, Fusiello A, Gherardi R. Structure-and-Motion Pipeline on a Hierarchical Cluster Tree[C]//IEEE International Conference on Computer Vision Workshops, Kyoto, Japan, 2009
    [74] Snavely N, Seitz S M, Szeliski R. Skeletal graphs for efficient structure from motion[C]//IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, USA, 2008
    [75] Shah R, Deshpande A, Narayanan P J. Multistage SFM:Revisiting Incremental Structure from Motion[C]//The 2nd International Conference on 3D Vision, Tokyo, Japan, 2014
    [76] Luo Z X, Zhou L, Bai X Y, et al. ASLFeat:Learning Local Features of Accurate Shape and Localization[C]//IEEE Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020
    [77] Jiang S, Jiang W S, Guo B X, et al. Learned Local Features for Structure from Motion of UAV Images:A Comparative Evaluation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14:10583-10597
    [78] Wang W, Zhao Y, Han P C, et al. TerrainFusion:Real-Time Digital Surface Model Reconstruction Based on Monocular SLAM[C]//IEEE International Conference on Intelligent Robots and Systems (IROS), Macao, China, 2019
  • 加载中
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Article Metrics

Article views(205) PDF downloads(41) Cited by()

Related
Proportional views

Recent Research of Incremental Structure from Motion for Unmanned Aerial Vehicle Images

doi: 10.13203/j.whugis20220130
Funds:

the National Natural Science Foundation of China (42001413)

Abstract: Objectives: Incremental structure from motion (SfM) has become the widely used workflow for aerial triangulation (AT) of unmanned aerial vehicle (UAV) images. Recently, extensive research has been conducted to improve the efficiency, precision and scalability of SfM-based AT for UAV images. Meanwhile, deep learning-based methods have also been exploited for the geometry processing in the fields of photogrammetry and computer vision, which have been verified with large potential in the AT of UAV images. This paper aims to give a review of recent work in the SfM-based AT for UAV images. Methods: First, the workflow of SfM-based AT is briefly presented in terms of feature matching and geometry solving, in which the former aims to obtain enough and accurate correspondences, and the latter attempts to solve unknown parameters. Second, literature review is given for feature matching and geometry solving. For feature matching, classical hand-crafted and recent learning-based methods are presented from the aspects of feature extraction, feature matching and outlier removal. For geometry solving, the principle of SfM-based AT is firstly given with some well-known and widely-used open-source SfM software. Efficiency improvement and large-scale processing are then summarized, which focus on improving the capability of SfM to process large-scale UAV images. Finally, further search is concluded from four aspects, including the change of data acquisition modes, the scalability for large-scale scenes, the development of communication and hardware, and the fusion of deep learning-based methods. Results: The review demonstrates that the existing research promotes the development of SfM-based AT towards the direction of high efficiency, high precision and high robustness, and also promotes the development of both commercial and open-source software packages. Conclusions: Considering the characteristics of UAV images, the efficiency, precision and robustness of SfM-based AT and its application need further improvement and exploitation. This paper could give an extensive conclusion and be a useful reference to the related researchers.

JIANG San, CHEN Wu, LI Qingquan, JIANG Wanshou. Recent Research of Incremental Structure from Motion for Unmanned Aerial Vehicle Images[J]. Geomatics and Information Science of Wuhan University. doi: 10.13203/j.whugis20220130
Citation: JIANG San, CHEN Wu, LI Qingquan, JIANG Wanshou. Recent Research of Incremental Structure from Motion for Unmanned Aerial Vehicle Images[J]. Geomatics and Information Science of Wuhan University. doi: 10.13203/j.whugis20220130
Reference (78)

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return