Volume 42 Issue 8
Aug.  2017
Turn off MathJax
Article Contents
Abstract
References
Related Articles
LI Linyang, LV Zhiping, CUI Yang, WANG Yupu, ZHOU Haitao. The Optimized Cloud Storage Method of Massive GNSS Small Files[J]. Geomatics and Information Science of Wuhan University, 2017, 42(8): 1068-1074. DOI: 10.13203/j.whugis20150136
Citation: LI Linyang, LV Zhiping, CUI Yang, WANG Yupu, ZHOU Haitao. The Optimized Cloud Storage Method of Massive GNSS Small Files[J]. Geomatics and Information Science of Wuhan University, 2017, 42(8): 1068-1074. DOI: 10.13203/j.whugis20150136
shu

The Optimized Cloud Storage Method of Massive GNSS Small Files

  • 1.

    State Key Laboratory of Geo-Information Engineering, Xi'an 710054, China

  • 2.

    School of Geographical Spatial Information, Information Engineering University, Zhengzhou 450052, China

Funds: 

The National Natural Science Foundation of China 41674019

the National Key Research and Development Program of China 2016YFB0501701

State Key Laboratory of Geo-information Engineering SKLGIE2016-M-1-2

More Information
  • Author Bio:

    LI Linyang, PhD candidate, specializes in GNSS data processing. E-mail: lilinyang810810@163.com

  • Corresponding author:

    LV Zhiping, PhD, professor. E-mail: ssscenter@126.com

  • Received Date: September 13, 2015
  • Published Date: August 04, 2017
    Graphical Abstract
    • Abstract
  • The data volume of GNSS is increasing exponentially, while HDFS is capable of handling the problem of the storage bottleneck of massive GNSS data, it is faced with much time consumption, poor file correlation and lack of optimization mechanisms. According to the matter of the low processing efficiency of massive GNSS small files faced by HDFS, a new cloud storage method is provided based on the types, characteristics and storage flow of GNSS data, the writing, adding, reading and deleting strategies are optimized. First the observation files and solution files are respectively merged, and the compressed Trie index is established on the merged files; and after splitting the existed index, the index blocks are distributed stored in each mode based on the matching algorithm. Data and products of 28 days from IGS are applied in the experiment, and the result shows that the memory consumption of each node can be decreased greatly, and the efficiency of writing, direct reading, reading after adding files, concurrent reading and deleting can be improved significantly, effective cloud storage of massive GNSS small files is hereafter realized.
    • FullText(HTML)
    • References (13)
  • [1]
    刘经南, 刘晖.建立我国卫星定位连续运行参考站网的若干思考[J].武汉大学学报·信息科学版, 2003, 28(5):27-31 http://ch.whu.edu.cn/CN/abstract/abstract4745.shtml

    Liu Jingnan, Liu Hui. Thought on Establishing a Continuous Operating Reference Station Network in China[J]. Geomatics and Information Science of Wuhan University, 2003, 28(5):27-31 http://ch.whu.edu.cn/CN/abstract/abstract4745.shtml
    [2]
    李林阳, 吕志平, 陈正生, 等.海量连续运行参考站网数据云存储模型[J].导航定位学报, 2014, 2(3):64-70 http://www.cnki.com.cn/Article/CJFDTOTAL-CHWZ201403014.htm

    Li Linyang, Lv Zhiping, Chen Zhengsheng, et al. Cloud Storage Model of Massive Continuous Operating Reference Station System Data[J]. Journal of Navigation and Positioning, 2014, 2(3):64-70 http://www.cnki.com.cn/Article/CJFDTOTAL-CHWZ201403014.htm
    [3]
    施闯, 章红平, 辜声峰, 等.云定位技术及云定位服务平台[J].武汉大学学报·信息科学版, 2015, 40(8):995-999 http://ch.whu.edu.cn/CN/abstract/abstract3402.shtml

    Shi Chuang, Zhang Hongping, Gu Shengfeng, et al. Technology of Cloud Positioning and Its Platform for Positioning Service[J]. Geomatics and Information Science of Wuhan University, 2015, 40(8):995-999 http://ch.whu.edu.cn/CN/abstract/abstract3402.shtml
    [4]
    Wei Dai, Mostafa B. An Improved Task Assignment Scheme for Hadoop Running in the Clouds[J]. Journal of Cloud Computing, 2013, 2(1):1-16 https://www.researchgate.net/publication/275483750_An_improved_task_assignment_scheme_for_Hadoop_running_in_the_clouds
    [5]
    Cloudera. The Small Files Problem[EB/OL]. http://www.cloudera.com/blog/2009/02/the-small-files-problem.html, 2013
    [6]
    Zhao Xiaoyong, Yang Yang, Sun Lili, et al. Metadata-Aware Small Files Storage Architecture on Hadoop[J]. Web Information Systems and Mining, 2012, 7529:136-143 doi: 10.1007/978-3-642-33469-6
    [7]
    Dong Bo, Zheng Qinghua, Feng Tia, et al. An Optimized Approach for Storing and Accessing Small Files on Cloud Storage[J]. Journal of Network and Computer Applications, 2012, 35(6):1847-1862 doi: 10.1016/j.jnca.2012.07.009
    [8]
    刘小俊, 徐正全, 潘少明.一种结合RDBMS和Hadoop的海量小文件存储方法[J].武汉大学学报·信息科学版, 2013, 38(1):113-115 http://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201301028.htm

    Liu Xiaojun, Xu Zhengquan, Pan Shaoming. A Massive Small File Storage Solution Combination of RDBMS and Hadoop[J]. Geomatics and Information Science of Wuhan University, 2013, 38(1):113-115 http://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201301028.htm
    [9]
    熊炼, 徐正全, 王涛, 等.云环境下的时空数据小文件存储策略[J].武汉大学学报·信息科学版, 2014, 39(10):1252-1256 http://ch.whu.edu.cn/CN/abstract/abstract3105.shtml

    Xiong Lian, Xu Zhengquan, Wang Tao, et al. On the Store Strategy of Small Spatio-Temporal Data Files in Cloud Environment[J]. Geomatics and Information Science of Wuhan University, 2014, 39(10):1252-1256 http://ch.whu.edu.cn/CN/abstract/abstract3105.shtml
    [10]
    Liu Xuhui, Han Jizhong, Zhong Yunqin, et al. Implementing WebGIS on Hadoop:A Case Study of Improving Small File I/O Performance on HDFS[C]. IEEE International Conference on CLUSTER Computing and Workshops, New Orleans, USA, 2009
    [11]
    IGS Central Bureau. IGS Technical Report[OL]. http://kb.igs.org/hc/en-us/articles/115000200811-Technical-Report-2016, 2016
    [12]
    Maxime C, Thierry L. Encyclopedia of Database Systems[M].USA:Springer, 2009:3179-3182
    [13]
    Feng Jianhua, Wang Jiannan, Li Guoliang. Trie-join:A Trie-based Method for Efficient String Similarity[J]. The VLDB Journal, 2012, 21(4):437-461 doi: 10.1007/s00778-011-0252-8
    • Related Articles

  • Related Articles

    [1]GUO Jiachun, LIU Yi, SHEN Wenbin, SHI Guigang. Concept Analysis of Map Projection and Its Applications Based on Manifold Mapping Principle[J]. Geomatics and Information Science of Wuhan University, 2024, 49(12): 2313-2322. DOI: 10.13203/j.whugis20230138
    [2]JIAO Chenchen, LI Songlin, LI Houpu, BIAN Shaofeng, ZHONG Yexun. Non‑iterative Algorithm for Calculating the Reference Latitude of Conformal Conic Projection[J]. Geomatics and Information Science of Wuhan University, 2023, 48(2): 301-307. DOI: 10.13203/j.whugis20200301
    [3]ZENG Xiaoniu, LI Xihai, LIU Jihao, NIU Chao, HOU Weijun. Simultaneous Interpolation and Denoising Method for Airborne Gravity Data Based on Improved Projection onto Convex Sets Theory[J]. Geomatics and Information Science of Wuhan University, 2020, 45(10): 1555-1562. DOI: 10.13203/j.whugis20180470
    [4]LIU Wenchao, WEN Chaojiang, BIAN Hongwei, ZHENG Xiaobing. An Improved Method of Polar Ellipsoid Transverse Mercator Projection Based on Double Projection[J]. Geomatics and Information Science of Wuhan University, 2019, 44(8): 1138-1143. DOI: 10.13203/j.whugis20180028
    [5]SUN Weiwei, JIANG Man, LI Weiyue. Band Selection Using Sparse Self-representation for Hyperspectral Imagery[J]. Geomatics and Information Science of Wuhan University, 2017, 42(4): 441-448. DOI: 10.13203/j.whugis20150052
    [6]LIAO Li, ZHOU Xueqin, LI Qingqing, CHEN Lu, ZHOU Jianzhong. A Dual Iterative Clustering Based Fuzzy Projection Pursuit Clustering Algorithm[J]. Geomatics and Information Science of Wuhan University, 2016, 41(7): 932-938. DOI: 10.13203/j.whugis20140152
    [7]ZHAO Hu, LI Lin, GONG Jianya. Universal Map Projection Selection[J]. Geomatics and Information Science of Wuhan University, 2010, 35(2): 244-247.
    [8]CHENMi, YIYaohua, LIDeren, QINQianqing. Application of Projection Pursuit Based on Dynamical Evolutionary Algorithm to Anomaly Target Detection in Hyperspectral Images[J]. Geomatics and Information Science of Wuhan University, 2006, 31(1): 55-58.
    [9]Chen Yongqi. Selection of Projection Technique and Reduction of Geodetic Observations for Huge Engineering Project SSC[J]. Geomatics and Information Science of Wuhan University, 1993, 18(2): 10-14.
    [10]Hu Yuju. The Possibility of Making Further Additions to the Map Projection Classifications——On the Poly-eylindrical Projection[J]. Geomatics and Information Science of Wuhan University, 1986, 11(3): 55-61.

  • Cited by

    Periodical cited type(5)

    1. 李殷娜，李正强，郑杨，侯伟真，徐文斌，马??，樊程，葛邦宇，姚前，史正. 基于非负矩阵分解的中红外地表特性光谱重建方法. 光谱学与光谱分析. 2024(02): 563-570 .
    2. 钟雷洋，周颖，高松，夏吉喆，李珍，李晓明，乐阳，李清泉. 突发公共卫生事件下的人口流动模式变化识别. 武汉大学学报(信息科学版). 2024(07): 1237-1249 .
    3. 李玉，甄畅，石雪，朱磊. 基于波段影像统计信息量加权K-means聚类的高光谱影像分类. 控制与决策. 2021(05): 1119-1126 .
    4. 赵玉英，任明武. 基于SNMF聚类与类间可分性因子的高光谱波段选择. 计算机与数字工程. 2021(09): 1884-1888 .
    5. 曾梦，宁彬，蔡之华，谷琼. 使用深度对抗子空间聚类实现高光谱波段选择. 计算机应用. 2020(02): 381-385 .

    Other cited types(2)

Catalog

    Get Citation
    PDF
    XML
    Article views (1797) PDF downloads (387) Cited by(7)
    Related

    Copyright © 2013 《武汉大学学报·信息科学版》编辑部

    Address：武汉大学文理学部本科生院楼北5楼Post Code：430072

    Tel：027-68778465,68788631E-mail:whuxxb@vip.163.com

    Submit Article

    Submission Guidelines

    e

    Contact Us

    Qrcode

    Supported by: Beijing Renhe Information Technology Co., Ltd.  Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return