The Optimized Cloud Storage Method of Massive GNSS Small Files

LI Linyang; LV Zhiping; CUI Yang; WANG Yupu; ZHOU Haitao

doi:10.13203/j.whugis20150136

Volume 42 Issue 8

Aug. 2017

Turn off MathJax

Article Contents

Abstract

References

Geomatics and Information Science of Wuhan University > 2017 > 42(8): 1068-1074. > DOI: 10.13203/j.whugis20150136

LI Linyang, LV Zhiping, CUI Yang, WANG Yupu, ZHOU Haitao. The Optimized Cloud Storage Method of Massive GNSS Small Files[J]. Geomatics and Information Science of Wuhan University, 2017, 42(8): 1068-1074. DOI: 10.13203/j.whugis20150136

Citation:

PDF (1316 KB)

The Optimized Cloud Storage Method of Massive GNSS Small Files

1.
State Key Laboratory of Geo-Information Engineering, Xi'an 710054, China
2.
School of Geographical Spatial Information, Information Engineering University, Zhengzhou 450052, China

Funds:

The National Natural Science Foundation of China 41674019

the National Key Research and Development Program of China 2016YFB0501701

State Key Laboratory of Geo-information Engineering SKLGIE2016-M-1-2

More Information

Author Bio:
LI Linyang, PhD candidate, specializes in GNSS data processing. E-mail: lilinyang810810@163.com
Corresponding author:
LV Zhiping, PhD, professor. E-mail: ssscenter@126.com
Received Date: September 13, 2015
Published Date: August 04, 2017

Graphical Abstract

Abstract

Abstract

The data volume of GNSS is increasing exponentially, while HDFS is capable of handling the problem of the storage bottleneck of massive GNSS data, it is faced with much time consumption, poor file correlation and lack of optimization mechanisms. According to the matter of the low processing efficiency of massive GNSS small files faced by HDFS, a new cloud storage method is provided based on the types, characteristics and storage flow of GNSS data, the writing, adding, reading and deleting strategies are optimized. First the observation files and solution files are respectively merged, and the compressed Trie index is established on the merged files; and after splitting the existed index, the index blocks are distributed stored in each mode based on the matching algorithm. Data and products of 28 days from IGS are applied in the experiment, and the result shows that the memory consumption of each node can be decreased greatly, and the efficiency of writing, direct reading, reading after adding files, concurrent reading and deleting can be improved significantly, effective cloud storage of massive GNSS small files is hereafter realized.
- GNSS,
- small files,
- Hadoop distributed file system,
- cloud storage,
- compact trie-join

FullText(HTML)

References (13)

References

[1]	刘经南, 刘晖.建立我国卫星定位连续运行参考站网的若干思考[J].武汉大学学报·信息科学版, 2003, 28(5):27-31 http://ch.whu.edu.cn/CN/abstract/abstract4745.shtml Liu Jingnan, Liu Hui. Thought on Establishing a Continuous Operating Reference Station Network in China[J]. Geomatics and Information Science of Wuhan University, 2003, 28(5):27-31 http://ch.whu.edu.cn/CN/abstract/abstract4745.shtml
[2]	李林阳, 吕志平, 陈正生, 等.海量连续运行参考站网数据云存储模型[J].导航定位学报, 2014, 2(3):64-70 http://www.cnki.com.cn/Article/CJFDTOTAL-CHWZ201403014.htm Li Linyang, Lv Zhiping, Chen Zhengsheng, et al. Cloud Storage Model of Massive Continuous Operating Reference Station System Data[J]. Journal of Navigation and Positioning, 2014, 2(3):64-70 http://www.cnki.com.cn/Article/CJFDTOTAL-CHWZ201403014.htm
[3]	施闯, 章红平, 辜声峰, 等.云定位技术及云定位服务平台[J].武汉大学学报·信息科学版, 2015, 40(8):995-999 http://ch.whu.edu.cn/CN/abstract/abstract3402.shtml Shi Chuang, Zhang Hongping, Gu Shengfeng, et al. Technology of Cloud Positioning and Its Platform for Positioning Service[J]. Geomatics and Information Science of Wuhan University, 2015, 40(8):995-999 http://ch.whu.edu.cn/CN/abstract/abstract3402.shtml
[4]	Wei Dai, Mostafa B. An Improved Task Assignment Scheme for Hadoop Running in the Clouds[J]. Journal of Cloud Computing, 2013, 2(1):1-16 https://www.researchgate.net/publication/275483750_An_improved_task_assignment_scheme_for_Hadoop_running_in_the_clouds
[5]	Cloudera. The Small Files Problem[EB/OL]. http://www.cloudera.com/blog/2009/02/the-small-files-problem.html, 2013
[6]	Zhao Xiaoyong, Yang Yang, Sun Lili, et al. Metadata-Aware Small Files Storage Architecture on Hadoop[J]. Web Information Systems and Mining, 2012, 7529:136-143 doi: 10.1007/978-3-642-33469-6
[7]	Dong Bo, Zheng Qinghua, Feng Tia, et al. An Optimized Approach for Storing and Accessing Small Files on Cloud Storage[J]. Journal of Network and Computer Applications, 2012, 35(6):1847-1862 doi: 10.1016/j.jnca.2012.07.009
[8]	刘小俊, 徐正全, 潘少明.一种结合RDBMS和Hadoop的海量小文件存储方法[J].武汉大学学报·信息科学版, 2013, 38(1):113-115 http://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201301028.htm Liu Xiaojun, Xu Zhengquan, Pan Shaoming. A Massive Small File Storage Solution Combination of RDBMS and Hadoop[J]. Geomatics and Information Science of Wuhan University, 2013, 38(1):113-115 http://www.cnki.com.cn/Article/CJFDTOTAL-WHCH201301028.htm
[9]	熊炼, 徐正全, 王涛, 等.云环境下的时空数据小文件存储策略[J].武汉大学学报·信息科学版, 2014, 39(10):1252-1256 http://ch.whu.edu.cn/CN/abstract/abstract3105.shtml Xiong Lian, Xu Zhengquan, Wang Tao, et al. On the Store Strategy of Small Spatio-Temporal Data Files in Cloud Environment[J]. Geomatics and Information Science of Wuhan University, 2014, 39(10):1252-1256 http://ch.whu.edu.cn/CN/abstract/abstract3105.shtml
[10]	Liu Xuhui, Han Jizhong, Zhong Yunqin, et al. Implementing WebGIS on Hadoop:A Case Study of Improving Small File I/O Performance on HDFS[C]. IEEE International Conference on CLUSTER Computing and Workshops, New Orleans, USA, 2009
[11]	IGS Central Bureau. IGS Technical Report[OL]. http://kb.igs.org/hc/en-us/articles/115000200811-Technical-Report-2016, 2016
[12]	Maxime C, Thierry L. Encyclopedia of Database Systems[M].USA:Springer, 2009:3179-3182
[13]	Feng Jianhua, Wang Jiannan, Li Guoliang. Trie-join:A Trie-based Method for Efficient String Similarity[J]. The VLDB Journal, 2012, 21(4):437-461 doi: 10.1007/s00778-011-0252-8

[1]	YAO Yibin, RAN Qishun, ZHANG Bao. Application of Improved Heuristic Segmentation Algorithm to Step Detection of GNSS Coordinate Time Series[J]. Geomatics and Information Science of Wuhan University, 2019, 44(5): 648-654. DOI: 10.13203/j.whugis20170322
[2]	SHEN Xin, LIU Yulin, LI Shixue, YAO Huang. An Optimization Design Method for High Temporal Resolution Remote Sensing Satellite Constellation Based on Improved PSO Algorithm[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1986-1993. DOI: 10.13203/j.whugis20180160
[3]	Zhan Weiwei, Wang Wei, Chen Nengcheng, Wang Chao. Path Planning Strategies for UAV Based on Improved A*Algorithm[J]. Geomatics and Information Science of Wuhan University, 2015, 40(3): 315-320.
[4]	ZHA Feng, XU Jiangning, LI Jingshu, HE Hongyang. Improvement of a Series of Fuzzy Damp Algorithms in SINS[J]. Geomatics and Information Science of Wuhan University, 2013, 38(6): 705-709.
[5]	ZHAN Jianfeng, FAN Chong, LI Tao. A Forest Firespot Automatic Detection Algorithm for HJ-IRS Imagery[J]. Geomatics and Information Science of Wuhan University, 2012, 37(11): 1321-1324.
[6]	LUO Zhicai, LIN Xu, ZHOU Boyang. Improved Algorithm of Autocovariance Least-Squares Noise Estimation[J]. Geomatics and Information Science of Wuhan University, 2012, 37(10): 1164-1167.
[7]	LI Zhenhai, LUO Zhicai, WANG Haihong, LI Qiong. Visualization of Gravity Vector Field Using Improved FLIC Algorithm[J]. Geomatics and Information Science of Wuhan University, 2011, 36(3): 276-279.
[8]	YIN Shuling, SHU Ning, LIU Xinhua. Classification of Remote Sensing Image Based on Adaptive GA and Improved BP Algorithm[J]. Geomatics and Information Science of Wuhan University, 2007, 32(3): 201-204.
[9]	YAO Huanmei, HUANG Rentao, GAN Fuxing, LIU Yang. Principal Component Analysis of the Water Quality Evaluation in East Lake[J]. Geomatics and Information Science of Wuhan University, 2005, 30(8): 732-735.
[10]	CHEN Jiangping, FU Zhongliang, XU Zhihong. An Improved Algorithm of Apriori[J]. Geomatics and Information Science of Wuhan University, 2003, 28(1): 94-99.

Cited By

Cited by

Periodical cited type(3)

1.	袁东，张健，余洋洋，张志良. 差异特征注意力引导的偏振图像高光移除. 光学技术. 2024(02): 247-256 .
2.	陈毅夫，何敬，刘刚，毛佳琪. 融合Swin-Transformer网络模型的水体高光区域提取. 遥感信息. 2023(04): 129-136 .
3.	乔玉晶，张思远，赵宇航. 高光弱纹理物体表面鲁棒重建方法. 光子学报. 2019(12): 172-182 .

Other cited types(2)

Get Citation

PDF

XML

Article views (1798) PDF downloads (387) Cited by(5)

The Optimized Cloud Storage Method of Massive GNSS Small Files

Abstract

References

Related Articles

Cited by

Periodical cited type(3)

Other cited types(2)

Catalog

Related

The Optimized Cloud Storage Method of Massive GNSS Small Files

Abstract

References

Related Articles

Cited by

Periodical cited type(3)

Other cited types(2)

Catalog

Related

Export File

Citation

Format

Content