CN109033137B - Dynamic RINEX data storage method and device - Google Patents
Dynamic RINEX data storage method and device Download PDFInfo
- Publication number
- CN109033137B CN109033137B CN201810577089.6A CN201810577089A CN109033137B CN 109033137 B CN109033137 B CN 109033137B CN 201810577089 A CN201810577089 A CN 201810577089A CN 109033137 B CN109033137 B CN 109033137B
- Authority
- CN
- China
- Prior art keywords
- data
- file
- rinex
- observation data
- intermediate file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a dynamic RINEX data storage method and a device, wherein the method comprises the following steps: receiving observation data from a data source, wherein the observation data is a type of data which is transmitted by the data source and has the highest frequency; the observation data is written into an intermediate file in real time in an intermediate format; after the observation data of one epoch is written, judging whether the intermediate file needs to be fragmented: if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time; if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file; and compressing the RINEX file and uploading the compressed RINEX file to the cloud. The invention can obtain the RINEX file for storing accurate frequency point data, and can not occupy a large amount of storage simultaneously.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a dynamic RINEX data storage method and device.
Background
With the development and popularization of satellite positioning technology, communication technology and computer network technology, the public demand for location services is becoming more and more important. Differential positioning has been widely accepted by the public as a convenient method for significantly improving positioning accuracy. The differential positioning needs to store a large amount of off-line observation data for algorithm tuning, off-line calculation and the like.
The off-line observation data is generally stored in a RINEX standard which is stored in a plain text mode, the frequency point definitions of all satellite systems contained in the file are written in the file header, and the contents of the subsequent files are completely typeset according to the file header, so that whether the file header is correct or not is determined, namely whether the observation data stored in the file is accurate or not is determined.
The existing technical scheme has two types: the first is to use a fixed RINEX file header, i.e. to use fixed frequency point definitions; the second method is to receive observation data of partial epoch, analyze which frequency points are included in the data, and generate a RINEX file header based on the frequency points.
The problems existing in the prior art are as follows:
1. frequency point loss can be caused. Because the type of the signal transmitted by each satellite is variable, no matter a fixed frequency point definition is used, or a RIENX file header is generated after receiving data of a part of epochs, all the epoch information cannot be covered after all, and as a result, the stored data is incomplete.
2. By using the fixed frequency point definition, a frequency point which does not exist actually may be written in the file header, which causes troubles in the use of the algorithm.
3. If the RINEX file is generated by analyzing after receiving all the epochs, the temporary storage occupation is huge. If data of 1 day or 2 days is generated, the temporary storage occupies hundreds of MB, and the method cannot be applied to large-scale storage.
Disclosure of Invention
The invention aims to provide a method for accurately storing RINEX observation data, so that the stored RINEX data not only contains the full amount of frequency point types, but also does not contain nonexistent frequency points. Based on this, the technical scheme adopted by the invention is as follows:
a dynamic RINEX data storage method, comprising the steps of:
receiving observation data from a data source, wherein the observation data is a type of data which is transmitted by the data source and has the highest frequency;
the observation data is written into an intermediate file in real time in an intermediate format;
after the observation data of one epoch (the collection of the observation data content at the same time point) is written, judging whether the intermediate file needs to be fragmented:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, the complete set of the frequency points (the combination of the frequency and the code of a satellite system and the representation of a unique signal type) of the intermediate file is analyzed, and an accurate RINEX file is generated;
and compressing the RINEX file and uploading the compressed RINEX file to the cloud.
Further, the data source is a receiver terminal, and the observation data is satellite data observed by the receiver terminal at a time.
Further, the observation data includes a complete set of satellites observed by the receiver terminal, and each satellite is marked by a different frequency point.
Further, the intermediate format is freely defined, including JSON serialization or binary.
Further, the time interval of the slicing is set according to the requirement.
Further, a RINEX file header is generated through the complete set of the frequency points of the intermediate file, the accuracy of the frequency points of the intermediate file is guaranteed, and then the contents of the RINEX file are continuously filled according to the RINEX file header to generate an accurate RINEX file.
Further, the RINEX file is compressed by a ZIP tool or an RNXCPM tool.
Further, the cloud comprises a public cloud or a private cloud.
The invention also provides a RINEX data using method, the RINEX data used by the user adopts the dynamic RINEX data storage method, and the user downloads the required RINEX data according to the time period requirement and combines the data for use.
The invention also provides a dynamic RINEX data storage device, comprising:
the data receiving unit is used for receiving observation data from a data source, wherein the observation data is a type of data which is sent by the data source and has the highest frequency;
the data writing unit is used for writing the observation data into an intermediate file in an intermediate format in real time;
the fragment judgment and processing unit is used for judging whether the intermediate file needs to be fragmented after the observation data of one epoch is written:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file;
and the file compression and uploading unit is used for compressing the RINEX file and uploading the compressed RINEX file to the cloud.
The invention also provides a memory storing a computer program executed by a processor to perform the steps of:
receiving observation data from a data source, wherein the observation data is a type of data which is transmitted by the data source and has the highest frequency;
the observation data is written into an intermediate file in real time in an intermediate format;
after the observation data of one epoch is written, judging whether the intermediate file needs to be fragmented:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file;
and compressing the RINEX file and uploading the compressed RINEX file to the cloud.
The invention has the following beneficial effects:
1. the invention uses all data of the fragment file to generate the RINEX file header, so that the problem of frequency point loss can not exist.
2. The invention uses all data of the fragment file to generate the RINEX file header, so that the problem of actually no frequency point during writing can be solved.
3. The invention adopts the configurable file fragmentation interval, and the file fragmentation is carried out after the fragmentation interval is met, so that the temporary storage occupation is not huge.
Drawings
FIG. 1 is a flow chart of the dynamic RINEX data storage of the present invention;
FIG. 2 is a block diagram of a dynamic RINEX data storage device according to the present invention.
Detailed Description
The invention combines file fragmentation with real-time writing in of intermediate file, so that a RINEX (Receiver Independent Exchange Format) file for storing accurate frequency point data can be obtained, a large amount of storage cannot be occupied concurrently, and the efficiency and the accuracy for concurrently storing RINEX observation data in a large scale are provided. The invention is further illustrated below with reference to the figures and examples.
The first embodiment is as follows:
FIG. 1 is a flow chart of dynamic RINEX data storage according to the present invention, comprising the following steps:
step 1, receiving observation data from a data source. The data source may be a receiver terminal, and the observation data is a type of data transmitted by the terminal with the highest frequency, and represents satellite data observed by the terminal at a specified time point, namely observation data of an epoch.
The observation data at a moment includes a complete set of satellites observed by the terminal at the moment, and each satellite has different frequency point marks, for example, the 1 st satellite of the GPS system has three frequency bands of L1, L2, and L5, the L2 frequency band has different codes of C, W, X, and the frequency band and the codes are combined into a unique frequency point. However, different terminals transmit different data according to their own processing strategies, and some terminals may transmit only the data of L1 and L2, and some terminals may also generate L5 data.
The satellite moves around the earth rapidly, so that the satellites observed by the terminal at different moments are different, and the frequency point sets of all the satellites observed correspondingly are also different. Therefore, all frequency point sets cannot be accurately analyzed by only depending on data at one moment.
And 2, writing the observation data into the intermediate file in an intermediate format in real time. The intermediate format can be freely defined by the user, such as JSON serialization, binary, etc., and the invention does not limit the choice of the intermediate format.
And 3, judging whether the file needs to be fragmented or not after the observation data of one epoch is written. By using the file fragments, the problem of huge temporary storage occupation can be solved. The interval of the shards can be set by itself, such as 15 minutes, half an hour and the like, the file of the shard interval is only a few MB in size, and a large number of files can be stored simultaneously. If the fragmentation is not needed, returning, and executing the step 2 when waiting for the next epoch data to arrive; if the fragment is to be fragmented, step 3 is performed.
And 4, reading all the data files in the intermediate format into a memory, and analyzing a complete set of the frequency points of the files. By using the complete set of the frequency points, an accurate RINEX file header can be generated, the frequency points of the intermediate file are ensured to be accurate, namely, the frequency points cannot be lost, and redundant frequency points cannot exist, and then the RINEX file content is continuously filled according to the RINEX file header.
RINEX is an international universal text file format for storing observation data, and the file is characterized in that a frequency point complete set of a satellite system is defined in a file header, and all subsequent contents are generated by depending on the file header. The correctness of the file header is crucial.
Many algorithm tools directly support the parsing of the format, and the invention ultimately still uses RINEX to store observation data to maintain the versatility of each algorithm tool.
And step 5, compressing the segment of RINEX file. RINEX is a text file, and has a huge compression space, and the storage space occupation can be obviously reduced by uploading the compressed text file.
The compression mode can be selected by a user, can be purely ZIP compression, or can be a common tool for compressing RINEX data, is designed for compressing RINEX observation data, and has higher compression rate by using an RNXCPM tool.
And 6, uploading the compressed RINEX file to a cloud. Finally, files of all the fragments are stored in the cloud, and the public cloud or the private cloud can be used for solving the problem of storage of a large number of files.
And 7, when the user uses the system, downloading the required data according to the time period requirement, and combining the data for use.
Example two:
the present invention also provides a dynamic RINEX data storage device, as shown in fig. 2, comprising:
the data receiving unit is used for receiving observation data from a data source, wherein the observation data is a type of data which is sent by the data source and has the highest frequency;
the data writing unit is used for writing the observation data into an intermediate file in an intermediate format in real time;
the fragment judgment and processing unit is used for judging whether the intermediate file needs to be fragmented after the observation data of one epoch is written:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file;
and the file compression and uploading unit is used for compressing the RINEX file and uploading the compressed RINEX file to the cloud.
Example three:
the invention also provides a memory storing a computer program executed by a processor to perform the steps of:
receiving observation data from a data source, wherein the observation data is a type of data which is transmitted by the data source and has the highest frequency;
the observation data is written into an intermediate file in real time in an intermediate format;
after the observation data of one epoch is written, judging whether the intermediate file needs to be fragmented:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file;
and compressing the RINEX file and uploading the compressed RINEX file to the cloud.
Although the present invention has been described with reference to the preferred embodiments, it is not intended to limit the present invention, and those skilled in the art can make variations and modifications of the present invention without departing from the spirit and scope of the present invention by using the methods and technical contents disclosed above.
Claims (11)
1. A method for dynamic RINEX data storage, the method comprising the steps of:
receiving observation data from a data source, wherein the observation data is a type of data which is transmitted by the data source and has the highest frequency;
the observation data is written into an intermediate file in real time in an intermediate format;
after the observation data of one epoch is written, judging whether the intermediate file needs to be fragmented:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file;
and compressing the RINEX file and uploading the compressed RINEX file to the cloud.
2. A dynamic RINEX data storage method as claimed in claim 1, wherein said data source is a receiver terminal and said observed data is satellite data observed by the receiver terminal at a time instant.
3. The method of claim 2, wherein said observation data comprises a full set of satellites observed by said receiver terminal, each satellite being identified by a different frequency bin.
4. The dynamic RINEX data storage method of claim 1, wherein the intermediate format is freely defined and comprises JSON serialization or binary.
5. The dynamic RINEX data storage method of claim 1, wherein the time intervals of the slices are self-setting on demand.
6. The dynamic RINEX data storage method of claim 1, wherein a RINEX file header is generated through a full set of frequency points of the intermediate file to ensure the accuracy of the frequency points of the intermediate file, and then the contents of the RINEX file are continuously filled according to the RINEX file header to generate an accurate RINEX file.
7. The dynamic RINEX data storage method of claim 1, wherein RINEX files are compressed via ZIP tools.
8. The dynamic RINEX data storage method of claim 1, wherein the cloud comprises a public cloud or a private cloud.
9. A method for using RINEX data, wherein the user uses the RINEX data to use the data storage method as claimed in any one of claims 1-8, and the user downloads and uses the required RINEX data according to the time slot requirement.
10. A dynamic RINEX data storage device, wherein the device comprises:
the data receiving unit is used for receiving observation data from a data source, wherein the observation data is a type of data which is sent by the data source and has the highest frequency;
the data writing unit is used for writing the observation data into an intermediate file in an intermediate format in real time;
the fragment judgment and processing unit is used for judging whether the intermediate file needs to be fragmented after the observation data of one epoch is written:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file;
and the file compression and uploading unit is used for compressing the RINEX file and uploading the compressed RINEX file to the cloud.
11. A memory storing a computer program, the computer program being executable by a processor to perform the steps of:
receiving observation data from a data source, wherein the observation data is a type of data which is transmitted by the data source and has the highest frequency;
the observation data is written into an intermediate file in real time in an intermediate format;
after the observation data of one epoch is written, judging whether the intermediate file needs to be fragmented:
if the intermediate file does not need to be fragmented, returning, waiting for the observation data of the next epoch to arrive, and writing the observation data into the intermediate file in real time;
if the intermediate file needs to be fragmented, all the intermediate file is read into a memory, and a complete set of frequency points of the intermediate file is analyzed to generate an accurate RINEX file;
and compressing the RINEX file and uploading the compressed RINEX file to the cloud.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810577089.6A CN109033137B (en) | 2018-06-06 | 2018-06-06 | Dynamic RINEX data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810577089.6A CN109033137B (en) | 2018-06-06 | 2018-06-06 | Dynamic RINEX data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109033137A CN109033137A (en) | 2018-12-18 |
| CN109033137B true CN109033137B (en) | 2021-11-05 |
Family
ID=64612301
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810577089.6A Active CN109033137B (en) | 2018-06-06 | 2018-06-06 | Dynamic RINEX data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109033137B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111475463B (en) * | 2020-04-01 | 2023-02-24 | 中国人民解放军火箭军工程大学 | GNSS observation data digital relation storage method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156719A (en) * | 2011-03-28 | 2011-08-17 | 武汉大学 | Methods for encoding and decoding observed value file of global positioning system |
| US8396944B2 (en) * | 2000-01-27 | 2013-03-12 | Marger Johnson & Mccollom, P.C. | Rich media file format and delivery methods |
| WO2014046917A1 (en) * | 2012-09-20 | 2014-03-27 | California Institute Of Technology | Enhanced broadcast ephemeris for high accuracy assisted gps positioning |
| CN104765876A (en) * | 2015-04-24 | 2015-07-08 | 中国人民解放军信息工程大学 | Massive GNSS small file cloud storage method |
| CN105843933A (en) * | 2016-03-30 | 2016-08-10 | 电子科技大学 | Index building method for distributed memory columnar database |
| CN106951475A (en) * | 2017-03-07 | 2017-07-14 | 郑州铁路职业技术学院 | Big data distributed approach and system based on cloud computing |
| CN107707629A (en) * | 2017-09-12 | 2018-02-16 | 千寻位置网络有限公司 | The method of GNSS message conversion checking |
| CN107704504A (en) * | 2017-08-29 | 2018-02-16 | 中国科学院光电研究院 | A kind of RINEX files are traced to the source information extracting method and system |
-
2018
- 2018-06-06 CN CN201810577089.6A patent/CN109033137B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8396944B2 (en) * | 2000-01-27 | 2013-03-12 | Marger Johnson & Mccollom, P.C. | Rich media file format and delivery methods |
| CN102156719A (en) * | 2011-03-28 | 2011-08-17 | 武汉大学 | Methods for encoding and decoding observed value file of global positioning system |
| WO2014046917A1 (en) * | 2012-09-20 | 2014-03-27 | California Institute Of Technology | Enhanced broadcast ephemeris for high accuracy assisted gps positioning |
| CN104765876A (en) * | 2015-04-24 | 2015-07-08 | 中国人民解放军信息工程大学 | Massive GNSS small file cloud storage method |
| CN105843933A (en) * | 2016-03-30 | 2016-08-10 | 电子科技大学 | Index building method for distributed memory columnar database |
| CN106951475A (en) * | 2017-03-07 | 2017-07-14 | 郑州铁路职业技术学院 | Big data distributed approach and system based on cloud computing |
| CN107704504A (en) * | 2017-08-29 | 2018-02-16 | 中国科学院光电研究院 | A kind of RINEX files are traced to the source information extracting method and system |
| CN107707629A (en) * | 2017-09-12 | 2018-02-16 | 千寻位置网络有限公司 | The method of GNSS message conversion checking |
Non-Patent Citations (2)
| Title |
|---|
| "Dual-frequency Multi-constellation SBAS Simulator";Doubek, J et al,;《 28th International Technical Meeting of The Satellite-Division-of-the-Institute-of-Navigation》;20150918;第1775-1782页 * |
| "基于粒计算的数据分片算法";吴润秀 等;《计算机应用》;20070630;第27卷(第6期);第1388-1391页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109033137A (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110321387B (en) | Data synchronization method, equipment and terminal equipment | |
| US20220035045A1 (en) | Ephemeris data processing method, electronic device and storage medium | |
| CN104090970A (en) | Interest point showing method and device | |
| CN101977435A (en) | Terminal positioning method and system | |
| CN111314392B (en) | Satellite navigation positioning auxiliary ephemeris data compression and transmission method | |
| CN112822260B (en) | File transmission method and device, electronic equipment and storage medium | |
| CN109656797B (en) | Log data association method and device | |
| CN111694828A (en) | Data processing method, device, system and storage medium | |
| CN109033137B (en) | Dynamic RINEX data storage method and device | |
| CN103488776B (en) | A kind of method that real-time online obtains map datum | |
| CN114091409A (en) | Method, system, equipment and storage medium for distributed asynchronous Excel analysis | |
| CN114328348A (en) | FPGA acceleration board card and market data processing method thereof | |
| CN112636808B (en) | Telescope remote data transmission and control method based on Beidou short message | |
| CN111628956A (en) | Format conversion method, device and system for network request transmission data | |
| CN101957820A (en) | Method and device for generating doorplate information library | |
| CN101577870A (en) | Biological investigation data mobile acquisition system | |
| CN102611716B (en) | Method, device and system for transmitting media file | |
| CN115002196B (en) | Data processing method and device and vehicle end acquisition equipment | |
| CN103490856B (en) | Mark5B-format VLBI data receiving decoding and error correcting system and method used for deep space exploration | |
| CN115794934A (en) | Production facility monitoring data and digital twin model integration system and method | |
| US20210181359A1 (en) | Mobile delivery device methods and systems | |
| CN111858093B (en) | Message processing method, device, equipment and storage medium | |
| CN103078905A (en) | Communication management method of GPS (Global Position System) terminal | |
| KR100721556B1 (en) | Method for processing dump data packet of low earth orbit satellite | |
| CN115348333A (en) | Data transmission method, system and equipment based on UDP (user Datagram protocol) double-end communication interaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |