CN105468748B - Distributed storage position data method and system - Google Patents
Distributed storage position data method and system Download PDFInfo
- Publication number
- CN105468748B CN105468748B CN201510836420.8A CN201510836420A CN105468748B CN 105468748 B CN105468748 B CN 105468748B CN 201510836420 A CN201510836420 A CN 201510836420A CN 105468748 B CN105468748 B CN 105468748B
- Authority
- CN
- China
- Prior art keywords
- data
- information
- information field
- position data
- data table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9537—Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed storage position data method and a system, wherein the method comprises the following steps: acquiring N pieces of position data reported by a terminal, and performing data extraction on the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field; and storing the record information into a relational database management system. In the method provided by the invention, the corresponding storage strategy can be adopted according to the characteristics of the position data, so that the access efficiency of the position data can be effectively reduced, and the position data can be read more conveniently.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a method and a system for storing location data in a distributed manner.
Background
In the construction of smart cities, a Geographic Information System (GIS) based location service is essential in a monitoring platform System. However, for a large number of monitoring objects, the position data is updated frequently, so that the storage pressure of the historical position data is large.
Disclosure of Invention
The invention provides a distributed storage position data method and a distributed storage position data system, which are used for solving the problems that in the prior art, monitoring objects are numerous, position data are frequently updated, and the storage pressure of historical position data is high.
The specific technical scheme is as follows:
a method of distributed storage of location data, the method comprising:
acquiring N pieces of position data reported by a terminal, wherein N is a positive integer greater than or equal to 1;
performing data extraction on the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field;
and storing the record information into a relational database management system.
Optionally, before acquiring the N pieces of position data reported by the terminal, the method further includes:
storing the position data acquired in the first time period into a first data table;
and storing the position data acquired in a second time period after the first time period into a second data table.
Optionally, the data extraction is performed on the N pieces of position data, so as to obtain record information at least including a position information field, a time information field, an alarm information field, and a status information field, and the method includes:
analyzing each piece of position data in the N pieces of position data to obtain position information, time information, alarm information and state information of each piece of position data;
integrating the position information of each piece of position data into a position information field, integrating the time information of each piece of position data into a time information field, integrating the alarm information of each piece of position data into an alarm information field, and integrating the state information of each piece of position data into a state information field;
record information including a location information field, a time information field, an alert information field, and a status information field is generated.
Optionally, the data extraction is performed on the N pieces of position data, so as to obtain record information at least including a position information field, a time information field, an alarm information field, and a status information field, and the method includes:
performing data compression on the N pieces of position data, and obtaining a data compression packet and an extraction code corresponding to the data compression packet, wherein the extraction code is used for extracting the data compression packet;
and storing the data compression packet to a distributed open source database, and adding the extraction code to the record information.
Optionally, after storing the record information in a relational database management system, the method further includes:
analyzing the obtained query information, and detecting whether an extraction code exists in the query information;
if the extraction code does not exist, acquiring query data in the first data table or the second data table or the relational database management system according to the query condition in the query information;
and if the extraction code exists, extracting the compressed data packet corresponding to the extraction code from the distributed open source database.
A distributed storage location data system, comprising:
the terminal comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring N pieces of position data reported by the terminal, and N is a positive integer greater than or equal to 1;
the data extraction module is used for extracting data of the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field;
and the processing module is used for storing the record information into a relational database management system.
Optionally, the method further includes:
the data import module is used for storing the position data acquired in the first time period into a first data table; and storing the position data acquired in a second time period after the first time period into a second data table.
Optionally, the data extraction module is specifically configured to analyze each piece of position data in the N pieces of position data to obtain position information, time information, alarm information, and state information of each piece of position data; integrating the position information of each piece of position data into a position information field, integrating the time information of each piece of position data into a time information field, integrating the alarm information of each piece of position data into an alarm information field, and integrating the state information of each piece of position data into a state information field; record information including a location information field, a time information field, an alert information field, and a status information field is generated.
Optionally, the data extraction module is further configured to perform data compression on the N pieces of position data, and obtain a data compression packet and an extraction code corresponding to the data compression packet, where the extraction code is used to extract the data compression packet; and storing the data compression packet to a distributed open source database, and adding the extraction code to the record information.
Optionally, the method further includes:
the analysis module is used for analyzing the acquired query information and detecting whether the extraction code exists in the query information;
the query module is used for acquiring query data in the first data table or the second data table or the relational database management system according to a query condition in the query information if the extraction code does not exist; and if the extraction code exists, extracting the compressed data packet corresponding to the extraction code from the distributed open source database.
The invention provides a method for storing position data in a distributed mode, which comprises the following steps: acquiring N pieces of position data reported by a terminal, and performing data extraction on the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field; and storing the record information into a relational database management system. In the method provided by the embodiment of the invention, the corresponding storage strategy can be adopted according to the characteristics of the position data, so that the access efficiency of the position data can be effectively reduced, and the position data can be read more conveniently.
Drawings
FIG. 1 is a flow chart of a method for storing location data in a distributed manner according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a distributed storage location data system according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a distributed storage position data method, which comprises the following steps: acquiring N pieces of position data reported by a terminal, and performing data extraction on the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field; and storing the record information into a relational database management system. In the method provided by the embodiment of the invention, the corresponding storage strategy can be adopted according to the characteristics of the position data, so that the access efficiency of the position data can be effectively reduced, and the position data can be read more conveniently.
The technical solutions of the present invention are described in detail with reference to the drawings and the specific embodiments, and it should be understood that the embodiments and the specific technical features in the embodiments of the present invention are merely illustrative of the technical solutions of the present invention, and are not restrictive, and the embodiments and the specific technical features in the embodiments of the present invention may be combined with each other without conflict.
Fig. 1 is a flowchart of a distributed storage location data method according to an embodiment of the present invention, where the method includes:
s101, acquiring N pieces of position data reported by a terminal;
s102, performing data extraction on the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field;
s103, storing the record information into a relational database management system.
First, in order to enable the obtained location data to be temporarily stored in the corresponding storage space in the embodiment of the present invention, the data obtained by the system is temporarily cached in a lightweight database Sqlite, which includes two data tables in the embodiment of the present invention, that is: the system stores the position data acquired in the first time period into the first data table and stores the position data acquired in the second time period after the first time period into the second data table.
For example, the system may store the location data acquired within the first 24 hours into the first data table, and after 24 hours, the system will switch and then store the location data within the subsequent 24 hours into the second data table. Therefore, the obtained position data can be guaranteed to be cached continuously, and the integrity and the accuracy of the position data are improved.
Certainly, the user can directly obtain the position data through the first data table and the second data table, so that the query efficiency can be guaranteed.
Further, in the embodiment of the present invention, after the system switches the cache of the location data from the first data table to the second data table, the system processes the location data in the first data table, and the specific processing procedure is as follows:
in the first time period, N pieces of position data are buffered in the first data table, where N may be a positive integer greater than or equal to 1.
The system analyzes each piece of position data in the N pieces of position data to obtain position information, time information, alarm information, state information and the like of each piece of position data. Of course, the corresponding license plate number, such as a license plate number or a ship license plate number, may also be analyzed, and the information category included in the location data is not limited in the embodiment of the present invention.
After the information corresponding to each piece of position data is analyzed, the system integrates the position information of each piece of position data in each piece of position data into one position information field, integrates the time information of each piece of position data into the time information field, integrates the alarm information of each piece of position data into the alarm information field, and integrates the state information of each piece of position data into the state information field. That is, the same information type constitutes a field, so that later inquiry can be more convenient.
Of course, after composing a field for the same type of information, the system will generate record information containing the location information field, time information field, alarm information field, and status information field, which will be saved into the relational database management system PostgreSQL.
When the data table is switched every time, the system executes the processing process, so that the data in the data table can be timely stored in the corresponding storage space, and the position data can be stored in different storage spaces in a distributed manner by the method, so that the distributed storage of the position data is realized.
Further, while storing the position data into PostgreSQL, the system compresses the N pieces of data, where the compression should be supported by the distributed database Hbase, and it should be ensured that the features of the position data are preserved under the condition of maximum compression ratio.
After the data compression is performed on the N pieces of position data to obtain a data compression packet, the system acquires an extraction code corresponding to the compression packet, and then stores the compression packet into the Hbase. Of course, the extraction code is added to the record information, for example, a separate field in the record information is used to place the extraction code, so as to ensure that the corresponding compressed packet can be directly found in the Hbase through the extraction code in the record information.
Of course, the position data for each switch-off can be stored in Hbase in the manner described above. Thus, the original position data can be conveniently inquired.
Further, in the embodiment of the present invention, after the system stores the location data in the corresponding database, the system may detect whether the query information of the terminal exists in real time, and when the query information is acquired, the system detects whether the query information includes the extraction code, and if the extraction code does not exist, the system acquires the query data in the first data table, the second data table, or the relational database management system, for example, if the query information includes a time point, the system manages the record information of the first data table, the second data table, or the relational database management system according to the time point to find out the corresponding location data, and returns the location data to the terminal.
Of course, in addition to querying according to time points, the system may also provide a classified query mode, such as: status information query, alarm information query, brand information query, etc., which can be used as a query condition to query the location data.
Of course, if the extraction code exists in the query information, the compressed data packet corresponding to the extraction code is extracted from the distributed open source database. That is to say, when the system extracts the extraction code from the query information, the system searches the compressed data packet corresponding to the extraction code in the distributed open source database directly according to the extraction code, and the original location data is stored in the compressed data packet.
Generally, in the embodiment of the present invention, a system obtains N pieces of position data reported by a terminal, and performs data extraction on the N pieces of position data to obtain record information at least including a position information field, a time information field, an alarm information field, and a state information field; and storing the record information into a relational database management system. In the method provided by the embodiment of the invention, the corresponding storage strategy can be adopted according to the characteristics of the position data, so that the access efficiency of the position data can be effectively reduced, and the position data can be read more conveniently.
Corresponding to a method for storing location data in a distributed manner in the embodiment of the present invention, an embodiment of the present invention further provides a location data system in a distributed manner, and as shown in fig. 2, the location data system in a distributed manner in the embodiment of the present invention is schematically configured, where the location data system includes:
an obtaining module 201, configured to obtain N pieces of position data reported by a terminal, where N is a positive integer greater than or equal to 1;
a data extraction module 202, configured to perform data extraction on the N pieces of position data to obtain record information at least including a position information field, a time information field, an alarm information field, and a state information field;
the processing module 203 is configured to store the record information in a relational database management system.
Further, in an embodiment of the present invention, the system further includes:
the data import module is used for storing the position data acquired in the first time period into a first data table; and storing the position data acquired in a second time period after the first time period into a second data table.
Further, in the embodiment of the present invention, the data extraction module 202 is specifically configured to analyze each piece of position data in the N pieces of position data to obtain position information, time information, alarm information, and state information of each piece of position data; integrating the position information of each piece of position data into a position information field, integrating the time information of each piece of position data into a time information field, integrating the alarm information of each piece of position data into an alarm information field, and integrating the state information of each piece of position data into a state information field; record information including a location information field, a time information field, an alert information field, and a status information field is generated.
Further, in this embodiment of the present invention, the data extraction module 202 is further configured to perform data compression on the N pieces of position data, and obtain a data compression packet and an extraction code corresponding to the data compression packet, where the extraction code is used to extract the data compression packet; and storing the data compression packet to a distributed open source database, and adding the extraction code to the record information.
Further, in an embodiment of the present invention, the system further includes:
the analysis module is used for analyzing the acquired query information and detecting whether the extraction code exists in the query information;
the query module is used for acquiring query data in the first data table or the second data table or the relational database management system according to a query condition in the query information if the extraction code does not exist; and if the extraction code exists, extracting the compressed data packet corresponding to the extraction code from the distributed open source database.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (4)
1. A method of distributed storage of location data, the method comprising:
storing the position data acquired in the first time period into a first data table;
storing position data acquired in a second time period which is continuous with the first time period and is after the first time period into a second data table;
alternately storing the acquired location data for the first data table and the second data table without interruption in time;
after the storage position of the position data is switched from the first data table to the second data table or from the second data table to the first data table, the following processing is carried out on the position data in the data table switched each time:
acquiring N pieces of position data reported by a terminal, wherein N is a positive integer greater than or equal to 1;
performing data extraction on the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field, wherein the step comprises the following steps of:
analyzing each piece of position data in the N pieces of position data to obtain position information, time information, alarm information and state information of each piece of position data;
integrating the position information of each piece of position data into a position information field, integrating the time information of each piece of position data into a time information field, integrating the alarm information of each piece of position data into an alarm information field, and integrating the state information of each piece of position data into a state information field;
generating record information containing a position information field, a time information field, an alarm information field and a state information field;
performing data compression on the N pieces of position data, and obtaining a data compression packet and an extraction code corresponding to the data compression packet, wherein the extraction code is used for extracting the data compression packet;
storing the data compression packet to a distributed open source database, and adding the extraction code to the recording information;
and storing the record information into a relational database management system.
2. The method of claim 1, wherein after storing the logging information in a relational database management system, the method further comprises:
analyzing the obtained query information, and detecting whether an extraction code exists in the query information;
if the extraction code does not exist, acquiring query data in the first data table or the second data table or the relational database management system according to the query condition in the query information;
and if the extraction code exists, extracting the compressed data packet corresponding to the extraction code from the distributed open source database.
3. A distributed storage location data system, comprising:
the data import module is used for storing the position data acquired in the first time period into a first data table; storing the position data acquired in a second time period after the first time period into a second data table; alternately storing the acquired location data for the first data table and the second data table without interruption in time;
the device comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring N pieces of position data reported by a terminal aiming at the position data in a data table switched every time after the storage position of the position data is switched from a first data table to a second data table or from the second data table to the first data table, wherein N is a positive integer greater than or equal to 1;
the data extraction module is used for extracting data of the N pieces of position data to obtain record information at least comprising a position information field, a time information field, an alarm information field and a state information field, integrating the position information of each piece of position data into one position information field, integrating the time information of each piece of position data into the time information field, integrating the alarm information of each piece of position data into the alarm information field, integrating the state information of each piece of position data into the state information field, and generating the record information comprising the position information field, the time information field, the alarm information field and the state information field; the device is further configured to perform data compression on the N pieces of position data, and obtain a data compression packet and an extraction code corresponding to the data compression packet, where the extraction code is used to extract the data compression packet; storing the data compression packet to a distributed open source database, and adding the extraction code to the recording information;
and the processing module is used for storing the record information into a relational database management system.
4. The system of claim 3, further comprising:
the analysis module is used for analyzing the acquired query information and detecting whether the extraction code exists in the query information;
the query module is used for acquiring query data in the first data table or the second data table or the relational database management system according to a query condition in the query information if the extraction code does not exist; and if the extraction code exists, extracting the compressed data packet corresponding to the extraction code from the distributed open source database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510836420.8A CN105468748B (en) | 2015-11-26 | 2015-11-26 | Distributed storage position data method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510836420.8A CN105468748B (en) | 2015-11-26 | 2015-11-26 | Distributed storage position data method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105468748A CN105468748A (en) | 2016-04-06 |
| CN105468748B true CN105468748B (en) | 2020-05-26 |
Family
ID=55606449
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510836420.8A Active CN105468748B (en) | 2015-11-26 | 2015-11-26 | Distributed storage position data method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105468748B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153744A (en) * | 2016-12-02 | 2018-06-12 | 上海中兴软件有限责任公司 | A kind of data storage system maintenance method and device |
| CN111930743A (en) * | 2020-07-29 | 2020-11-13 | 武汉中地先进技术研究院有限公司 | SQLite-based spatial data local storage method, medium and electronic device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104050269A (en) * | 2014-06-23 | 2014-09-17 | 上海帝联信息科技股份有限公司 | Log compression method and device and log decompression method and device |
| CN104778188A (en) * | 2014-02-24 | 2015-07-15 | 贵州电网公司信息通信分公司 | Distributed device log collection method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020205B (en) * | 2012-12-05 | 2018-07-31 | 中科天玑数据科技股份有限公司 | Compression/decompression method based on hardware accelerator card in a kind of distributed file system |
| CN104298695B (en) * | 2013-07-19 | 2020-06-16 | 腾讯科技（深圳）有限公司 | Data caching method and device and server |
| CN103607463B (en) * | 2013-11-25 | 2016-08-17 | 中国电信集团系统集成有限责任公司 | Location data-storage system and storage method |
| US9495466B2 (en) * | 2013-11-27 | 2016-11-15 | Oracle International Corporation | LIDAR model with hybrid-columnar format and no indexes for spatial searches |
-
2015
- 2015-11-26 CN CN201510836420.8A patent/CN105468748B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778188A (en) * | 2014-02-24 | 2015-07-15 | 贵州电网公司信息通信分公司 | Distributed device log collection method |
| CN104050269A (en) * | 2014-06-23 | 2014-09-17 | 上海帝联信息科技股份有限公司 | Log compression method and device and log decompression method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105468748A (en) | 2016-04-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104281672B (en) | Method and device for processing log data | |
| CN103607463B (en) | Location data-storage system and storage method | |
| CN113297135A (en) | Data processing method and device | |
| CN103702053A (en) | Video storage and search method and system as well as monitoring system | |
| CN104933175B (en) | Performance data correlation analysis method and performance monitoring system | |
| US9390170B2 (en) | Methods and systems for arranging and searching a database of media content recordings | |
| US20140280127A1 (en) | Methods and Systems for Arranging and Searching a Database of Media Content Recordings | |
| CN102436465A (en) | Telemetry data compression storage and rapid query method of ontrack spacecraft | |
| CN106792653B (en) | Communication number information processing method, device and system | |
| CN111241217B (en) | Data processing method, device and system | |
| CN105468748B (en) | Distributed storage position data method and system | |
| CN107391769B (en) | Index query method and device | |
| CN110321392A (en) | Data base management system based on sensor monitor data file | |
| KR20130129503A (en) | System and method of vehicle tracing | |
| CN112632568A (en) | Temperature data storage and acquisition method, system, electronic equipment and storage medium | |
| CN110909263B (en) | Method and device for determining companion relationship of identity characteristics | |
| CN102831174B (en) | Method and system for rapidly checking structured information | |
| CN105260465A (en) | Graph data processing service method and apparatus | |
| CN112784025A (en) | Method and device for determining target event | |
| CN111813773A (en) | Power grid meter reading data storage method, uploading method, device and storage device | |
| CN109271097A (en) | Data processing method, data processing equipment and server | |
| CN107992538B (en) | Message log generation method and device, query method and information processing system | |
| CN102752582A (en) | Intelligent video analysis system and method | |
| CN107070987B (en) | Data acquisition method and system for distributed object storage system | |
| US11386340B2 (en) | Method and apparatus for performing block retrieval on block to be processed of urine sediment image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |