CN113138991A - Vehicle state data storage system and storage method - Google Patents
Vehicle state data storage system and storage method Download PDFInfo
- Publication number
- CN113138991A CN113138991A CN202110539903.7A CN202110539903A CN113138991A CN 113138991 A CN113138991 A CN 113138991A CN 202110539903 A CN202110539903 A CN 202110539903A CN 113138991 A CN113138991 A CN 113138991A
- Authority
- CN
- China
- Prior art keywords
- data
- state data
- vehicle
- query
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Abstract
The invention belongs to the technical field of vehicle management, and particularly relates to a vehicle state data storage system which comprises a state data writing module, a state data storage module and a state data storage module, wherein the state data writing module is used for acquiring state data of a vehicle and writing the state data into different databases according to a vehicle ID; the state data filing module is used for grouping the data written into the database when the vehicle state data are uploaded less, synchronously filing the data to an OSS (open service system), and deleting the filed data in the database; and the state data query module is used for disassembling the query task and then querying related data in a database or an OSS. The defects in the prior art are overcome, concurrent writing of ten-thousand state data, storage of dozens of TB state data and quick query are achieved at lower hardware cost through a database and OSS combined method and a unique design writing, archiving and querying method.
Description
Technical Field
The invention belongs to the technical field of vehicle management, and particularly relates to a vehicle state data storage system and a vehicle state data storage method.
Background
With the rapid increase of the number of registered vehicles, one strip of state data is reported at intervals of 20 seconds, and about 5760 strips are reported for each vehicle every day. The data volume of a year is dozens of TB, and the writing speed of each second reaches tens of thousands of data. The original database storage mode is adopted, writing is frequently blocked, the query speed is slow, and the cost is high.
Disclosure of Invention
The invention aims to provide a vehicle state data storage system, which overcomes the defects of the prior art, and solves the problems of concurrent writing of ten-thousand-level state data, storage of dozens of TB state data and quick query at lower hardware cost by combining a database with an OSS (operating system) and by uniquely designing a writing, archiving and querying method.
In order to solve the problems, the technical scheme adopted by the invention is as follows:
a vehicle state data storage system includes
The state data writing module is used for acquiring the state data of the vehicle and writing the state data into different databases according to the ID of the vehicle;
the state data filing module is used for grouping the data written into the database when the vehicle state data are uploaded less, synchronously filing the data to an OSS (open service system), and deleting the filed data in the database;
and the state data query module is used for disassembling the query task and then querying related data in a database or an OSS.
Further, the status data writing module comprises a plurality of parallel writing units, each writing unit is composed of a configuration table, a status data storage table and a writing queue, the configuration table is composed of table name prefixes, writable flags and readable flag field entries, and the storage table is composed of a readable and writable status table and a non-writable status table.
Further, the status data archiving module includes a data grouping unit, a data uploading unit and a data deleting unit, the data grouping unit reads related data from the non-writable status table, sorts and groups the related data, the data uploading unit uploads the grouped data to the OSS for archiving, and the data deleting unit is configured to delete the archived data in the database.
Further, the state data query module comprises a task splitting unit, a data query unit and a data merging unit, wherein the task splitting unit is used for splitting a task according to the vehicle ID and the query time, the data query unit is used for querying and extracting data according to the split task, and the data merging unit is used for merging the queried data and then feeding back the merged data.
The invention also provides a vehicle state data storage method, which comprises the following steps:
(1) analyzing and processing the state data uploaded by the vehicle after receiving the state data, calculating a database number according to the vehicle ID, and adding the database number to a corresponding database write queue;
(2) the database write queue acquires a table name prefix of the writable table according to the parameter values of the configuration table, calculates a table index number according to the vehicle ID, generates table names at the same time, and writes the table names into the state data storage table in batches;
(3) when the uploading traffic of the vehicle state data is less, the writable and readable state in the table is adjusted;
(4) processing the state data storage table in batch, reading the state data in the table, grouping according to the vehicle ID and the recording year, month and day, and arranging according to the recording time in an ascending order;
(5) generating a key value by recording the year, month, day and vehicle ID; searching whether the same key data file exists on the OSS, if so, reading the data file, combining the data file with the data inquired in the step 3, converting the combined data into the data file, and updating the data file into the OSS; if the data does not exist, directly converting the data inquired in the step (3) into a data file and storing the data file into an OSS;
(6) and after all data are processed, the readable marks of the data storage table are modified, and the data in the archived state data storage table are deleted.
Further, the query of the archived data state specifically includes the following steps:
(1) receiving a state data query task, decomposing the query task into a plurality of subtasks according to a single vehicle and a single day according to the vehicle ID and the query time, and pushing the subtasks to a multi-task query queue;
(2) waiting for all the query subtasks to complete the query; and merging results of all the query subtasks and returning the results to the query task.
Compared with the prior art, the invention has the following beneficial effects:
according to the vehicle state data storage system, the vehicle state data in the database are synchronized to the OSS for long-term storage through the matching and allocation of the writing, filing and query modules, so that the problems of writing blockage and slow query speed are avoided, and the data storage efficiency is improved.
According to the vehicle state data storage method, the concurrent writing of ten-thousand-level state data, the storage of dozens of TB state data and the quick query are achieved at lower hardware cost through a database and OSS combined method and through a unique design writing, archiving and querying method.
Drawings
FIG. 1 is a functional block diagram of a vehicle state data storage system.
Fig. 2 is a schematic configuration diagram of a vehicle state data storage system.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
As shown in FIGS. 1-2, the present embodiment discloses a vehicle state data storage system including
The state data writing module is used for acquiring the state data of the vehicle and writing the state data into different databases according to the ID of the vehicle;
the state data filing module is used for grouping the data written into the database when the vehicle state data are uploaded less, synchronously filing the data to an OSS (open service system), and deleting the filed data in the database;
and the state data query module is used for disassembling the query task and then querying related data in a database or an OSS.
Further, the status data writing module comprises a plurality of parallel writing units, each writing unit is composed of a configuration table, a status data storage table and a writing queue, the configuration table is composed of table name prefixes, writable flags and readable flag field entries, and the storage table is composed of a readable and writable status table and a non-writable status table.
Further, the status data archiving module includes a data grouping unit, a data uploading unit and a data deleting unit, the data grouping unit reads related data from the non-writable status table, sorts and groups the related data, the data uploading unit uploads the grouped data to the OSS for archiving, and the data deleting unit is configured to delete the archived data in the database.
Further, the state data query module comprises a task splitting unit, a data query unit and a data merging unit, wherein the task splitting unit is used for splitting a task according to the vehicle ID and the query time, the data query unit is used for querying and extracting data according to the split task, and the data merging unit is used for merging the queried data and then feeding back the merged data.
Example 2
The embodiment discloses a vehicle state data storage method, which comprises the following steps:
(1) and after receiving the state data uploaded by the vehicle, analyzing and processing the state data, calculating a database number according to the vehicle ID, and adding the database number to a corresponding database write queue.
(2) The database write queue acquires the table name prefix of the writable table according to the parameter value of the configuration table (GPS _ ConfigParam), calculates the table index number according to the vehicle ID, generates the table name at the same time, and writes the table name into the state data storage table in batches.
Configuration table: GPS _ ConfigParam
| TablePrefix | WriteFlag | ReadFlag |
| GPS_HistoryData1 | 1 | 1 |
| GPS_HistoryData2 | 0 | 0 |
WriteFlag and ReadFlag fields 1 indicates writable or readable and 0 indicates non-writable or non-readable.
State data storage table 1: consists of a table name prefix GPS _ HistoryData1 and a table index number, GPS _ HistoryData1_ XX.
State data storage table 2: consists of a table name prefix GPS _ HistoryData2 and a table index number, GPS _ HistoryData2_ XX.
(3) Database number and table index number generation rules: the database number (vehicle ID mod 100) mod database number, and the table index number (vehicle ID/100) mod score table total.
(4) When the uploading traffic of the vehicle state data is less, the writable and readable state of the table in the configuration table (GPS _ ConfigParam) is adjusted, and the writable and readable table name prefix is GPS _ HistoryData1, the writable flag of GPS _ HistoryData1 is firstly synchronously changed into 0, and the writable flag and readable flag of GPS _ HistoryData2 are changed into 1.
(5) And a batch processing state data storage table GPS _ HistoryData1_ XX reads the state data in the table, groups the state data according to the vehicle ID and the recording year, month and day, and arranges the state data in ascending order according to the recording time.
(6) And recording the year, month, day and vehicle ID, and generating a key value according to the YYYYMMDD \ vehicle ID rule. And searching whether the same key data file exists from the OSS. If the data file exists, the data file is read and combined with the data inquired in the step 3, and the combined data is converted into the data file and updated into the OSS. If the data does not exist, the data inquired in the step 3 is directly converted into a data file and stored in the OSS.
(7) After all data processing is finished, the readable marks of the data storage table are modified, the data storage table is dormant for 2 minutes, and modification information is waited to take effect in the whole system; deleting data in the archived state data storage table.
Further, the query of the archived data state specifically includes the following steps:
(1) and receiving the state data query task, decomposing the query task into a plurality of subtasks according to the vehicle ID and the query time and according to a single vehicle and a single day, and pushing the subtasks to a multi-task query queue.
(2) Waiting for all the query subtasks to complete the query; and merging results of all the query subtasks and returning the results to the query task.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (6)
1. A vehicle state data storage system characterized in that: comprises that
The state data writing module is used for acquiring the state data of the vehicle and writing the state data into different databases according to the ID of the vehicle;
the state data filing module is used for grouping the data written into the database when the vehicle state data are uploaded less, synchronously filing the data to an OSS (open service system), and deleting the filed data in the database;
and the state data query module is used for disassembling the query task and then querying related data in a database or an OSS.
2. A vehicle state data storage system according to claim 1, characterized in that: the state data writing module comprises a plurality of parallel writing units, each writing unit consists of a configuration table, a state data storage table and a writing queue, each configuration table consists of a table name prefix, a writable mark and a readable mark field item, and each storage table consists of a readable and writable state table and a non-writable state table.
3. A vehicle state data storage system according to claim 1, characterized in that: the status data archiving module comprises a data grouping unit, a data uploading unit and a data deleting unit, wherein the data grouping unit reads related data from the non-writable status table, sequences and groups the related data, the data uploading unit uploads the grouped data to the OSS for archiving, and the data deleting unit is used for deleting the archived data in the database.
4. A vehicle state data storage system according to claim 1, characterized in that: the state data query module comprises a task splitting unit, a data query unit and a data merging unit, wherein the task splitting unit is used for splitting a task according to a vehicle ID and query time, the data query unit is used for querying and extracting data according to the split task, and the data merging unit is used for merging the queried data and feeding back the merged data.
5. A vehicle state data storage method characterized by: the method comprises the following steps:
(1) analyzing and processing the state data uploaded by the vehicle after receiving the state data, calculating a database number according to the vehicle ID, and adding the database number to a corresponding database write queue;
(2) the database write queue acquires a table name prefix of the writable table according to the parameter values of the configuration table, calculates a table index number according to the vehicle ID, generates table names at the same time, and writes the table names into the state data storage table in batches;
(3) when the uploading traffic of the vehicle state data is less, the writable and readable state in the table is adjusted;
(4) processing the state data storage table in batch, reading the state data in the table, grouping according to the vehicle ID and the recording year, month and day, and arranging according to the recording time in an ascending order;
(5) generating a key value by recording the year, month, day and vehicle ID; searching whether the same key data file exists on the OSS, if so, reading the data file, combining the data file with the data inquired in the step 3, converting the combined data into the data file, and updating the data file into the OSS; if the data does not exist, directly converting the data inquired in the step (3) into a data file and storing the data file into an OSS;
(6) and after all data are processed, the readable marks of the data storage table are modified, and the data in the archived state data storage table are deleted.
6. A vehicle state data storage method according to claim 5, characterized in that: the method for querying the archived data state specifically comprises the following steps:
(1) receiving a state data query task, decomposing the query task into a plurality of subtasks according to a single vehicle and a single day according to the vehicle ID and the query time, and pushing the subtasks to a multi-task query queue;
(2) waiting for all the query subtasks to complete the query; and merging results of all the query subtasks and returning the results to the query task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110539903.7A CN113138991A (en) | 2021-05-18 | 2021-05-18 | Vehicle state data storage system and storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110539903.7A CN113138991A (en) | 2021-05-18 | 2021-05-18 | Vehicle state data storage system and storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113138991A true CN113138991A (en) | 2021-07-20 |
Family
ID=76817232
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110539903.7A Pending CN113138991A (en) | 2021-05-18 | 2021-05-18 | Vehicle state data storage system and storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113138991A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101799832A (en) * | 2010-03-31 | 2010-08-11 | 山东高效能服务器和存储研究院 | Method for rapidly writing mass data into database once |
| CN109710614A (en) * | 2018-12-28 | 2019-05-03 | 深圳市同行者科技有限公司 | A kind of method and device of real-time data memory and inquiry |
| US20190294505A1 (en) * | 2018-03-20 | 2019-09-26 | Sap Se | Object store for database backups |
| CN110311953A (en) * | 2019-05-24 | 2019-10-08 | 杭州网络传媒有限公司 | A kind of media article uploads and storage system and method |
| CN110351532A (en) * | 2019-08-08 | 2019-10-18 | 杭州阿启视科技有限公司 | Video big data cloud platform cloud storage service method |
| CN110389939A (en) * | 2019-02-18 | 2019-10-29 | 华南理工大学 | A kind of Internet of Things storage system based on NoSQL and distributed file system |
| CN111741105A (en) * | 2020-06-18 | 2020-10-02 | 华域汽车系统股份有限公司 | Real-time transmission system and method for problem data of intelligent driving system |
-
2021
- 2021-05-18 CN CN202110539903.7A patent/CN113138991A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101799832A (en) * | 2010-03-31 | 2010-08-11 | 山东高效能服务器和存储研究院 | Method for rapidly writing mass data into database once |
| US20190294505A1 (en) * | 2018-03-20 | 2019-09-26 | Sap Se | Object store for database backups |
| CN109710614A (en) * | 2018-12-28 | 2019-05-03 | 深圳市同行者科技有限公司 | A kind of method and device of real-time data memory and inquiry |
| CN110389939A (en) * | 2019-02-18 | 2019-10-29 | 华南理工大学 | A kind of Internet of Things storage system based on NoSQL and distributed file system |
| CN110311953A (en) * | 2019-05-24 | 2019-10-08 | 杭州网络传媒有限公司 | A kind of media article uploads and storage system and method |
| CN110351532A (en) * | 2019-08-08 | 2019-10-18 | 杭州阿启视科技有限公司 | Video big data cloud platform cloud storage service method |
| CN111741105A (en) * | 2020-06-18 | 2020-10-02 | 华域汽车系统股份有限公司 | Real-time transmission system and method for problem data of intelligent driving system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111339103B (en) | Data exchange method and system based on full-quantity fragmentation and incremental log analysis | |
| CN103853820A (en) | Data processing method and data processing system | |
| CN102467521A (en) | Easily-extensible multi-level classification search method and system | |
| CN106503214A (en) | A kind of complex rule matching process based on Redis memory databases | |
| CN110096509A (en) | Realize that historical data draws the system and method for storage of linked list modeling processing under big data environment | |
| CN103377210A (en) | Method for creating incremental navigation database and method for updating same | |
| CN105630934A (en) | Data statistic method and system | |
| CN107783974B (en) | Data processing system and method | |
| CN105787058A (en) | User label system and data pushing system based on same | |
| CN101963993B (en) | Method for fast searching database sheet table record | |
| CN116166849A (en) | Data management method, device, equipment and storage medium | |
| CN108984626B (en) | Data processing method and device and server | |
| CN111008183A (en) | Storage method and system for business wind control log data | |
| CN113138991A (en) | Vehicle state data storage system and storage method | |
| CN113094442A (en) | Full data synchronization method, device, equipment and medium | |
| CN111813773A (en) | Power grid meter reading data storage method, uploading method, device and storage device | |
| CN116701355A (en) | Data view processing method, device, computer equipment and readable storage medium | |
| US20200278980A1 (en) | Database processing apparatus, group map file generating method, and recording medium | |
| CN109902148A (en) | A kind of method of the enterprise name auto-complete of address book contact | |
| CN115344603A (en) | Freight rate data storage method and device, electronic equipment and computer storage medium | |
| CN111414382A (en) | Slow SQ L polymerization display method and system based on MongoDB | |
| CN113570444A (en) | Non-tax payment report processing method and query system based on non-tax payment book | |
| CN112684986A (en) | Mass data processing method | |
| CN112084340A (en) | Management method for organizational structure change version based on neo4j graph database | |
| CN101191730A (en) | Information search system of autocar navigation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210720 |
|
| RJ01 | Rejection of invention patent application after publication |