CN115617770B - Data disk storage management method for vehicle state signal data storage - Google Patents
Data disk storage management method for vehicle state signal data storage Download PDFInfo
- Publication number
- CN115617770B CN115617770B CN202211440899.XA CN202211440899A CN115617770B CN 115617770 B CN115617770 B CN 115617770B CN 202211440899 A CN202211440899 A CN 202211440899A CN 115617770 B CN115617770 B CN 115617770B
- Authority
- CN
- China
- Prior art keywords
- data
- vehicle
- item
- memory
- disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a data disk storage management method for storing vehicle state signal data, which comprises the following steps: defining item as BSON format, and associating a plurality of items by a bidirectional linked list structure; the vehicle reported signal data is stored in different vehicle files through a specified classification column; recording the memory positions of the previous element and the next element; the log system is used for recording logs on the operation command and the time node in the inserted data; creating a buffer area in the memory, and controlling the frequency of writing data into a disk according to the time interval and the size of the buffer area; querying data according to the name space and the classification condition; and the synchronous backup of the data in distributed deployment is realized by transmitting the operation log. According to the invention, different vehicle data are stored in different vehicle files, and the vehicle files are stored in different disk space positions, so that the data information of a specified vehicle in a specified time segment can be conveniently inquired under large data volume; the database is matched with the car machine, and the data structure can be used in a targeted extension mode at the later stage.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a data disk storage management method for vehicle state signal data storage.
Background
With the coming of the internet era and the rapid development of network technologies, various network applications become more and more popular, data collection and analysis related to internet devices become especially important for users, and storage management of databases becomes an important influencing factor in data collection and analysis work.
Currently, the mainstream databases are divided into two major types according to the data storage mode: relational and non-relational NoSQL, which can satisfy most application service scenarios. In order to improve the operation efficiency, the existing NoSQL (non-relational database) method for increasing the I/O frequency is used for putting data into a memory for operation, and a part of operation is to use a single thread to perform data operation so as to avoid the thread safety problem and the performance loss caused by thread context switching; in the aspect of a data persistence mechanism, a serialized binary JSON mode or a custom storage structure is used, and log recording is used for preventing the loss of the recorded data after the database is crashed and restarted.
However, with the continuous development of current scientific technology including the internet industry, the existing databases cannot meet the requirements of data storage performance for data collection related operations of some new internet devices.
For a new energy vehicle application service scene, in order to ensure that a new energy vehicle runs safely and stably, a vehicle body firmware signal needs to be monitored and analyzed in real time, the reporting frequency of vehicle body firmware signal data is very frequent after the vehicles are produced in batches, and the magnitude scale of the data is very large.
However, the existing database technology cannot perform customized analysis on data of a specified size, and query efficiency is low when the data size is too large (for example, data size of hundreds of millions).
Disclosure of Invention
In view of this, the present invention aims to provide a data disk storage management method, which performs fragmentation storage on data of different users or internet devices in the same service, improves data storage performance, solves storage security of a large amount of signal data, and improves accurate query efficiency of a large time slice in data of more than ten million levels; and the data of a single user and a single Internet device can be directionally analyzed according to the service requirement, and the data of the whole quantity of data can be analyzed, so that the analysis result can be obtained more quickly, and intelligent analysis support is provided for the fault signal.
The invention provides a data disk storage management method for storing vehicle state signal data, which comprises the following steps:
s1, constructing a data set collection of signal data reported by vehicles, defining the data format of an item data item of each vehicle file in the data set as a BSON (Binary-json) format, storing vehicle state signal data of a time point for each item data item, and storing all signal data reported by the vehicles through a plurality of item data items, wherein the item data items are used for storing the signal data reported by the vehicles
A plurality of the item data items in each vehicle file are associated in a doubly linked list structure (previous item pointer, next item pointer);
all signal data reported by the vehicle comprise: vehicle state, battery voltage, battery health, speed, and discharge state;
s2, storing the vehicle reported signal data stored in each data set collection into different vehicle files through a specified classification column, wherein the vehicle reported signal data are stored in the data sets collection
Integrating a plurality of vehicle files consisting of item data items in the same name space to form a record table;
the namespace is a folder path of a data file for storing vehicle state signal data and is stored in a magnetic disk; the data files can be stored by a multi-level folder, and the data files are dispersed according to time and vehicles;
s3, recording the memory positions of the previous item data element and the next item data element in the item data item, wherein
If the item data element is a head node or a tail node, the last item pointer of the head node is null, and the next item pointer of the tail node is null; the item pointer is null i.e. no data element is deposited,
the head node of the item data item stores whether the current item data item is complete data,
if the data is incomplete data, the last page of the incomplete data is connected with the information of the vehicle state signal data,
the data content of the item data item stores the real data of the vehicle state signal data,
the tail node of the item data item stores whether the item data item is complete data;
s4, in the process of inserting data into the data set, a log system is used for carrying out log recording on each operation command and each time node; and recording the last data content before the current operation, wherein
When a problem occurs, recovering the last operation or rolling back to the correct version of data through a log record file;
in the process of inserting data, the data can not be correctly written into a disk because a CPU does not normally operate, and in order to ensure the safety of the inserted data, a log system is used for carrying out log recording on each operation command and time node; for example, when data XXXX is inserted, a log is recorded first, and specific operation content of the operation is recorded, and then the operation content is written into a disk, and when a problem occurs in writing, recovery can be performed according to the log; recording the last data content before the operation, for example, if an original value of a certain data store is 1, and the result after the operation is 2, recording the original value 1 in the log, and also recording the data after the operation 2;
s5, creating a buffer area in the memory, controlling the frequency of data writing into a disk according to the time interval and the size of the buffer area, firstly recording the operation on the data into a log, and then writing the data into the disk when the data in the buffer memory reaches the set proportion or the set time interval so as to improve the response speed, wherein the buffer area is created in the memory, the operation on the data is written into the log, and the data is written into the disk when the data in the buffer memory reaches the set proportion or the set time interval, and the response speed is improved, wherein the buffer area is created in the memory, and the data is written into the disk according to the time interval and the size of the buffer area
The buffer area provides an independent log to ensure the data recovery after the service crash;
in the process of writing a large amount of data, if the data is directly written into a disk, the speed is slow, in order to ensure the efficiency of disk writing, a buffer area is created in a memory, the operation on the data is firstly recorded into a log, and then the data is written into the disk when the data in the cache reaches a set proportion, such as 80% or every 5 s;
s6, vehicle data query operation is carried out according to the name space and the classification conditions, item data items of the vehicle files are loaded into a memory, and matching, sorting and aggregation operation are carried out on the memory, wherein
If a plurality of types of contents are required to be comprehensively inquired, performing fragment type result screening in a streaming loading mode, and performing final integration, sequencing and paging operations on a result set;
and S7, realizing synchronous backup of the data in distributed deployment by transmitting the operation log.
Further, the method for logging each operation command and time node by using the logging system in the step S4 includes:
the method comprises the steps of selecting a database to be cooperatively recorded with a vehicle machine, wherein the vehicle machine stores data information in a short time, and when data loss occurs, the database synchronously loses data through the vehicle machine, so that corresponding recovery measures can be provided under the condition that logs of a database server are lost, the logs are preserved consistently through the vehicle machine in a two-way mode, high data safety is achieved, and the condition that vehicle state signal data recording is guaranteed to be constant.
Further, the method for writing data into a disk in the step S5 includes:
storing the data into different vehicle files through the content of the primary key of the data;
for example, three types of data A, B, C are stored in the primary key, and the data of the three primary keys are stored in three files, a ' \ B ' \ C '.
Further, the method for performing fragmented result screening by using a streaming loading manner in the S6 step includes:
finding a corresponding disk storage address according to a namespace, then finding a file in the corresponding storage address according to a queried primary key, namely a set of stored item data items, loading the item data items in the file into a memory in batches by using a fragment loading mode, performing corresponding condition filtering (such as screening and the like) on the file in the memory, performing full operations of integration, sequencing and paging on the data retained in the memory, and returning a result in the memory to a caller.
Further, the method of the vehicle data query operation of the S6 step includes the steps of:
s61, finding a corresponding vehicle file storage directory according to the search condition of the vehicle: a part of paths of the namespace, searching folders grouped according to time under the vehicle file storage directory: the vehicle folder and the time folder form a complete naming space until the position of the corresponding data file is found;
s62, directly jumping to the corresponding line number of the starting time according to the index information stored in the data file, and loading all effective files into a memory, wherein
In the process of loading the effective files into the memory, if more searching conditions exist (for example, the electric quantity is more than 10 percent), directly filtering data which do not meet the conditions, and if the sorting conditions exist, sorting;
s63, integrating all results, and splicing according to a sorting rule;
and S64, returning the data to the client.
Further, the method for implementing synchronous backup of data deployed in a distributed manner in step S7 includes:
and selecting master and slave nodes for the distributed deployed multi-node through a Raft algorithm.
For example, 2 nodes a and B, where a is a master node and is responsible for writing data, B is a slave node and is responsible for backing up data, when a has data change, an operation log is transmitted to B through network I/O, and B synchronizes data by reading the operation log.
Further, the step S5 of obtaining the data in the cache at a set ratio or at a set time interval is: the data in the buffer reaches 80% or every 5s.
Further, the classification condition of the S6 step includes: the content of the primary key, wherein the primary key is the type identification of the data. Each data has a primary key, the primary key is a type identifier of the data, and the content of the primary key can be used as a classification condition.
The present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the data disk storage management method for vehicle status signal data storage as described above.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the data disk storage management method for vehicle status signal data storage as described above when executing the program.
Compared with the prior art, the invention has the beneficial effects that:
according to the invention, the related data of different vehicles are stored in different vehicle files, the vehicle files are stored in different disk space positions, and the data information of the designated vehicle in the designated time segment can be conveniently inquired under large data volume by defining the definition of the storage format and adopting the log record, so that the inquiry efficiency is improved; the database can be matched with a vehicle machine, and the data structure can be expanded and used in targeted applications such as failure analysis statistics and the like of vehicle machine parts at the later stage.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
In the drawings:
FIG. 1 is a schematic diagram of the structure of a data set according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a computer device according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating the structure of an item data item according to an embodiment of the present invention;
FIG. 4 is a flow chart of a data disc storage management method for vehicle status signal data storage in accordance with the present invention;
FIG. 5 is a flowchart of a method of vehicle data query operation according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and products consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if," as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination," depending on the context.
The embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
An embodiment of the present invention provides a data disk storage management method for storing vehicle status signal data, and as shown in fig. 4, the method includes the following steps:
s1, constructing a data set collection of signal data reported by vehicles, defining the data format of an item data item of each vehicle file in the data set as a BSON (Binary-json) format, storing vehicle state signal data of a time point for each item data item, and storing all signal data reported by the vehicles through a plurality of item data items, wherein the item data items are used for storing the signal data reported by the vehicles
A plurality of the item data items in each vehicle file are associated in a doubly linked list structure (last item pointer, next item pointer);
the specific structure of the item data item of this embodiment is shown in fig. 3;
all signal data reported by the vehicle comprise: vehicle state, battery voltage, battery health, speed, and discharge state;
s2, storing the vehicle reported signal data stored in each data set collection into different vehicle files through a specified classification column, wherein the vehicle reported signal data are stored in each data set collection
A plurality of vehicle files composed of item data items are finally integrated in the same namespace (collection namespace) (see fig. 1) to form a record table;
the namespace is a folder path of a data file for storing vehicle state signal data and is stored in a magnetic disk; the data files can be stored by a multi-level folder, and the data files are dispersed according to time and vehicles;
s3, recording the memory positions of the previous item data element and the next item data element in the item data item, wherein
If the item data element is a head node or a tail node, the last item pointer of the head node is null, and the next item pointer of the tail node is null; the item pointer is null or does not deposit a data element,
the head node of the item data item stores whether the current item data item is complete data,
if the data is incomplete data, the last page of the incomplete data is connected with the information of the vehicle state signal data,
the data content of the item data item stores the real data of the vehicle state signal data,
the tail node of the item data item stores whether the item data item is complete data;
s4, in the process of inserting data into the data set, a log system is used for carrying out log recording on each operation command and each time node; and recording the last data content before the current operation, wherein
When a problem occurs, recovering the last operation or rolling back to the correct version of data through a log record file;
in the process of inserting data, the data can not be correctly written into a disk because a CPU does not normally operate, and in order to ensure the safety of the inserted data, a log system is used for carrying out log recording on each operation command and time node; for example, when data XXXX is inserted, a log is recorded first, and specific operation content of the operation is recorded, and then the operation content is written into a disk, and when a problem occurs in writing, recovery can be performed according to the log; recording the last data content before the operation, for example, if an original value of a certain data store is 1, and the result after the operation is 2, recording the original value 1 in the log, and also recording the data after the operation 2;
the method for logging each operation command and time node by using the logging system comprises the following steps:
the method comprises the steps that a database and a vehicle machine are selected for collaborative recording, wherein the vehicle machine stores data information in a short time, and when data loss occurs, the database synchronously loses data through the vehicle machine, so that corresponding recovery measures can be provided under the condition that logs of a database server are lost, and the vehicle machine bidirectional logs are kept consistently to achieve high data safety so as to ensure the recording of vehicle state signal data;
s5, creating a buffer area in the memory, controlling the frequency of data writing into a disk according to the time interval and the size of the buffer area, firstly recording the operation on the data into a log, and then writing the data into the disk when the data in the buffer memory reaches the set proportion or the set time interval so as to improve the response speed, wherein the buffer area is created in the memory, the operation on the data is written into the log, and the data is written into the disk when the data in the buffer memory reaches the set proportion or the set time interval, and the response speed is improved, wherein the buffer area is created in the memory, and the data is written into the disk according to the time interval and the size of the buffer area
The buffer provides a separate log to ensure data recovery after a service crash;
in the process of writing a large amount of data, if the data is directly written into a disk, the speed is slow, in order to ensure the efficiency of disk writing, a buffer area is created in a memory, the operation on the data is firstly recorded into a log, and then the data is written into the disk when the data in the cache reaches a set proportion, such as 80% or every 5 s;
the method for writing data into the magnetic disk comprises the following steps:
storing the data into different vehicle files through the content of the primary key of the data;
in this embodiment, three types of data A, B, C are stored in the primary key, and the data of the three types of primary keys are stored in three files, namely, a ' \ B ' \ C ';
s6, vehicle data query operation is carried out according to the name space and the classification conditions, item data items of the vehicle files are loaded into a memory, and matching, sorting and aggregation operation are carried out on the memory, wherein
If a plurality of types of contents are required to be comprehensively inquired, performing fragment type result screening in a streaming loading mode, and performing final integration, sequencing and paging operations on a result set;
the classification conditions include: the content of the primary key, wherein the primary key is the type identification of the data; each data has a primary key, the primary key is a type identifier of the data, and the content of the primary key can be used as a classification condition;
the method for performing fragment-type result screening by using a streaming loading mode comprises the following steps:
finding a corresponding disk storage address according to a namespace, then finding a file in the corresponding storage address according to an inquired primary key, referring to a vertical column (collection of item data items) in fig. 1, loading the item data items in the file into a memory in batches by using a fragment loading mode, performing corresponding condition filtering (such as screening) on the file in the memory, performing full operation of integration, sequencing and paging on the data retained in the memory, and returning the result in the memory to a caller;
in this embodiment, query needs to be performed 22 years, 9 months, 1 day, 10: 9 months, 3 days, 10 from 00 to 22 years: 00, as shown in fig. 5, the method of the vehicle data query operation includes the steps of:
s61, finding a corresponding vehicle file storage directory according to the search condition of the vehicle: a part of paths of the namespace, searching folders grouped according to time under the vehicle file storage directory: the vehicle folder and the time folder form a complete naming space until the position of the corresponding data file is found;
s62, directly jumping to the corresponding line number of the starting time according to the index information stored in the data file, and loading all effective files into a memory, wherein
In the process of loading the effective files into the memory, if more searching conditions exist (for example, the electric quantity is more than 10 percent), directly filtering data which do not meet the conditions, and if the sorting conditions exist, sorting;
s63, integrating all results, and splicing according to a sorting rule;
s64, returning data to the client;
and S7, realizing synchronous backup of the data in distributed deployment by transmitting the operation log.
The method for implementing the synchronous backup of the data deployed in the distributed manner in the step S7 includes:
selecting master and slave nodes for the multi-node distributed deployment through a Raft algorithm;
in this embodiment, there are 2 nodes a and B, where a is a master node and is responsible for writing data, and B is a slave node and is responsible for backing up data, and when data change occurs in a, an operation log is transmitted to B through network I/O, and B synchronizes data by reading the operation log.
According to the embodiment of the invention, the relevant data of different vehicles are stored in different vehicle files, the vehicle files are stored in different disk space positions, and the data information of the specified vehicle in the specified time segment can be conveniently inquired under large data volume by defining the definition of the storage format and adopting the log record, so that the inquiry efficiency is improved; the database can be matched with a vehicle machine, and the data structure can be expanded and used in targeted applications such as failure analysis statistics of vehicle machine parts at the later stage.
Fig. 2 is a schematic structural diagram of a computer device provided in an embodiment of the present invention; referring to fig. 2 of the drawings, the computer apparatus comprises: an input device 23, an output device 24, a memory 22, and a processor 21; the memory 22 for storing one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 are caused to implement the data disk storage management method for vehicle status signal data storage as provided in the above embodiments; wherein the input device 23, the output device 24, the memory 22 and the processor 21 may be connected by a bus or other means, as exemplified by the bus connection in fig. 2.
The memory 22, which is a computer-readable and writable storage medium, may be used for storing a software program, a computer-executable program, and program instructions corresponding to the data disk storage management method for vehicle status signal data storage according to the embodiment of the present invention; the memory 22 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like; further, the memory 22 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device; in some examples, the memory 22 may further include memory located remotely from the processor 21, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 23 may be used to receive input numeric or character information and to generate key signal inputs relating to user settings and function control of the apparatus; the output device 24 may include a display device such as a display screen.
The processor 21 executes various functional applications of the device and data processing by executing software programs, instructions and modules stored in the memory 22, namely, implements the above-described data disk storage management method for vehicle state signal data storage.
The computer device provided above can be used to execute the data disk storage management method for vehicle status signal data storage provided in the above embodiments, and has corresponding functions and advantages.
Embodiments of the present invention also provide a storage medium containing computer-executable instructions for performing a data disk storage management method for vehicle status signal data storage as provided in the above embodiments when executed by a computer processor, the storage medium being any of various types of memory devices or storage devices, the storage medium comprising: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or combinations thereof; in addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet); the second computer system may provide program instructions to the first computer for execution. A storage medium includes two or more storage media that may reside in different locations, such as in different computer systems that are connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the data disk storage management method for vehicle status signal data storage described in the above embodiment, and may also perform related operations in the data disk storage management method for vehicle status signal data storage provided by any embodiment of the present invention.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can be within the protection scope of the invention.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention; various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data disk storage management method for vehicle status signal data storage, comprising the steps of:
s1, constructing a data set collection of signal data reported by a vehicle, defining the data format of an item data item of each vehicle file in the data set as a BSON format, storing the vehicle state signal data of a time point in each item data item, and storing all the signal data reported by the vehicle through a plurality of item data items, wherein the item data items are used for storing the signal data reported by the vehicle
A plurality of item data items in each vehicle file are associated in a doubly linked list structure, and all signal data reported by the vehicle comprises: vehicle state, battery voltage, battery health, speed, and discharge state;
s2, storing the vehicle reported signal data stored in each data set collection into different vehicle files through a specified classification column, wherein the vehicle reported signal data are stored in each data set collection
Integrating a plurality of vehicle files formed by item data items in the same name space to form a record table, wherein the name space is a folder path of the data files for storing vehicle state signal data and is stored in a magnetic disk;
s3, recording the memory positions of the previous item data element and the next item data element in the item data item, wherein
If the item data element is a head node or a tail node, the last item pointer of the head node is null, the next item pointer of the tail node is null,
the head node of the item data item stores whether the current item data item is complete data, if the current item data item is incomplete data, the last page of the incomplete data is connected with the information of the vehicle state signal data,
the data content of the item data item stores the real data of the vehicle state signal data,
the tail end point of the item data item stores whether the item data item is complete data;
s4, in the process of inserting data into the data set, a log system is used for logging each operation command and each time node and recording the last data content before the operation, wherein the log system is used for recording the log of each operation command and each time node, and the log system is used for recording the last data content before the operation
When a problem occurs, recovering the last operation or rolling back to the data of the correct version through the log record file;
s5, creating a buffer area in the memory, controlling the frequency of data writing into a disk according to the time interval and the size of the buffer area, firstly recording the operation on the data into a log, and then writing the data into the disk when the data in the buffer memory reaches the set proportion or the set time interval so as to improve the response speed, wherein the buffer area is created in the memory, the operation on the data is written into the log, and the data is written into the disk when the data in the buffer memory reaches the set proportion or the set time interval, and the response speed is improved, wherein the buffer area is created in the memory, and the data is written into the disk according to the time interval and the size of the buffer area
The buffer provides a separate log to ensure data recovery after a service crash;
s6, vehicle data query operation is carried out according to the name space and the classification conditions, item data items of the vehicle files are loaded into a memory, and matching, sorting and aggregation operation are carried out on the memory, wherein
If a plurality of types of contents are required to be comprehensively inquired, performing fragment type result screening in a streaming loading mode, and performing final integration, sequencing and paging operations on a result set;
and S7, realizing synchronous backup of the data deployed in a distributed manner by transmitting the operation log.
2. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method of logging each operation command and time node using a logging system of the step S4 comprises:
and selecting a database to perform cooperative recording with the vehicle machine, wherein the vehicle machine stores data information in a short time, and when data loss occurs, the database synchronously loses the data through the vehicle machine.
3. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method of writing data to a disk of the S5 step comprises:
and storing the data into different vehicle files through the content of the primary key of the data.
4. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method for performing segmented result filtering using streaming loading in step S6 comprises:
finding out the corresponding disk storage address according to the name space, then finding out the file in the corresponding storage address according to the inquired main key, loading item data items in the file into the memory in batches by using a segment loading mode, carrying out corresponding condition filtering on the file in the memory, carrying out full operations of integration, sequencing and paging on the data retained in the memory, and returning the result in the memory to a calling party.
5. The data disk storage management method for vehicle status signal data storage according to claim 4, wherein said vehicle data query operation of step S6 comprises the steps of:
s61, finding a corresponding vehicle file storage directory according to the search condition of the vehicle: a part of paths of the namespace, searching folders grouped according to time under the vehicle file storage directory: the vehicle folder and the time folder form a complete naming space until the position of the corresponding data file is found;
s62, directly jumping to the corresponding line number of the starting time according to the index information stored in the data file, and loading all effective files into a memory, wherein in the process of loading the effective files into the memory, if more search conditions exist, data which do not meet the conditions are directly filtered out, and if a sorting condition exists, sorting is carried out;
s63, integrating all results, and splicing according to a sorting rule;
and S64, returning the data to the client.
6. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method for implementing synchronous backup of data deployed in a distributed manner in the step S7 comprises:
and selecting master and slave nodes for the multi-node distributed deployment through a Raft algorithm.
7. The data disk storage management method for storing the vehicle status signal data as claimed in claim 1, wherein the data in the buffer memory of the step S5 reaches a set proportional size or reaches a set time interval: the data in the buffer reaches 80% or every 5s.
8. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the classification condition of the S6 step comprises: the content of the primary key, the primary key is the type identification of the data.
9. A computer-readable storage medium, having stored thereon a computer program, the program being adapted to carry out the steps of the data disk storage management method for vehicle status signal data storage of any one of claims 1 to 8 when executed by a processor.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of a data disc storage management method for vehicle status signal data storage according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211440899.XA CN115617770B (en) | 2022-11-17 | 2022-11-17 | Data disk storage management method for vehicle state signal data storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211440899.XA CN115617770B (en) | 2022-11-17 | 2022-11-17 | Data disk storage management method for vehicle state signal data storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115617770A CN115617770A (en) | 2023-01-17 |
| CN115617770B true CN115617770B (en) | 2023-03-28 |
Family
ID=84878721
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211440899.XA Active CN115617770B (en) | 2022-11-17 | 2022-11-17 | Data disk storage management method for vehicle state signal data storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115617770B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104516955A (en) * | 2014-12-16 | 2015-04-15 | 北京中交兴路车联网科技有限公司 | Massive vehicle-mounted machine track data storage method |
| US20160171793A1 (en) * | 2014-12-11 | 2016-06-16 | Hyundai Motor Company | Apparatus for processing a plurality of logging policies and method thereof |
| CN109445703A (en) * | 2018-10-26 | 2019-03-08 | 黄淮学院 | A kind of Delta compression storage assembly based on block grade data deduplication |
| CN115048149A (en) * | 2022-05-23 | 2022-09-13 | 杨正 | Application cache scalable processing method, device, equipment and medium |
-
2022
- 2022-11-17 CN CN202211440899.XA patent/CN115617770B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160171793A1 (en) * | 2014-12-11 | 2016-06-16 | Hyundai Motor Company | Apparatus for processing a plurality of logging policies and method thereof |
| CN104516955A (en) * | 2014-12-16 | 2015-04-15 | 北京中交兴路车联网科技有限公司 | Massive vehicle-mounted machine track data storage method |
| CN109445703A (en) * | 2018-10-26 | 2019-03-08 | 黄淮学院 | A kind of Delta compression storage assembly based on block grade data deduplication |
| CN115048149A (en) * | 2022-05-23 | 2022-09-13 | 杨正 | Application cache scalable processing method, device, equipment and medium |
Non-Patent Citations (4)
| Title |
|---|
| Jijin Wang 等."Vehicle Network Node Behavior Classification based on Optimized Bayesian Classifier".《Proceedings of 5th International Conference on Vehicle》.2019,全文. * |
| 张春风."非结构化车联网大数据存储与处理技术研究与应用".《中国优秀硕士学位论文全文数据库 工程科技II辑》.2019,全文. * |
| 韩印 等."快速公交多模式调度与沿线信号控制协同优化".《铁道科学与工程学报》.2022,第19卷(第9期),全文. * |
| 高杰."数据库信息系统中数据字典的一种通用实现方法".《信息科学》.2010,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115617770A (en) | 2023-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7689574B2 (en) | Index and method for extending and querying index | |
| US9183268B2 (en) | Partition level backup and restore of a massively parallel processing database | |
| CN103262043B (en) | The method and system of the meticulous recovery of performing database from differential backup | |
| CN109871373B (en) | Data storage method and device and computer readable storage medium | |
| US10417265B2 (en) | High performance parallel indexing for forensics and electronic discovery | |
| US9418094B2 (en) | Method and apparatus for performing multi-stage table updates | |
| US20120131022A1 (en) | Methods and systems for merging data sets | |
| CN102750317B (en) | Method and device for data persistence processing and data base system | |
| MX2009000589A (en) | Data processing over very large databases. | |
| CN107391544B (en) | Processing method, device and equipment of column type storage data and computer storage medium | |
| CN107491487A (en) | A kind of full-text database framework and bitmap index establishment, data query method, server and medium | |
| US10642530B2 (en) | Global occupancy aggregator for global garbage collection scheduling | |
| CN111241122A (en) | Task monitoring method and device, electronic equipment and readable storage medium | |
| US10719554B1 (en) | Selective maintenance of a spatial index | |
| WO2015183316A1 (en) | Partially sorted log archive | |
| CN107506466B (en) | Small file storage method and system | |
| US10599614B1 (en) | Intersection-based dynamic blocking | |
| CN109491834A (en) | A kind of data reconstruction method and device applied to database | |
| US7984072B2 (en) | Three-dimensional data structure for storing data of multiple domains and the management thereof | |
| CN108038253B (en) | Log query processing method and device | |
| CN115617770B (en) | Data disk storage management method for vehicle state signal data storage | |
| CN102955808A (en) | Data acquisition method and distributed file system | |
| CN109299106B (en) | Data query method and device | |
| CN108256019A (en) | Database key generation method, device, equipment and its storage medium | |
| CN108376104B (en) | Node scheduling method and device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |