CN115617770A

CN115617770A - Data disk storage management method for vehicle state signal data storage

Info

Publication number: CN115617770A
Application number: CN202211440899.XA
Authority: CN
Inventors: 高云鹏; 于纪平; 李沛阳; 李汶轩; 王经纬
Original assignee: Dafenqi Power Technology Beijing Co ltd
Current assignee: Dafenqi Power Technology Beijing Co ltd
Priority date: 2022-11-17
Filing date: 2022-11-17
Publication date: 2023-01-17
Anticipated expiration: 2042-11-17
Also published as: CN115617770B; WO2024104219A1

Abstract

The invention provides a data disk storage management method for storing vehicle state signal data, which comprises the following steps: defining item as BSON format, and associating a plurality of items by a bidirectional linked list structure; the vehicle reported signal data is stored in different vehicle files through a specified classification column; recording the memory positions of the previous element and the next element; the log system is used for recording logs on the operation command and the time node in the inserted data; creating a buffer area in the memory, and controlling the frequency of writing data into a disk according to the time interval and the size of the buffer area; querying data according to the name space and the classification condition; and realizing synchronous backup of data in distributed deployment by transmitting the operation log. According to the invention, different vehicle data are stored in different vehicle files, and the vehicle files are stored in different disk space positions, so that data information of a specified vehicle in a specified time slice can be conveniently inquired under large data volume; the database is matched with the car machine, and the data structure can be used in a targeted extension mode at the later stage.

Description

Data disk storage management method for vehicle state signal data storage

Technical Field

The invention relates to the technical field of data storage, in particular to a data disk storage management method for vehicle state signal data storage.

Background

With the coming of the internet era and the rapid development of network technologies, various network applications become more and more popular, data collection and analysis related to internet devices become especially important for users, and storage management of databases becomes an important influencing factor in data collection and analysis work.

Currently, the mainstream databases are divided into two major types according to the data storage mode: relational and non-relational NoSQL, which can satisfy most application service scenarios. In order to improve the operation efficiency, the existing NoSQL (non-relational database) method for increasing the I/O frequency is used for putting data into a memory for operation, and a part of operation is to use a single thread to perform data operation so as to avoid the thread safety problem and the performance loss caused by thread context switching; in the aspect of a data persistence mechanism, a serialized binary JSON mode or a custom storage structure is used, and log recording is used for preventing the loss of the recorded data after the database is crashed and restarted.

However, with the continuous development of current scientific technology including the internet industry, the existing databases cannot meet the requirements of data storage performance for data collection related operations of some new internet devices.

For a new energy vehicle application service scene, in order to ensure that a new energy vehicle runs safely and stably, real-time monitoring and analysis are required to be performed on vehicle body firmware signals, the reporting frequency of vehicle body firmware signal data is very frequent after the vehicles are produced in batches, and the magnitude scale of the data is also very large.

However, the existing database technology cannot perform customized analysis on data of a specified size, and query efficiency is low when the data size is too large (for example, data size of hundreds of millions).

Disclosure of Invention

In view of this, the present invention aims to provide a data disk storage management method, which performs fragmentation storage on data of different users or internet devices in the same service, improves data storage performance, solves storage security of a large amount of signal data, and improves accurate query efficiency of a large time slice in data of more than ten million levels; and the data of a single user and a single Internet device can be directionally analyzed according to the service requirement, and the data of the whole quantity of data can be analyzed, so that the analysis result can be obtained more quickly, and intelligent analysis support is provided for the fault signal.

The invention provides a data disk storage management method for storing vehicle state signal data, which comprises the following steps:

s1, constructing a data set collection of signal data reported by vehicles, defining the data format of an item data item of each vehicle file in the data set as a BSON (Binary-json) format, storing vehicle state signal data of a time point for each item data item, and storing all signal data reported by the vehicles through a plurality of item data items, wherein the item data items are used for storing the signal data reported by the vehicles

A plurality of the item data items in each vehicle file are associated in a doubly linked list structure (last item pointer, next item pointer);

all signal data reported by the vehicle comprise: vehicle state, battery voltage, battery health, speed, and discharge state;

s2, storing the vehicle reported signal data stored in each data set collection into different vehicle files through a specified classification column, wherein the vehicle reported signal data are stored in the data sets collection

Integrating a plurality of vehicle files consisting of item data items in the same name space to form a record table;

the namespace is a folder path of a data file for storing vehicle state signal data and is stored in a disk; the data files can be stored by a multi-level folder, and the data files are dispersed according to time and vehicles;

s3, recording the memory positions of the previous item data element and the next item data element in the item data item, wherein

If the item data element is a head node or a tail node, the last item pointer of the head node is null, and the next item pointer of the tail node is null; the item pointer is null i.e. no data element is deposited,

the head node of the item data item stores whether the current item data item is complete data,

if the data is incomplete data, the last page of the incomplete data is connected with the information of the vehicle state signal data,

the data content of the item data item stores the real data of the vehicle state signal data,

the tail node of the item data item stores whether the item data item is complete data;

s4, in the process of inserting data into the data set, a log system is used for carrying out log recording on each operation command and each time node; and recording the last data content before the current operation, wherein

When a problem occurs, recovering the data which is operated last time or rolled back to a correct version through the log record file;

in the process of inserting data, the data can not be correctly written into a disk because a CPU does not normally operate, and in order to ensure the safety of the inserted data, a log system is used for carrying out log recording on each operation command and time node; for example, when data XXXX is inserted, a log is recorded first, and specific operation content of the operation is recorded, and then the operation content is written into a disk, and when a problem occurs in writing, recovery can be performed according to the log; recording the last data content before the operation, for example, if an original value of a certain data store is 1, and the result after the operation is 2, recording the original value 1 in the log, and also recording the data after the operation 2;

s5, creating a buffer area in the memory, controlling the frequency of data writing into a disk according to the time interval and the size of the buffer area, firstly recording the operation on the data into a log, and then writing the data into the disk when the data in the cache reaches a set proportion size or a set time interval so as to improve the response speed, wherein the buffer area is formed in the memory, the operation on the data is controlled according to the time interval and the size of the buffer area, and the data is written into the disk when the data in the cache reaches the set proportion size or the set time interval so as to improve the response speed, and the data in the cache is stored in the memory

The buffer provides a separate log to ensure data recovery after a service crash;

in the process of writing a large amount of data, if the data is directly written into a disk, the speed is slow, in order to ensure the efficiency of disk writing, a buffer area is created in a memory, the operation on the data is firstly recorded into a log, and then the data is written into the disk when the data in the cache reaches a set proportion, such as 80% or every 5 s;

s6, vehicle data query operation is carried out according to the name space and the classification conditions, item data items of the vehicle files are loaded into a memory, and matching, sorting and aggregation operation are carried out on the memory, wherein

If a plurality of types of contents are required to be comprehensively inquired, performing fragment type result screening in a streaming loading mode, and performing final integration, sequencing and paging operations on a result set;

and S7, realizing synchronous backup of the data deployed in a distributed manner by transmitting the operation log.

Further, the method for logging each operation command and time node by using the logging system in the step S4 includes:

the database is selected to be cooperatively recorded with the car machine, the car machine stores data information in a short time, when data loss occurs, the database synchronously loses data through the car machine, so that corresponding recovery measures can be provided under the condition that logs of a database server are lost, the logs are kept consistently through the car machine, high data safety is achieved, and the condition that vehicle state signal data are recorded without loss is guaranteed.

Further, the method for writing data into a disk in the step S5 includes:

storing the data into different vehicle files through the content of the primary key of the data;

for example, three types of data a, B, and C are stored in the main key, and the data of the three main keys are stored in three files a ' \ B ' \ C '.

Further, the method for performing fragmented result screening by using a streaming loading manner in the S6 step includes:

finding a corresponding disk storage address according to a namespace, then finding a file in the corresponding storage address according to a queried primary key, namely a set of stored item data items, loading the item data items in the file into a memory in batches by using a fragment loading mode, performing corresponding condition filtering (such as screening and the like) on the file in the memory, performing full operations of integration, sequencing and paging on the data retained in the memory, and returning a result in the memory to a caller.

Further, the method of the vehicle data query operation of the S6 step includes the steps of:

s61, finding a corresponding vehicle file storage directory according to the search condition of the vehicle: a part of paths of the namespace, searching folders grouped according to time under the vehicle file storage directory: the vehicle folder and the time folder form a complete naming space until the position of the corresponding data file is found;

s62, directly jumping to the corresponding line number of the starting time according to the index information stored in the data file, and loading all effective files into a memory, wherein

In the process of loading the effective files into the memory, if more searching conditions exist (for example, the electric quantity is more than 10 percent), directly filtering data which do not meet the conditions, and if the sorting conditions exist, sorting;

s63, integrating all results, and splicing according to a sorting rule;

and S64, returning the data to the client.

Further, the method for implementing synchronous backup of data deployed in a distributed manner in step S7 includes:

and selecting master and slave nodes for the multi-node distributed deployment through a Raft algorithm.

For example, 2 nodes a and B, where a is a master node and is responsible for writing data, B is a slave node and is responsible for backing up data, when a has data change, an operation log is transmitted to B through network I/O, and B synchronizes data by reading the operation log.

Further, the step S5 of obtaining the data in the cache at the set ratio or the set time interval is: the data in the buffer reaches 80% or every 5s.

Further, the classification condition of the S6 step includes: the content of the primary key, wherein the primary key is the type identification of the data. Each data has a primary key, the primary key is a type identifier of the data, and the content of the primary key can be used as a classification condition.

The present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the data disk storage management method for vehicle status signal data storage as described above.

The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the data disk storage management method for vehicle status signal data storage as described above when executing the program.

Compared with the prior art, the invention has the beneficial effects that:

according to the invention, the related data of different vehicles are stored in different vehicle files, the vehicle files are stored in different disk space positions, and the data information of the designated vehicle in the designated time segment can be conveniently inquired under large data volume by defining the definition of the storage format and adopting the log record, so that the inquiry efficiency is improved; the database can be matched with a vehicle machine, and the data structure can be expanded and used in targeted applications such as failure analysis statistics and the like of vehicle machine parts at the later stage.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.

In the drawings:

FIG. 1 is a schematic diagram of a structure of a data set according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a computer device according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating the structure of an item data item according to an embodiment of the present invention;

FIG. 4 is a flow chart of a data disc storage management method for vehicle status signal data storage in accordance with the present invention;

FIG. 5 is a flowchart of a method of vehicle data query operation according to an embodiment of the present invention.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and products consistent with certain aspects of the present disclosure, as detailed in the appended claims.

The terminology used in the disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if," as used herein, may be interpreted as "at \8230; \8230when" or "when 8230; \823030when" or "in response to a determination," depending on the context.

The embodiments of the present invention will be described in further detail with reference to the accompanying drawings.

An embodiment of the present invention provides a data disk storage management method for storing vehicle status signal data, which is shown in fig. 4 and includes the following steps:

A plurality of the item data items in each vehicle file are associated in a doubly linked list structure (previous item pointer, next item pointer);

the specific structure of the item data item of this embodiment is shown in fig. 3;

all signal data reported by the vehicle include: vehicle state, battery voltage, battery health, speed, and discharge state;

A plurality of vehicle files composed of item data items are finally integrated in the same namespace (collection namespace) (see fig. 1) to form a record table;

the namespace is a folder path of a data file for storing vehicle state signal data and is stored in a magnetic disk; the data files can be stored by a multi-level folder, and the data files are dispersed according to time and vehicles;

When a problem occurs, recovering the last operation or rolling back to the correct version of data through a log record file;

the method for logging each operation command and time node by using the logging system comprises the following steps:

the method comprises the steps that a database and a vehicle machine are selected for collaborative recording, wherein the vehicle machine stores data information in a short time, and when data loss occurs, the database synchronously loses data through the vehicle machine, so that corresponding recovery measures can be provided under the condition that logs of a database server are lost, and the vehicle machine bidirectional logs are kept consistently to achieve high data safety so as to ensure the recording of vehicle state signal data;

s5, creating a buffer area in the memory, controlling the frequency of data writing into a disk according to the time interval and the size of the buffer area, firstly recording the operation on the data into a log, and then writing the data into the disk when the data in the buffer memory reaches the set proportion or the set time interval so as to improve the response speed, wherein the buffer area is created in the memory, the operation on the data is written into the log, and the data is written into the disk when the data in the buffer memory reaches the set proportion or the set time interval, and the response speed is improved, wherein the buffer area is created in the memory, and the data is written into the disk according to the time interval and the size of the buffer area

The buffer area provides an independent log to ensure the data recovery after the service crash;

the method for writing data into the magnetic disk comprises the following steps:

in the embodiment, three types of data A, B and C are stored in the main key, and the data of the three main keys can be stored into A ' \ B ' \ C ' in three files;

s6, vehicle data query operation is carried out according to the name space and the classification conditions, item data items of the vehicle files are loaded into a memory, and matching, sorting and aggregation operation are carried out in the memory, wherein

the classification conditions include: the content of the primary key, wherein the primary key is the type identification of the data; each data has a main key, the main key is a type identifier of the data, and the content of the main key can be used as a classification condition;

the method for performing fragment-type result screening by using a streaming loading mode comprises the following steps:

finding a corresponding disk storage address according to a namespace, then finding a file in the corresponding storage address according to an inquired main key, referring to a vertical column (a set for storing item data items) in fig. 1, loading the item data items in the file into a memory in batches by using a fragment-type loading mode, performing corresponding condition filtering (such as screening) on the file in the memory, performing full operations of integration, sequencing and paging on data retained in the memory, and returning a result in the memory to a calling party;

in this embodiment, query needs to be performed 22 years, 9 months, 1 day, 10: 9 months, 3 days, 10 from 00 to 22 years: 00, as shown in fig. 5, the method of the vehicle data query operation includes the steps of:

s63, integrating all results, and splicing according to a sorting rule;

s64, returning data to the client;

The method for realizing synchronous backup of data in distributed deployment in the step S7 includes:

selecting master and slave nodes for the multi-node distributed deployment through a Raft algorithm;

in this embodiment, there are 2 nodes a and B, where a is a master node and is responsible for writing data, and B is a slave node and is responsible for backing up data, and when data change occurs in a, an operation log is transmitted to B through network I/O, and B synchronizes data by reading the operation log.

According to the embodiment of the invention, the related data of different vehicles are stored in different vehicle files, the vehicle files are stored in different disk space positions, and the data information of the designated vehicle in a designated time slice is conveniently inquired under large data volume by defining the definition of a storage format and adopting log records, so that the inquiry efficiency is improved; the database can be matched with a vehicle machine, and the data structure can be expanded and used in targeted applications such as failure analysis statistics of vehicle machine parts at the later stage.

Fig. 2 is a schematic structural diagram of a computer device provided in an embodiment of the present invention; referring to fig. 2 of the drawings, the computer apparatus comprises: an input device 23, an output device 24, a memory 22 and a processor 21; the memory 22 for storing one or more programs; when the one or more programs are executed by the one or more processors 21, cause the one or more processors 21 to implement the data disk storage management method for vehicle status signal data storage as provided in the above embodiments; the input device 23, the output device 24, the memory 22 and the processor 21 may be connected by a bus or other means, and fig. 2 illustrates the connection by the bus as an example.

The memory 22, which is a computer-readable and writable storage medium, may be used for storing a software program, a computer-executable program, and program instructions corresponding to the data disk storage management method for vehicle status signal data storage according to the embodiment of the present invention; the memory 22 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like; further, the memory 22 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device; in some examples, the memory 22 may further include memory located remotely from the processor 21, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 23 may be used to receive input numeric or character information and to generate key signal inputs relating to user settings and function controls of the apparatus; the output device 24 may include a display device such as a display screen.

The processor 21 executes various functional applications of the device and data processing by executing software programs, instructions and modules stored in the memory 22, namely, implements the above-described data disk storage management method for vehicle state signal data storage.

The computer device provided above can be used to execute the data disk storage management method for vehicle status signal data storage provided in the above embodiments, and has corresponding functions and advantages.

Embodiments of the present invention also provide a storage medium containing computer-executable instructions for performing a data disk storage management method for vehicle status signal data storage as provided in the above embodiments when executed by a computer processor, the storage medium being any of various types of memory devices or storage devices, the storage medium comprising: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or combinations thereof; in addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet); the second computer system may provide program instructions to the first computer for execution. A storage medium includes two or more storage media that may reside in different locations, such as in different computer systems connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.

Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the data disk storage management method for vehicle status signal data storage described in the above embodiment, and may also perform related operations in the data disk storage management method for vehicle status signal data storage provided by any embodiment of the present invention.

So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can be within the protection scope of the invention.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention; various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data disk storage management method for vehicle status signal data storage, comprising the steps of:

s1, constructing a data set collection of signal data reported by a vehicle, defining the data format of an item data item of each vehicle file in the data set as a BSON format, storing the vehicle state signal data of a time point in each item data item, and storing all the signal data reported by the vehicle through a plurality of item data items, wherein the item data items are used for storing the signal data reported by the vehicle

A plurality of item data items in each vehicle file are associated in a doubly linked list structure, and all signal data reported by the vehicle comprises: vehicle state, battery voltage, battery health, speed, and discharge state;

s2, storing the vehicle reported signal data stored in each data set collection into different vehicle files through a specified classification column, wherein the vehicle reported signal data are stored in each data set collection

Integrating a plurality of vehicle files formed by item data items in the same name space to form a record table, wherein the name space is a folder path of the data files for storing vehicle state signal data and is stored in a magnetic disk;

If the item data element is a head node or a tail node, the last item pointer of the head node is null, the next item pointer of the tail node is null,

the head node of the item data item stores whether the current item data item is complete data, if the current item data item is incomplete data, the last page of the incomplete data is connected with the information of the vehicle state signal data,

s4, in the process of inserting data into the data set, a log system is used for logging each operation command and each time node, and the last data content before the operation is recorded, wherein the log system is used for recording the log of each operation command and each time node, and the last data content before the operation is recorded

When a problem occurs, recovering the last operation or rolling back to the data of the correct version through the log record file;

and S7, realizing synchronous backup of the data in distributed deployment by transmitting the operation log.

2. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method of logging each operation command and time node using a logging system of the step S4 comprises:

and selecting a database to perform cooperative recording with the vehicle machine, wherein the vehicle machine stores data information in a short time, and when data loss occurs, the database synchronously loses the data through the vehicle machine.

3. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method of writing data to a disk of the step S5 comprises:

and storing the data into different vehicle files through the content of the primary key of the data.

4. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method for performing segmented result filtering using streaming loading in step S6 comprises:

finding out a corresponding disk storage address according to the namespace, then finding out a file in the corresponding storage address according to the inquired primary key, loading item data items in the file into the memory in batches by using a fragment loading mode, carrying out corresponding condition filtering on the file in the memory, carrying out full operation of integration, sequencing and paging on the data retained in the memory, and returning the result in the memory to a caller.

5. The data disk storage management method for vehicle status signal data storage according to claim 4, wherein the vehicle data query operation of step S6 comprises the steps of:

s62, directly jumping to the corresponding line number of the starting time according to the index information stored in the data file, and loading all effective files into a memory, wherein in the process of loading the effective files into the memory, if more search conditions exist, data which do not meet the conditions are directly filtered out, and if a sorting condition exists, sorting is carried out;

s63, integrating all results, and splicing according to a sorting rule;

and S64, returning the data to the client.

6. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the method for implementing synchronous backup of data deployed in a distributed manner in the step S7 comprises:

and selecting master and slave nodes for the distributed deployed multi-node through a Raft algorithm.

7. The data disk storage management method for storing vehicle status signal data as claimed in claim 1, wherein the data in the buffer memory of step S5 reaches a set proportional size or reaches a set time interval as follows: the data in the buffer reaches 80% or every 5s.

8. The data disk storage management method for vehicle status signal data storage according to claim 1, wherein the classification condition of the S6 step comprises: the content of the primary key, wherein the primary key is the type identification of the data.

9. A computer-readable storage medium, having stored thereon a computer program, the program being adapted to carry out the steps of the data disk storage management method for vehicle status signal data storage of any one of claims 1 to 8 when executed by a processor.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of a data disk storage management method for vehicle status signal data storage according to any one of claims 1 to 8.