CN114356226B - Sensor data storage method, device, equipment and storage medium - Google Patents
Sensor data storage method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114356226B CN114356226B CN202111552766.7A CN202111552766A CN114356226B CN 114356226 B CN114356226 B CN 114356226B CN 202111552766 A CN202111552766 A CN 202111552766A CN 114356226 B CN114356226 B CN 114356226B
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- sensor data
- column
- reference table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000013500 data storage Methods 0.000 title claims abstract description 32
- 238000004590 computer program Methods 0.000 claims description 9
- 238000013506 data mapping Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000012545 processing Methods 0.000 abstract description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C5/0841—Registering performance data
- G07C5/0875—Registering performance data using magnetic data carriers
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Recording Measured Values (AREA)
Abstract
The present application relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for storing sensor data. The method comprises the following steps: using column storage for the acquired sensor data, and establishing a reference table facing to the row, wherein the reference table refers to the position of the data in the column storage file; and setting a storage pointer corresponding to the data of the sensor in the column-type storage file, wherein the reference table stores the storage pointer. The application solves the technical problems of large storage space and low storage efficiency of the existing sensor data storage.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for storing sensor data.
Background
In automatic driving or the Internet of things, different sensors are used for collecting data according to different data to be detected, different frame rates are used for the different sensors, and arrival time after data collection is different.
In automatic driving or internet of things, algorithm training and simulation are often performed by finding data of different sensors at the same time.
In order to acquire data of different sensors at the same time, there are two conventional methods:
1. Using redundant storage, if a certain sensor frame rate is lower for each frame, storing the data of the most adjacent frame without corresponding data;
2. For each sensor, looking forward the number of the previous frame when in use;
mode 1 in the conventional case requires a large memory space, and scheme 2 requires loading of the scanning sensor, which is inefficient.
Disclosure of Invention
Therefore, the embodiment of the application provides a sensor data storage method, a device, equipment and a storage medium, which can solve the technical problems of large storage space and low storage efficiency of the existing sensor data storage, and the specific technical scheme comprises the following steps:
in a first aspect, embodiments of the present application provide a method, an apparatus, a device, and a storage medium for storing sensor data, where the method includes:
Using column storage for the acquired sensor data, and establishing a reference table facing to the row, wherein the reference table refers to the position of the data in the column storage file;
and setting a storage pointer corresponding to the data of the sensor in the column-type storage file, wherein the reference table stores the storage pointer.
By adopting the technical scheme, the reference table is arranged, and the situation of more data redundancy is solved through smaller data storage.
Preferably, the method further comprises: and setting a data acquisition strategy according to the acquired data type.
By adopting the technical scheme, the method and the device can adapt to various data acquisition conditions through different data acquisition requests, and acquire data in a mode of fast data acquisition rate and/or less data redundancy.
Preferably, setting the data acquisition policy according to the acquired data type includes:
judging whether the acquisition time in the data acquisition request is a continuous time period or not;
If the continuous time period is the continuous time period, the reference table locates the position corresponding to the first storage pointer of the sensor data to be acquired in the continuous time period, the sensor data in the column storage file is read from the position of the first storage pointer by means of the column storage of the same column storage address continuously.
If the time period is discontinuous, that is, the acquisition time is a certain preset time, reading a row of data corresponding to the preset time by the reference table, and reading the sensor data in the column storage file according to the storage pointer of the row of data.
By adopting the technical scheme, reasonable data acquisition strategies are convenient to select quickly, so that acquired data redundancy is reduced, and data acquisition efficiency is improved.
Preferably, the acquired sensor data is stored using the list:
The sensor data is written to the buffer area and written to the columnar storage file by the buffer area.
By adopting the technical scheme, the problems of slower efficiency and poor data integrity of the columnar storage file during data writing are solved.
Preferably, the buffer area includes a buffer queue and/or a line-oriented buffer file.
Preferably, when the reference table is updated at each preset time, the storage pointer of the latest data in the corresponding columnar storage file is updated to the position of the current preset time of the reference table.
By adopting the technical scheme, the integrity of the sensor data at each moment in the reference table is ensured under the condition of reducing data redundancy.
Preferably, if the length of the sensor data in the column-type storage file is smaller than 4GB, the storage pointer corresponding to the sensor data adopts an integral storage starting position and length of the sensor data; otherwise, the storage pointer corresponding to the sensor data adopts a long-form storage of the initial position and the length of the sensor data.
By adopting the technical scheme, the type of the storage pointer is set according to the length of the data, and the memory size of the reference table is reduced under the condition of meeting the storage requirement.
Preferably, the sensor data is a set of multiple groups of single sensor data, the single sensor data is stored in the same column in the column storage file, the data acquired by a single sensor at a certain acquisition time is stored in a corresponding cell of a column in the column storage file, and the cells of the single sensor data stored in multiple columns in the column storage file are aligned with respect to the acquisition time; the cells in the reference table store storage pointers for mapping data in the corresponding cells of the single sensor data in the column-type storage file; multiple cells in the same row in the reference table respectively store storage pointers of data mapping in multiple groups of single sensor data with the same acquisition time corresponding to the cells of the acquisition time.
Preferably, the types of the sensor data include: positioning data synchronization device data, camera data, laser radar data and millimeter wave radar data.
In a second aspect, a sensor data storage device, the device comprising:
the storage module is used for storing the acquired sensor data in a column mode, and establishing a reference table facing to the rows, wherein the reference table refers to the position of the data in the column storage file;
And the pointer establishment module is used for setting a storage pointer corresponding to the data of the sensor in the columnar storage file, and the reference table stores the storage pointer.
In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor and a computer program stored in the memory and running on the processor, the processor implementing the steps of the sensor data storage method of any one of the preceding claims when the computer program is executed.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium, comprising a computer program stored thereon, which when executed by a processor, implements the steps of the sensor data storage method of any of the preceding claims.
In summary, compared with the prior art, the technical scheme provided by the embodiment of the application has the following beneficial effects:
1. Setting a reference table, wherein the reference table is internally provided with data of a column storage file by a memory pointer, and the situation of more data redundancy is solved by smaller data storage;
2. By setting the buffer area, the problems of low writing speed and poor data integrity of the column-type storage file are solved;
3. and updating the reference table at preset time, and updating the storage pointer corresponding to the latest data in the current columnar storage file to the reference table every time when the reference table is updated, so that the integrity of the sensor data at each time in the reference table is ensured under the condition of reducing data redundancy.
Drawings
Fig. 1 is a flow chart of a method for storing sensor data according to an embodiment of the application.
Fig. 2 is a schematic flow chart of a sensor data storage method according to another embodiment of the application.
FIG. 3 is a second flowchart of a sensor data storage method according to another embodiment of the present application.
Fig. 4 is a schematic structural diagram of a sensor data storage device according to an embodiment of the present application.
Detailed Description
The present embodiment is only for explanation of the present application and is not to be construed as limiting the present application, and modifications to the present embodiment, which may not creatively contribute to the present application as required by those skilled in the art after reading the present specification, are all protected by patent laws within the scope of claims of the present application.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In addition, the term "and/or" in the present application is merely an association relationship describing the association object, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. In the present application, unless otherwise specified, the term "/" generally indicates that the associated object is an "or" relationship.
Embodiments of the application are described in further detail below with reference to the drawings.
In the actual vehicle sensor data storage, when the vehicle collects the position data of the vehicle and the speed data of the vehicle at different time points by different sensors, if the line storage is used for the sensor data, the position data and the speed data at each time point are placed in one line, and when the speed data of the vehicle for a period of time need to be read, the following two reading modes exist:
1. reading the data of the whole continuous line can read the position data more, and the data redundancy is caused.
2. For each row, only the speed data is read, so that after the speed data is read, the magnetic head needs to skip the position, then the speed data of the next row is read, and the number of times of magnetic head skip is equal to the number of rows. In a common hard disk, the read efficiency of the hard disk is greatly affected by moving a magnetic head for a plurality of times, and the data read efficiency is low.
If columnar storage is used for the sensor data, the velocity data for a plurality of time nodes is stored in a continuous medium, thus avoiding situations such as reading garbage and multiple head relocations in columnar storage in the context of reading only the velocity data. However, when it is necessary to read the position data and the speed data simultaneously, the position data and the speed data can be read at one time by using the line memory, and the efficiency is higher than that of the column memory.
Referring to fig. 1, in one embodiment of the present application, a method for storing sensor data is provided, the main steps of the method are described as follows:
s1: using column storage for the acquired sensor data, and establishing a reference table facing to the row, wherein the reference table refers to the position of the data in the column storage file;
s2: and setting a storage pointer corresponding to the data of the sensor in the column-type storage file, wherein the reference table stores the storage pointer.
Specifically, a line-oriented reference table, i.e., a reference table, is stored using a line.
Updating data detected by the sensor at preset intervals, and storing the data detected by the sensor. In this embodiment, the sensor data collected by the sensor is stored in a column storage manner to generate a column storage file, that is, the data in the same column is the data of the same sensor, and the storage addresses of the data in the same column are continuous. The data types of the same sensor are the same, and ambiguity problem does not exist, so that the data types of each column are the same by adopting column type storage, thereby being beneficial to data analysis; for the data stored in the same column, the repeatability of the data is high, and the column type storage has great advantages in data compression; and the data in the same column are stored in the same position, so that the data are convenient to sort.
On the basis of the column-type storage files, the application establishes the reference table, each column-type storage file corresponds to one reference table, the reference table adopts line storage, the contents in the reference table are used for pointing to the positions of the data in the column-type storage file (namely the storage addresses of the data), and the contents in the reference table correspond to the data in the column-type storage file one by one.
In automatic driving, a large number of sensors and calculated perceived and planned result data need to be collected, and the frequency of the data generally has different frequencies according to the performance of hardware itself, the computational complexity and the actual requirements. For example, when the positioning and multiple cameras are stored, if the positioning and multiple cameras are aligned to 100HZ, a large amount of null values are generated by the images, and for the images, a few megabytes of single data are usually generated, redundant data are directly stored in the column storage, and the space is wasted; the method comprises the steps of adopting a column storage mode to store sensor data, continuously storing the same sensor data with the same data type, and adopting column storage to enable data among all columns of data to be not aligned in the process of storing the file in a column mode, namely enabling data acquired by different sensors at the same time to be not aligned.
The application also has the following advantages:
(1) All sensor data in a certain frame, namely at a certain moment, are required to be acquired, and only the content of the row corresponding to the moment reference table is required to be read, so that the data acquisition is convenient, and the situation of data redundancy caused by the need of reading a plurality of columns of data during column storage is avoided;
(2) When the data of a certain sensor at a certain moment is required to be acquired, when the data of a row corresponding to the reference table is read, the data redundancy of storage pointers corresponding to other sensor data of the row is generated, and the data quantity of the storage pointers is generally far smaller than the data of the sensor pointed by the storage pointers, so that the generated data redundancy is less;
(3) The storage pointer in the reference table is used for positioning the position of the data in the column-type storage file, wherein the same column of data is continuously stored in the column-type storage file, so that the position of the storage position of the required column in the column-type storage file is conveniently positioned, and the data can be conveniently and quickly acquired.
The reference table is arranged, and the problem of more data redundancy is solved through smaller data storage.
Further, the sensor data is a set of multiple groups of single sensor data, the single sensor data is stored in the same column in the column storage file, the data acquired by a single sensor at a certain acquisition time is stored in a corresponding cell of a column in the column storage file, which corresponds to the single sensor data, and the cells of the single sensor data storage data in multiple columns in the column storage file are aligned with respect to the acquisition time; the storage pointers of the data in the corresponding cells of the single sensor data in the mapping column type storage file are stored in the cells in the reference table; multiple cells in the same row in the reference table respectively store storage pointers of data mapping in multiple groups of single sensor data with the same acquisition time corresponding to the cells of the acquisition time.
Specifically, the column-type storage file is in a form of a table, the data of one sensor is stored in one column of the column-type storage file, and the storage positions of the data of the sensor are continuous. In a column of data corresponding to a certain sensor in the column-type storage file, each cell is data content acquired at a corresponding moment, and the moment corresponding to each row of data in the column-type storage file is the same, namely, in the column-type storage file, the data corresponding to a column is data of all sensors acquired at the same moment.
The storage pointers corresponding to the sensor data with the same acquisition time are stored in the same row of the reference table, namely, one column of data of the reference table is the storage pointer corresponding to the data of all the moments currently acquired by a certain sensor.
Further, the data types of the sensor data include: the positioning data synchronization device data, the camera data, the laser radar data and the millimeter wave radar data can be other sensors with different frequencies in other embodiments, which are not described herein.
Further, if the length of the sensor data in the column-type storage file is smaller than 4GB, the storage pointer corresponding to the sensor data adopts an integral type to store the initial position and the length of the sensor data; otherwise, the storage pointer corresponding to the sensor data adopts a long-form storage of the initial position and the length of the sensor data.
The integer is 4 bytes and the long integer is 8 bytes, and the size of the file of the reference table is reduced by setting the data type of the storage pointer according to the size of the data in the columnar storage file.
Referring to fig. 2, further, in another embodiment, the method further includes step S3:
S3: and setting a data acquisition strategy according to the acquired data type.
A data acquisition strategy is preset, and is set according to the type of data to be acquired, and in this embodiment, the data acquisition strategy is specifically a data acquisition path.
And determining a data acquisition path according to the type of the data acquisition request, and improving the data acquisition rate.
Specifically, in this embodiment, the data acquisition requests are different, and the acquisition requests may be the same or different types of data acquisition that need to be acquired. For example, the types of the data to be acquired are different, namely, the data to be acquired are known to be in different columns in the column-type storage file at present, and the data are acquired according to the reference table, so that the data redundancy can be reduced, and the data reading rate can be improved; when the data types of the data to be acquired are the same, the same column of the data to be acquired in the column-type storage file can be known, and when the data types to be acquired are acquired, the data can be directly read from the column-type storage file without reading the reference table. In other embodiments of the present application, the data acquisition request may be different from the data acquisition request, for example, the data acquisition request may be for acquiring all data of different sensors, the data acquisition request may be for acquiring all data of all sensors at a certain time, etc., which will not be described herein.
According to the embodiment, through different data acquisition requests, data are acquired in a mode of adapting to various data acquisition conditions, wherein the data acquisition rate is high and/or the data redundancy is low.
Referring to fig. 3, further, in the present embodiment, step S3 includes:
s31: judging whether the acquisition time in the data acquisition request is a continuous time period or not;
S32: if the reference table is a continuous time period, locating the position corresponding to the first storage pointer of the sensor data to be acquired in the continuous time period, continuously storing the same column storage address by means of the column storage pointer, and reading the sensor data in the column storage file from the position of the first storage pointer, for example, all the sensor data of a certain sensor need to be acquired, the sensor data in a certain continuous time period of a certain sensor need to be acquired, all the sensor data of a plurality of sensors need to be acquired, the sensor data of a plurality of sensors need to be acquired in a certain time period, and the like;
s33: if the time period is discontinuous, that is, the acquisition time is a certain preset time, reading a row of data corresponding to the preset time by the reference table, and reading the sensor data in the column storage file according to the storage pointer of the row of data, for example, all the sensor data of the certain preset time need to be acquired, part of the sensor data of the certain preset time need to be acquired, certain sensor data of the certain preset time need to be acquired, and the like.
Further, when the acquisition time is a continuous time period, the reference table is used for positioning the position of the first storage pointer in the time period, the position of the first storage pointer is used for acquiring the data of the column in the column storage file of the time period, if the data of a certain preset moment in the column data is empty, a mark is set at the preset moment, and when the column data is extracted and returned, the data of the previous preset moment of the preset moment is filled to the preset moment according to the mark, so that the integrity of the data is realized.
The sensor data is stored in a column mode, redundancy of the data storage is reduced, but the problems of low data writing efficiency and poor data integrity exist. Further, in another embodiment, step S1 includes setting a buffer area of the sensor data after receiving the sensor data, and writing the sensor data from the buffer area to the columnar storage file, including but not limited to the following three ways:
1. S1: and writing the received sensor data into a buffer queue, and writing the received sensor data into a column-type storage file by the buffer queue.
2. S1: and writing the received sensor data into a line-oriented buffer file, and writing the buffer file into a column-type storage file.
3. S1: and writing the received sensor data into a buffer queue and a line-oriented buffer file, and writing the buffer queue and the buffer file into a column-type storage file.
Further, after the sensor data is written into the column-type storage file from the buffer area, the corresponding data in the buffer area is deleted, so that the data redundancy is reduced, and the storage space is released.
Further, in another embodiment, when the reference table is updated at each preset time, the storage pointer of the latest data in the corresponding columnar storage file is updated to the position of the current preset time of the reference table.
For the sensors with different frame rates, if new sensor data are not written in at the current preset time, the storage pointer of the latest data is the data of the last preset time, and the storage pointer of the sensor data of the last preset time is written in the reference table.
In this embodiment, the time of storing the pointer in the preset update reference table is also the preset time of updating the sensor data in the columnar storage file, the preset time is provided with a plurality of times, and the time intervals between adjacent preset times are the same.
In this embodiment, when the storage pointers of all the sensor data in the column storage file acquired at a certain preset time and stored in each row in the reference table are updated, only the writing operation is performed, that is, only the storage pointer corresponding to the current row is updated.
Illustrating: the reference table stores the stored pointers corresponding to the sensor data of the sensor 1 and the sensor 2 at the times t1 and t2, and the stored data pointers of the reference table at the time t2 include the stored pointer ref11 corresponding to the sensor 1 and the stored pointer ref21 corresponding to the sensor 2. At time t3, data date12 of the sensor 1 is added to the columnar storage file, at this time, a storage pointer ref12 of the data12 is correspondingly generated, and the storage pointer ref12 is written into the reference table at the position corresponding to the sensor 1 at time t3, but because the frame rate of the sensor 2 is different from that of the sensor 1, at this time, the updated sensor data of the sensor 2 is not written into the columnar storage file, then, in the columnar storage file, the storage pointer corresponding to the latest data of the sensor 2 is ref21, and the storage pointer ref21 is written into the reference table at the position corresponding to the sensor 2 at time t 3.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
Referring to fig. 4, in one embodiment of the present application, a sensor data storage device is provided, which corresponds to the sensor data storage method of the above embodiment one by one. The sensor data storage device includes;
the storage module is used for storing the acquired sensor data in a column mode, and establishing a reference table facing to the rows, wherein the reference table refers to the position of the data in the column storage file;
And the pointer establishment module is used for setting a storage pointer corresponding to the data of the sensor in the columnar storage file, and the reference table stores the storage pointer.
The various modules of the sensor data storage device described above may be implemented in whole or in part in software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Further, if the length of the sensor data in the column-type storage file is smaller than 4GB, the storage pointer corresponding to the sensor data adopts an integral type to store the initial position and the length of the sensor data; otherwise, the storage pointer corresponding to the sensor data adopts a long-form storage of the initial position and the length of the sensor data.
Further, in another embodiment, the method further includes a policy setting module:
And the strategy setting module is used for setting a data acquisition strategy according to the acquired data type.
Further, the policy setting module is further configured to determine whether the acquisition time in the data acquisition request is a continuous time period; if the reference table is a continuous time period, locating the position corresponding to the first storage pointer of the sensor data to be acquired in the continuous time period, continuously storing the same column storage address by means of the column storage pointer, and reading the sensor data in the column storage file from the position of the first storage pointer, for example, all the sensor data of a certain sensor need to be acquired, the sensor data in a certain continuous time period of a certain sensor need to be acquired, all the sensor data of a plurality of sensors need to be acquired, the sensor data of a plurality of sensors need to be acquired in a certain time period, and the like; if the time period is discontinuous, namely the acquisition time is a certain preset time, reading one row of data corresponding to the preset time by a reference table, and reading the sensor data in the column-type storage file according to a storage pointer of the row of data, for example, all the sensor data needing to be acquired at a certain preset time, part of the sensor data needing to be acquired at a certain preset time, a certain sensor data needing to be acquired at a certain preset time and the like.
Further, when the acquisition time is a continuous time period, the reference table is used for positioning the position of the first storage pointer in the time period, the position of the first storage pointer is used for acquiring the data of the column in the column storage file of the time period, if the data of a certain preset moment in the column data is empty, a mark is set at the preset moment, and when the column data is extracted and returned, the data of the previous preset moment of the preset moment is filled to the preset moment according to the mark.
Further, in another embodiment, the storage module sets a buffer area of the sensor data after receiving the sensor data, and writes the sensor data into the columnar storage file from the buffer area, including but not limited to the following three ways:
1. And writing the received sensor data into a buffer queue, and writing the received sensor data into a column-type storage file by the buffer queue.
2. And writing the received sensor data into a line-oriented buffer file, and writing the buffer file into a column-type storage file.
3. And writing the received sensor data into a buffer queue and a line-oriented buffer file, and writing the buffer queue and the buffer file into a column-type storage file.
Further, after the sensor data is written into the column-type storage file from the buffer area, the corresponding data in the buffer area is deleted, so that the data redundancy is reduced.
Further, in another embodiment, when the reference table is updated at each preset time, the storage pointer of the latest data in the corresponding columnar storage file is updated to the position of the current preset time of the reference table.
Further, the sensor data is a set of multiple groups of single sensor data, the single sensor data is stored in the same column in the column storage file, the data acquired by a single sensor at a certain acquisition time is stored in a corresponding cell of a column in the column storage file, which corresponds to the single sensor data, and the cells of the single sensor data storage data in multiple columns in the column storage file are aligned with respect to the acquisition time; the storage pointers of the data in the corresponding cells of the single sensor data in the mapping column type storage file are stored in the cells in the reference table; multiple cells in the same row in the reference table respectively store storage pointers of data mapping in multiple groups of single sensor data with the same acquisition time corresponding to the cells of the acquisition time.
Further, the types of sensor data include: positioning data synchronization device data, camera data, laser radar data and millimeter wave radar data.
In one embodiment of the present application, a computer device is provided, which may be a server. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device may be implemented by any type of volatile or nonvolatile memory device, including but not limited to: magnetic disk, optical disk, EEPROM (Electrically erasable programmable Read Only Memory), EPROM (Erasable Programmable Read Only Memory ), SRAM (Static Random Access Memory, static random Access Memory), ROM (Read-Only Memory), magnetic Memory, flash Memory, PROM (Programmable Read-Only Memory). The memory of the computer device provides an environment for the running of an operating system and computer programs stored therein. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements the steps of the sensor data storage method described in the above embodiments.
In one embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the sensor data storage method steps of the above embodiments. The computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic disk, floppy disk, and the like.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus of the present application is divided into different functional units or modules to perform all or part of the above-described functions.
Claims (11)
1. A method of storing sensor data, the method comprising:
Using column storage for the acquired sensor data, and establishing a reference table facing to the row, wherein the reference table refers to the position of the data in the column storage file;
setting a storage pointer corresponding to data of the sensor in the column-type storage file, wherein the reference table stores the storage pointer;
The sensor data are a plurality of groups of single sensor data, the single sensor data are stored in the same column in the column storage file, the data acquired by a single sensor at a certain acquisition time are stored in corresponding cells of a column in the column storage file, which correspondingly store the single sensor data, and the cells of the single sensor data storage data in a plurality of columns in the column storage file are aligned with respect to the acquisition time; the cells in the reference table store storage pointers for mapping data in the corresponding cells of the single sensor data in the column-type storage file; multiple cells in the same row in the reference table respectively store storage pointers of data mapping in multiple groups of single sensor data with the same acquisition time corresponding to the cells of the acquisition time.
2. The method of storing sensor data according to claim 1, wherein the method further comprises: and setting a data acquisition strategy according to the acquired data type.
3. The sensor data storage method of claim 2, wherein setting the data acquisition policy according to the acquired data type comprises:
judging whether the acquisition time in the data acquisition request is a continuous time period or not;
If the continuous time period is the continuous time period, the reference table is used for positioning the position corresponding to the first storage pointer of the sensor data to be acquired in the continuous time period, the same column storage address is continuously stored by means of the column, and the sensor data in the column storage file is read from the position of the first storage pointer;
If the time period is discontinuous, that is, the acquisition time is a certain preset time, reading a row of data corresponding to the preset time by the reference table, and reading the sensor data in the column storage file according to the storage pointer of the row of data.
4. The sensor data storage method according to claim 1, wherein the acquired sensor data is stored using a column type as:
The sensor data is written to the buffer area and written to the columnar storage file by the buffer area.
5. The method of claim 4, wherein the buffer area comprises a buffer queue and/or a line-oriented buffer file.
6. The method according to claim 1, wherein when the reference table is updated at each preset time, the storage pointer of the latest data in the corresponding columnar storage file is updated to the position of the reference table at the current preset time.
7. The method for storing sensor data according to claim 1, wherein if the length of the sensor data in the column-type storage file is less than 4GB, the storage pointer corresponding to the sensor data stores the start position and the length of the sensor data in an integer manner; otherwise, the storage pointer corresponding to the sensor data adopts a long-form storage of the initial position and the length of the sensor data.
8. The sensor data storage method of claim 1, wherein the type of sensor data comprises: positioning data synchronization device data, camera data, laser radar data and millimeter wave radar data.
9. A sensor data storage device, the device comprising:
The storage module is used for storing acquired sensor data in a column mode, and establishing a reference table facing to a row, wherein the reference table refers to the position of data in a column-type storage file, the sensor data is a set of multiple groups of single sensor data, the single sensor data are stored in the same column in the column-type storage file, the data acquired by a single sensor at a certain acquisition time are stored in corresponding cells of the column in the column-type storage file, and the cells of the single sensor data storage data in the multiple columns in the column-type storage file are aligned with respect to the acquisition time;
The pointer establishment module is used for setting a storage pointer corresponding to the data of the sensor in the column-type storage file, the reference table stores the storage pointer, and the storage pointer of the data in the unit cell corresponding to the single sensor data in the mapping column-type storage file is stored in the unit cell in the reference table; multiple cells in the same row in the reference table respectively store storage pointers of data mapping in multiple groups of single sensor data with the same acquisition time corresponding to the cells of the acquisition time.
10. A computer device comprising a memory, a processor and a computer program stored in the memory and running on the processor, the processor implementing the steps of the sensor data storage method of any one of claims 1-8 when the computer program is executed.
11. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the steps of the sensor data storage method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111552766.7A CN114356226B (en) | 2021-12-17 | 2021-12-17 | Sensor data storage method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111552766.7A CN114356226B (en) | 2021-12-17 | 2021-12-17 | Sensor data storage method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114356226A CN114356226A (en) | 2022-04-15 |
| CN114356226B true CN114356226B (en) | 2024-07-16 |
Family
ID=81100183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111552766.7A Active CN114356226B (en) | 2021-12-17 | 2021-12-17 | Sensor data storage method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114356226B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824574A (en) * | 2015-12-07 | 2016-08-03 | 国家电网公司 | Memory data storage method |
| CN109542898A (en) * | 2018-10-30 | 2019-03-29 | 天津字节跳动科技有限公司 | Date storage method, device, electronic equipment and the storage medium of data bank table |
| CN113656364A (en) * | 2021-08-05 | 2021-11-16 | 福瑞泰克智能系统有限公司 | Sensor data processing method, device and computer readable storage medium |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101609331B (en) * | 2009-07-30 | 2011-03-23 | 西安科技大学 | Data storing and reading method suitable for industrial process control |
| US9195657B2 (en) * | 2010-03-08 | 2015-11-24 | Microsoft Technology Licensing, Llc | Columnar storage of a database index |
| CN102737033B (en) * | 2011-03-31 | 2015-02-04 | 国际商业机器公司 | Data processing equipment and data processing method thereof |
| CN102521304A (en) * | 2011-11-30 | 2012-06-27 | 北京人大金仓信息技术股份有限公司 | Hash based clustered table storage method |
| US8782100B2 (en) * | 2011-12-22 | 2014-07-15 | Sap Ag | Hybrid database table stored as both row and column store |
| CN104794119B (en) * | 2014-01-17 | 2018-04-03 | 阿里巴巴集团控股有限公司 | Storage and transmission method and system for middleware message |
| CN104866508B (en) * | 2014-02-26 | 2019-05-03 | 中国电信股份有限公司 | The method and apparatus of file is managed under cloud environment |
| CN108334565A (en) * | 2018-01-15 | 2018-07-27 | 贵州易鲸捷信息技术有限公司 | A kind of data mixing storage organization, data store query method, terminal and medium |
| CN111159227B (en) * | 2019-12-20 | 2023-04-14 | 建信金融科技有限责任公司 | Data query method, device, equipment and storage medium |
-
2021
- 2021-12-17 CN CN202111552766.7A patent/CN114356226B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824574A (en) * | 2015-12-07 | 2016-08-03 | 国家电网公司 | Memory data storage method |
| CN109542898A (en) * | 2018-10-30 | 2019-03-29 | 天津字节跳动科技有限公司 | Date storage method, device, electronic equipment and the storage medium of data bank table |
| CN113656364A (en) * | 2021-08-05 | 2021-11-16 | 福瑞泰克智能系统有限公司 | Sensor data processing method, device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114356226A (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20130097402A1 (en) | Data prefetching method for distributed hash table dht storage system, node, and system | |
| CN102843396A (en) | Data writing and reading method and device in distributed caching system | |
| US20210170575A1 (en) | Robot cluster scheduling system | |
| CN110555001B (en) | Data processing method, device, terminal and medium | |
| CN103440290A (en) | Big data loading system and method | |
| CN108491163B (en) | Big data processing method and device and storage medium | |
| CN110737388A (en) | Data pre-reading method, client, server and file system | |
| CN112148221B (en) | Method, device, equipment and storage medium for inspecting redundant array of inexpensive disks | |
| CN112256433B (en) | Partition migration method and device based on Kafka cluster | |
| US10198180B2 (en) | Method and apparatus for managing storage device | |
| CN113438184B (en) | Network card queue management method and device and electronic equipment | |
| CN111198845A (en) | Data migration method, readable storage medium and computing device | |
| CN107562851A (en) | A kind of update method of data, device and electronic equipment | |
| CN108319634B (en) | Directory access method and device for distributed file system | |
| CN108132759B (en) | Method and device for managing data in file system | |
| CN108021562B (en) | Disk storage method and device applied to distributed file system and distributed file system | |
| CN112039999A (en) | Method and system for accessing distributed block storage system in kernel mode | |
| CN114237989B (en) | Database service deployment and disaster tolerance method and device | |
| CN104134027A (en) | Sequence flow recognition method and sequence flow recognition device | |
| CN111488323A (en) | Data processing method and device and electronic equipment | |
| CN111399753A (en) | Method and device for writing pictures | |
| CN114356226B (en) | Sensor data storage method, device, equipment and storage medium | |
| WO2017045500A1 (en) | Storage array management method and apparatus | |
| CN113296685B (en) | Data processing method and device and computer readable storage medium | |
| CN117520278A (en) | Multi-client high-precision directory quota control method for distributed file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |