CN114556283A

CN114556283A - Method and device for data writing, consistency checking and reading

Info

Publication number: CN114556283A
Application number: CN201980101436.0A
Authority: CN
Inventors: 张丰伟; 陈鹏; 周玉
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-11-13
Filing date: 2019-11-13
Publication date: 2022-05-27
Anticipated expiration: 2039-11-13
Also published as: CN114556283B; WO2021092784A1

Abstract

A method and apparatus for data write-in, consistency check and read, through writing into the storage medium through the multiple channels according to the order of sequential cycle in parallel a plurality of data blocks reorganized on the basis of the time stamp one by one, on one hand can make a plurality of blocks in the memorizer age evenly compared with the way of writing into in page by page in the prior art, have improved the life time of the memorizer, on the other hand can realize parallelization to a greater extent, improve the stability of data write-in; in addition, by using the timestamp and the block data index information, the data reading speed and stability can be improved, the service efficiency of the solid state disk can be particularly improved, and the application requirements of an automatic driving scene are better met.

Description

Method and device for writing data, checking consistency and reading data

Technical Field

The present application relates to storage technologies, and in particular, to a method and an apparatus for writing, checking consistency, and reading data.

Background

During the travel of the vehicle, it is very necessary to record various travel-related data because: on one hand, the recorded data can be used for positioning problems and input to an algorithm or a service module for simulation verification and model training, so that the development efficiency and the test efficiency of automatic driving are greatly improved; on the other hand, in the commercial operation of automatic driving, if an accident occurs, various vehicle-mounted data before and after the accident are required to be used as a part of an accident evidence chain for accident qualification.

The mainstream data storage technology in the prior art is a Solid State Disk (SSD) storage technology. The SSD memory consists mainly of a controller, firmware algorithms and flash memory granules: the controller is essentially a small processor, is a control center of the whole SSD memory, and is used for executing various instructions and algorithms for managing scheduling, reading and writing, deleting data and the like; the firmware algorithm is a program for driving the controller, and the controller uses a control program in the firmware algorithm to execute operations such as automatic signal processing, wear leveling, error checking and correcting, bad block management, garbage collection and encryption and the like; the flash memory grain is a storage medium in an SSD memory, and the controller operates a plurality of blocks (blocks) of flash memory grains in parallel through a plurality of channels (channels), wherein one block comprises 64 or 128 pages (pages), and the storage capacity of one page is typically 4 KB. SSD memory has the characteristics of "read and write per page" and "erase per block", and specifically, SSD memory has a page as a basic access unit, but must have a block as a minimum erase unit when erasing data.

Fig. 1 to 3 illustrate a write operation of an SSD memory with 8 channels as an example, in the prior art: in fig. 1 to 3, only one block is shown corresponding to each channel, and actually each channel corresponds to a plurality of blocks; the SSD controller controls 8 channels to be distributed with data according to the sequence circulation (the sequence circulation refers to that data are distributed in sequence according to the sequence that the channel numbers are reached from small, and the channel with the largest number is distributed and then continuously distributed from the channel with the smallest number), and controls 8 channels to perform write operation in parallel; when 4KB data is to be written initially, only one page of memory space is occupied, channel0 is taken as a starting channel (other channels can also be taken as starting channels) in fig. 1, and the 4KB data is written into one page in a block corresponding to channel0 (see light gray squares in fig. 1-3); if the data of 16KB needs to be continuously written and the storage space of four pages needs to be occupied, respectively writing the data of 4KB into one page in the block corresponding to each channel by using the channels 1 to 4 in parallel (see dark gray squares in FIGS. 2 to 3); if it is necessary to continue writing 20KB of data and five more pages of memory are needed, then channel 5, channel 6, channel 7, channel0 and channel1 are used in parallel to write 4KB of data into one page in the block corresponding to each channel (see black squares in fig. 2-3).

In the automatic driving process, data of various devices such as a plurality of cameras, laser radars, ultrasonic sensors and the like need to be recorded, complex algorithm and service module data also need to be recorded, the data volume is very large, the requirements of data from different sources on the reading and writing speed of a memory are greatly different, the number of files involved in data reading is large, and the requirements on a file management system are high. In the prior art, the data reading and writing efficiency of an automatic driving scene is low, the stability is poor, and the requirements of an automatic driving system cannot be met, so that the storage method in the prior art needs to be optimized.

Disclosure of Invention

In order to overcome the above problems in the prior art, embodiments of the present invention provide a method and an apparatus for data writing, consistency checking, and reading.

In a first aspect, a data writing method is provided, including: time stamping each of N1 packets, wherein N1 is a positive integer; sorting the N1 data packets according to the time sequence of the time stamps of the N1 data packets; sequentially organizing the N1 sorted data packets into a plurality of data pages based on the storage granularity of the pages, wherein each data page in the plurality of data pages comprises N3 data packets, and N3 is a positive integer; sequentially organizing the plurality of data pages into a plurality of data blocks based on the storage granularity of the blocks, wherein each data block in the plurality of data blocks comprises block data index information and N2 data packets, N2 is a positive integer, N1 is more than or equal to N2 is more than or equal to N3, the block data index information comprises the number of the data block, and the number of the plurality of data blocks has a sequential cyclic correspondence relationship with a plurality of channels for writing the plurality of data blocks into a storage medium; and respectively writing at least one data block corresponding to each channel in the plurality of channels into the storage medium sequentially through each channel according to the serial number of the at least one data block. Further, the time stamp is used to mark the time information of the data packet, such as the start of the collection time of the data in the data packet, the end of the collection time of the data in the data packet, or the time point of the data packet transmitted to the input interface of the memory. Further, the storage granularity of the page and the storage granularity of the block are inherent properties of the memory, for example, the storage granularity of one page is typically 4KB or 8KB, and the storage granularity of one block is 64 or 128 pages. Further, the block data index information exists in at least one data page in each data block, for example, in each data page in each data block, or only in the first data page in each data block. Further, writing multiple blocks of data through multiple channels is a parallel operation. Further, serial read and write operations are performed on a plurality of data pages within each data block through each channel. The sequence cycle is that a plurality of channels are allocated with data blocks in turn according to the sequence of the numbers of the data blocks from small to large, one data block is allocated to one data channel at a time, and after the last channel is allocated with one data block, the data blocks with continuous numbers are continuously allocated from the first allocated channel to the previous channel according to the turn sequence. The reorganized data blocks are written into the storage medium in parallel through the channels according to the sequence circulation order block by block, so that the blocks in the memory can be aged uniformly, the service life of the memory is prolonged, parallelization can be realized to a greater extent, and the stability of data writing is improved; in addition, by using the time stamp and the tile data index information, data reading speed and stability may be improved.

According to the first aspect, in a first possible implementation manner of the data writing method, the block data index information further includes the N₂Time information of each packet. The time information includes but is not limited to the N₂The start time, end time, or duration of the data within each data packet. By marking the time information of the data stored in the data block, block-level time management of the data content in the memory is facilitated, such as controlling reading and analyzing the data content in a specified time range.

According to the first aspect or the first possible implementation manner of the first aspect, in a second possible implementation manner of the data writing method, each data page further includes page data index information, and the page data index information includes the N₃Time information of each packet. The time information includes but is not limited to the N₃The start time, end time, or duration of the data within each data packet. By marking the time information of the data stored in the data page, the time management of the data content in the memory at the page level is facilitated, such as controlling to read and analyze the data content in a specified time range.

According to the first aspect or any one of the foregoing implementation manners of the first aspect, in a third possible implementation manner of the data writing method, each data packet corresponds to one topic type, and the block data index information further includes the N₂Subject type information of each packet. By marking the theme type information of the data stored in the data block, the management of the theme type of the data content in the memory is facilitated, such as controlling to read and analyze the data content of the specified theme type.

In a fourth possible implementation form of the data writing method according to the first aspect as such or any one of the implementation forms of the first aspect above, the topic type is used to identify a source or content of each data packet. Such as sensors that collect data or modules that process data. The content is, for example, audio data, video data, or image data.

In a second aspect, a data consistency checking method is provided, including: respectively checking whether a tail data page of the tail data block is free of data aiming at the tail data block last written in through each channel of a plurality of channels, wherein the plurality of channels are used for writing a plurality of data blocks into a storage medium, each data block of the plurality of data blocks comprises a plurality of data pages, at least one data page of the plurality of data pages comprises block data index information, the block data index information comprises the number of the data block, and a sequential cyclic corresponding relation exists between the number of the plurality of data blocks and the plurality of channels; when there is no data in the tail data page of at least one of the tail data blocks, acquiring a minimum number S1 of the at least one tail data block; checking whether the numbers of the plurality of data blocks among the plurality of data blocks written through the plurality of channels are consecutive according to the numbers of the plurality of tail data blocks written through the plurality of channels; when there is at least one discontinuity in the numbers of the plurality of data blocks, determining a starting number of the at least one discontinuity S2; determining the smaller value between S1 and S2; and erasing the data blocks with the numbers larger than or equal to the smaller value in the storage medium. Further, the erasure refers to physically deleting data from the storage medium. Further, when there is no tail data block of the tail data page without data, the smaller value between S1 and S2 is S2; when the numbers of the data blocks are consecutive, the smaller value between S1 and S2 is S1. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the number of the data blocks from small to large, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the channels are continuously allocated with the data blocks with the serial numbers from the first allocated channel to the previous channels according to the alternate sequence. By performing the above-described data consistency check operation within the data block, a data consistency check of a smaller unit of storage may be achieved relative to the prior art, thereby further optimizing the performance of the memory.

In a third aspect, a data reading method is provided, including: determining a correspondence between a plurality of channels and numbers of a plurality of data blocks by scanning block data index information of a data block last written through each of the plurality of channels, the plurality of data blocks being written to a storage medium through the plurality of channels, each of the plurality of data blocks including the block data index information, the block data index information including the number of the data block; sequentially reading block data index information of at least part of the data blocks in the plurality of data blocks according to the corresponding relation; determining one or more data blocks meeting preset requirements in the plurality of data blocks according to the block data index information of at least part of the data blocks; and sequentially analyzing and reading the storage contents in the one or more data blocks according to the number of the one or more data blocks. Wherein the at least part comprises part or all. One or more data blocks meeting the preset requirement can be determined only by reading the block data index information of at least part of the data blocks, so that the reading of all data in the whole data block is avoided, the reading speed and the reading stability can be improved, the data can be selectively read through the block data index information, the complexity of file management is reduced, and the method is more suitable for the application scene of automatic driving.

In a first possible implementation manner of the data reading method according to the third aspect, the block data index information includes at least one of time information or topic type information of a plurality of data packets in each data block, and the preset requirement includes at least one of a requirement on the time information or a requirement on the topic type information. The time information includes, but is not limited to, a start time, an end time, or a duration of data within the plurality of data packets. By marking the time information of the data stored in the data block, block-level time management of the data content in the memory is facilitated, such as controlling reading and analyzing the data content in a specified time range. By marking the theme type information of the data stored in the data block, the management of the theme type of the data content in the memory is facilitated, such as controlling to read and analyze the data content of the specified theme type.

In a second possible implementation manner of the data reading method according to the third aspect or the first possible implementation manner of the third aspect, the topic type is used to identify a source or content of each of the plurality of data packets. Such as sensors that collect data or modules that process data. The content is, for example, audio data, video data, or image data.

In a fourth aspect, there is provided a data writing apparatus comprising: a timestamp marking module for N₁Each of the data packets is time stamped, where N₁Is a positive integer; a sorting module for sorting the N₁Each data packet is according to the N₁Sequencing the time sequence of the timestamps of the data packets; a data page organization module for organizing the sorted N₁The storage granularity of the data packets based on the pages is sequentially organized into a plurality of data pages, and each data page in the plurality of data pages comprises N₃A data packet of N₃Is a positive integer; a data block organization module, configured to sequentially organize the multiple data pages into multiple data blocks based on a storage granularity of the blocks, where each data block of the multiple data blocks includes block data index information and N₂A data packet of which N₂Is a positive integer and N₁≥N ₂≥N ₃The block data index information comprises the number of the data block, and a sequential cyclic correspondence exists between the number of the data blocks and a plurality of channels for writing the data blocks into a storage medium; a channel writing module for respectively writing at least one data block corresponding to each of the plurality of channels according to the at least one channelAnd sequentially writing the serial numbers of the data blocks into the storage medium through each channel. Further, the time stamp is used to mark the time information of the data packet, such as the start of the collection time of the data in the data packet, the end of the collection time of the data in the data packet, or the time point of the data packet transmitted to the input interface of the memory. Further, the storage granularity of the page and the storage granularity of the block are inherent properties of the memory, for example, the storage granularity of one page is typically 4KB or 8KB, and the storage granularity of one block is 64 or 128 pages. Further, the block data index information exists in at least one data page in each data block, for example, in each data page in each data block, or only in the first data page in each data block. Further, writing multiple blocks of data through multiple channels is a parallel operation. Further, serial read and write operations are performed on a plurality of data pages within each data block through each channel. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the number of the data blocks from small to large, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the channels are continuously allocated with the data blocks with the serial numbers from the first allocated channel to the previous channels according to the alternate sequence. The reorganized data blocks are written into the storage medium in parallel through the channels according to the sequence circulation sequence block by block, so that the blocks in the memory can be aged uniformly, the service life of the memory is prolonged, parallelization can be realized to a greater extent, and the stability of data writing is improved; in addition, by using the time stamp and the tile data index information, data reading speed and stability may be improved.

In a first possible implementation manner of the data writing apparatus according to the fourth aspect, the block data index information further includes the N₂Time information of each packet. The time information includes but is not limited to the N₂The start time, end time, or duration of the data within each data packet.By marking the time information of the data stored in the data block, block-level time management of the data content in the memory is facilitated, such as controlling reading and analyzing the data content in a specified time range.

In a second possible implementation manner of the data writing apparatus according to the fourth aspect or the first possible implementation manner of the fourth aspect, each data page further includes page data index information, and the page data index information includes the N₃Time information of each packet. The time information includes but is not limited to the N₃The start time, end time, or duration of the data within each data packet. By marking the time information of the data stored in the data page, the time management of the data content in the memory at the page level is facilitated, such as controlling to read and analyze the data content in a specified time range.

According to a fourth aspect, or any implementation manner of the foregoing fourth aspect, in a third possible implementation manner of the data writing apparatus, each data packet corresponds to a topic type, and the chunk data index information further includes the N₂Subject type information of each packet. By marking the theme type information of the data stored in the data block, the management of the theme type of the data content in the memory is facilitated, such as controlling to read and analyze the data content of the specified theme type.

In a fourth possible implementation form of the data writing apparatus according to the fourth aspect as such or any one of the previous implementation forms of the fourth aspect, the topic type is used to identify a source or content of each data packet. Such as sensors that collect data or modules that process data. The content is, for example, audio data, video data, or image data.

In a fifth aspect, a data consistency checking apparatus is provided, including: a first checking module, configured to check whether a tail data page of a tail data block is free of data for the tail data block last written through each of a plurality of channels, respectively, where the plurality of channels are used to write a plurality of data blocks into a storage medium, each of the plurality of data blocks includes a plurality of data pages, at least one of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and a sequential cyclic correspondence exists between the number of the plurality of data blocks and the plurality of channels; a first number obtaining module, configured to obtain a minimum number S1 of at least one tail data block when there is no data in a tail data page of the at least one tail data block; a second checking module, configured to check whether numbers of the plurality of data blocks written through the plurality of channels are consecutive according to numbers of the plurality of tail data blocks written through the plurality of channels; a second number obtaining module, configured to determine, when at least one discontinuity exists in the numbers of the multiple data blocks, a starting number S2 of the at least one discontinuity; a comparison module for determining a smaller value between S1 and S2; and the data block erasing module is used for erasing the data blocks with the numbers larger than or equal to the smaller value in the storage medium. Further, the erasure refers to physically deleting data from the storage medium. Further, when there is no tail data block of the tail data page without data, the smaller value between S1 and S2 is S2; when the numbers of the data blocks are consecutive, the smaller value between S1 and S2 is S1. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the number of the data blocks from small to large, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the channels are continuously allocated with the data blocks with the serial numbers from the first allocated channel to the previous channels according to the alternate sequence. By performing the above-described data consistency check operation within the data block, a data consistency check of a smaller unit of storage may be achieved relative to the prior art, thereby further optimizing the performance of the memory.

In a sixth aspect, there is provided a data reading apparatus comprising: a correspondence determining module configured to determine a correspondence between a plurality of channels and numbers of a plurality of data blocks by scanning block data index information of a data block last written through each of the plurality of channels, the plurality of data blocks being written to a storage medium through the plurality of channels, each of the plurality of data blocks including the block data index information, the block data index information including the number of the data block; the block data index information reading module is used for sequentially reading the block data index information of at least part of the data blocks in the plurality of data blocks according to the corresponding relation; a to-be-read data block determining module, configured to determine, according to the block data index information of the at least part of data blocks, one or more data blocks that meet a preset requirement among the multiple data blocks; and the content analysis module is used for sequentially analyzing and reading the storage contents in the one or more data blocks according to the serial numbers of the one or more data blocks. Wherein the at least part comprises part or all. One or more data blocks meeting the preset requirement can be determined only by reading the block data index information of at least part of the data blocks, so that the reading of all data in the whole data block is avoided, the reading speed and the reading stability can be improved, the data can be selectively read through the block data index information, the complexity of file management is reduced, and the method is more suitable for the application scene of automatic driving.

In a first possible implementation manner of the data reading apparatus according to the sixth aspect, the block data index information includes at least one of time information or subject type information of a plurality of data packets in each data block, and the preset requirement includes at least one of a requirement on the time information or a requirement on the subject type information. The time information includes, but is not limited to, a start time, an end time, or a duration of data within the plurality of data packets. By marking the time information of the data stored in the data block, block-level time management of the data content in the memory is facilitated, such as controlling reading and analyzing the data content in a specified time range. By marking the theme type information of the data stored in the data block, the management of the theme type of the data content in the memory is facilitated, such as controlling to read and analyze the data content of the specified theme type.

In a second possible implementation form of the data reading apparatus according to the sixth aspect or the first possible implementation form of the sixth aspect, the topic type is used to identify a source or content of each of the plurality of data packets. Such as sensors that collect data or modules that process data. The content is, for example, audio data, video data, or image data.

In a seventh aspect, a data writing apparatus is provided, which includes a memory and a processor, the memory stores computer program instructions, and the processor executes the computer program instructions to perform the operations according to the first aspect or any one of the implementation manners of the first aspect.

In an eighth aspect, there is provided a data consistency checking apparatus comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform the operations of the second aspect.

In a ninth aspect, there is provided a data reading apparatus comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform operations as described in the third aspect or any one of the above implementations of the third aspect.

A tenth aspect provides a computer storage medium comprising computer instructions that, when executed by a processor, cause the data writing apparatus to perform a method according to the first aspect or any one of the above implementations of the first aspect.

In an eleventh aspect, there is provided a computer storage medium comprising computer instructions which, when executed by a processor, cause the data consistency checking apparatus to perform the method of the second aspect.

In a twelfth aspect, there is provided a computer storage medium comprising computer instructions which, when executed by a processor, cause the data reading apparatus to perform the method according to the third aspect or any one of the above implementation manners of the third aspect.

A thirteenth aspect provides a computer program product for causing a data writing apparatus to perform a method according to the first aspect or any one of the above implementations of the first aspect, when the computer program product runs on a processor.

In a fourteenth aspect, a computer program product is provided, which, when run on a processor, causes the data consistency checking apparatus to perform the method according to the second aspect.

A fifteenth aspect provides a computer program product for causing a data reading apparatus to perform a method according to the third aspect or any one of the above implementation manners of the third aspect, when the computer program product runs on a processor.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of writing 4KB of data within an SSD memory in the prior art.

FIG. 2 is a diagram of a prior art SSD memory write 4KB of data followed by 16KB of data.

FIG. 3 is a diagram of a prior art SSD memory with 20KB of data written to it followed by 20KB of data.

Fig. 4 is a hardware configuration diagram of a first exemplary application scenario of the present invention.

Fig. 5 is a hardware configuration diagram of a second exemplary application scenario of the present invention.

Fig. 6 is a hardware configuration diagram of a third exemplary application scenario of the present invention.

Fig. 7 is a flowchart of a data writing method according to an embodiment of the present invention.

Fig. 8 is a schematic diagram of a data writing method according to a second embodiment of the present invention.

Fig. 9 is a schematic diagram illustrating the composition of block data index information in the data writing method according to the second embodiment of the present invention.

Fig. 10 is a flowchart of a data consistency checking method according to a third embodiment of the present invention.

Fig. 11 is a schematic diagram of a data consistency checking method according to a fourth embodiment of the present invention.

Fig. 12 is a flowchart of a data reading method according to a fifth embodiment of the present invention.

Fig. 13 is a schematic diagram of a data reading method according to a sixth embodiment of the present invention.

Fig. 14 is a schematic diagram of a data reading method according to a seventh embodiment of the present invention.

Fig. 15 is a block diagram of a data writing device according to an eighth embodiment of the present invention.

Fig. 16 is a block diagram of a data consistency check apparatus according to a ninth embodiment of the present invention.

Fig. 17 is a block diagram of a data reading apparatus according to a tenth embodiment of the present invention.

Fig. 18 is a block diagram of an apparatus according to an eleventh embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 4-6 show three typical application scenarios of the present invention when the present invention is applied to an intelligent vehicle to implement driving data recording. It should be noted that the application scenarios of the present invention are not limited to vehicle data recording, and include all application scenarios involving data writing, consistency checking, or reading. The MDC (multi domaincontroller, multi-domain controller) in the intelligent vehicle can access the signals of different sensors, analyze and process the signals, and finally issue a control command, and fig. 4 to 6 implement the recording of driving data based on the MDC. It should be noted that, in practical applications, the recording of the driving data may also be implemented based on other components with control functions in the vehicle besides the MDC.

In fig. 4, a separate vehicle data recording subsystem 403 is provided in the MDC401 and dedicated to vehicle data recording, and the vehicle data recording subsystem 403 may be packaged separately as an aircraft black box to improve data security. An original main control CPU402 in the MDC401 receives driving data collected by a plurality of sensors (including a millimeter wave radar 406, an ultrasonic radar 407, a laser radar 408, a GPS module 409, and a camera 410), and sends the driving data to a driving data recording subsystem 403, and finally, the data is stored in a storage medium 405 of the driving data recording subsystem 403. The storage medium 405 is preferably implemented using flash memory pellets (or referred to as solid state storage pellets). The driving data recording subsystem 403 is further configured with a data processing chip 404, where the data processing chip 404 is present in a form including, but not limited to, a Central Processing Unit (CPU), a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Micro Controller Unit (MCU), or the like, and is used to implement data processing or control functions, and the data processing chip 404 may further include a storage device for storing part of data, control software, or drive firmware.

The internal structure of the MDC501 in fig. 5 is the same as the MDC401 in fig. 4, except that the main control CPU502 in fig. 5 does not participate in the recording of driving data, which is received directly from a plurality of sensors (including a millimeter wave radar 506, an ultrasonic radar 507, a laser radar 508, a GPS module 509, and a camera 510) by a driving data recording subsystem 503.

Unlike fig. 4 and 5, an independent driving data recording subsystem is provided in the MDC for driving recording, in fig. 6, an external memory 603 is externally connected to the MDC601, and the external memory 603 is preferably an SSD solid state disk. Similar to the internal structure of a general solid state disk, the external memory 603 includes a main control chip 604 and a flash memory chip 605. The main control CPU602 in the MDC601 receives the driving data collected by a plurality of sensors (including the millimeter wave radar 606, the ultrasonic radar 607, the laser radar 608, the GPS module 609, and the camera 610), and sends the driving data to the external memory 603, and finally the data is stored in the flash memory chip 605 of the external memory 603.

An embodiment of the present invention provides a data writing method, referring to fig. 1, including:

step 701, is N₁Each of the data packets is time stamped, where N₁Is a positive integer.

In this embodiment, N is₁Each of the data packets is from a data acquisition device, N₁The data packets are from a plurality of different data acquisition devices (such as cameras, laser radars, millimeter wave radars, ultrasonic sensors or human-computer interaction devices) and each have a frame structure, so that attribute information, particularly time information, of each data packet can be identified, an acquisition time starting point of data in each data packet is obtained, and a time stamp is marked for each data packet by the acquisition time starting point. The time stamp can also be used to mark other time information, such as the end of the acquisition time of the data in the data packet or the point in time at which the data packet is transmitted to the memory input interface.

Step 702, converting the N₁Each data packet is according to the N₁The time sequence of the time stamps of the data packets is ordered.

Due to the N₁The data packets relate to different theme types, so the step is to uniformly sort the data packets of a plurality of theme types based on the time information.

Step 703, sorting the N₁The storage granularity of the data packets based on the pages is sequentially organized into a plurality of data pages, and each data page in the plurality of data pages comprises N₃A data packet of which N₃Is a positive integer.

Each data page comprises page data index information used for bearing the N₃Time information and subject type information of each data packet, the subject type information being used to identify the N₃The data in each data packet comes from the data acquisition equipment, and the time information is N₃The start time and the end time of the whole of the recorded content within each data packet, and the start time and the end time of the recorded content for each topic type.

Step 704, organizing the multiple data pages into multiple data blocks in sequence based on the storage granularity of the blocks, where each data block of the multiple data blocks includes block data index information and N₂A data packet of which N₂Is a positive integer and N₁≥N ₂≥N ₃The block data index information includes the number of the data block, and there is a sequential cyclic correspondence between the number of the data blocks and a plurality of channels for writing the data blocks to a storage medium. Further, the tile data index information may further include the N₂Time information of each packet. Each data packet corresponds to a topic type, and the block data index information further includes the N₂Subject type information of each packet. The topic type is used to identify the source or content of each data packet.

In this embodiment, the storage granularity of a page is 4KB, and one block includes 128 pages, so that the storage granularity of a block is 512 KB. The block data index information exists only in the first data page of each data block. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the number of the data blocks from small to large, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the channels are continuously allocated with the data blocks with the serial numbers from the first allocated channel to the previous channels according to the alternate sequence.

Step 705, writing at least one data block corresponding to each channel of the plurality of channels into the storage medium sequentially through each channel according to the number of the at least one data block. Wherein writing the plurality of block data through the plurality of channels is a parallel operation, and writing the plurality of data pages within each data block through each channel is a serial operation.

An embodiment of the present invention also provides a data writing method, referring to fig. 8, including:

step 801, a packet sequence A { A }₀，A ₁，……}、B{B ₀，B ₁，……}、C{C ₀，C ₁，……}、D{D ₀，D ₁… … } and E { E }₀，E ₁… … is time stamped.

The laser radar, the millimeter wave radar, the camera, the robot operating system and the map manager are vehicle-mounted data acquisition equipment which respectively pack real-time acquired data into a data packet sequence A { A }₀，A ₁，……}、B{B ₀，B ₁，……}、C{C ₀，C ₁，……}、D{D ₀，D ₁… … } and E { E }₀，E ₁… …, and the above-mentioned packet sequences have different generation rates. In order to facilitate later test analysis and problem location, the data packet needs to be stored. Since each data packet contains data acquisition time information, each data packet may be time-stamped t in step 801 according to the data acquisition time information _i。

Step 802, sorting the data packets in the data packet sequences A-E according to the time sequence of the timestamps to obtain a data packet sequence X { B } including data packets from multiple data acquisition devices₀，C ₀，B ₁，D ₀，E ₀，C ₁，A ₀，A ₁，E ₁，D ₁，……}。

Step 803, the data packet sequence X is organized into a plurality of data pages in sequence based on the storage granularity of the pages. In this embodiment, the storage granularity of the page is 4K, and after considering the storage space occupied by the header information of the data packet and the data page, each data page can bear N₂Payload information of each packet.

Step 804, the multiple data pages obtained in step 803 are sequentially organized into multiple data blocks based on the storage granularity of the blocks, and the first data page of each data block includes block data index information. Fig. 9 illustrates an exemplary composition of the block data index information, including the following fields:

a Page Header Type, wherein the Header information is designated to be block data index information or Page data index information, and each data Page further comprises Page data index information in the second implementation, so that the first data Page of each data block comprises both the block data index information and the Page data index information, and the field is used for distinguishing the two types of Header information;

channel Num: the number of channels;

channel ID: the channel number of the block/page;

the number of the theme types related to the record of the local block/page;

the first record offset of the 0 th theme type recorded in the block/page;

the last record offset of the 0 th theme type recorded in the block/page;

start Time: starting time of all records of the block/page;

end Time: the termination time of all records of the block/page;

serial Num: the number of the data blocks is globally unique, the number of the data blocks is restarted from 0 after reaching the maximum value, the number of the data blocks obtained in the step 504 is started from 0, and the data blocks are { Block-0, Block-1, Block-2, … …, Block-m, … … }, and the number of the data blocks can also be started from other numbers according to the situation;

dlen: the effective data length (including header information) in one block/page.

And step 805, respectively writing the data blocks { Block-0, Block-1, Block-2, … …, Block-m, … … } into the storage medium in a channel parallel writing mode sequentially through channels respectively corresponding to the data blocks according to the serial numbers of the data blocks.

Wherein, there is a sequential loop correspondence relationship between the numbers of the data blocks and the channels, and the meaning of the sequential loop is described as an example in step 805 in fig. 8:

in fig. 8, 4 channels are used to perform the write operation, where the 4 channels are respectively Channel0, Channel1, Channel2 and Channel3, { Block-0, Block-4, Block-8, … … } writes to the storage medium through Channel0, { Block-1, Block-5, Block-9, … … } writes to the storage medium through Channel1, { Block-2, Block-6, Block-10, … … } writes to the storage medium through Channel2, { Block-3, Block-7, Block-11, … … } writes to the storage medium through Channel3, and the Block-by-Block write operation is performed in each Channel according to the number of the data Block, for example, the Block-0 is first written through Channel0, and the Block-0 write operation of the Block-1 through Channel0 is performed after the Block-0 write operation is completed.

In addition to sequential loops, the 4 channels follow the provision of parallel writes, such as Block-0 via Channel0, Block-1 via Channel1, Block-2 via Channel2, and Block-3 via Channel3, all performed simultaneously, although there may be a difference in the progress of the write operation between the channels, such as Block-6 via Channel2, the Block-0 via Channel0 may not be completed.

In the data writing method according to the first and second embodiments of the present invention, the plurality of data blocks reorganized based on the timestamp are written into the storage medium in parallel through the plurality of channels block by block according to the sequential and cyclic order, so that on one hand, compared with the prior art, the method of writing in page by page can uniformly age the plurality of data blocks in the memory, thereby prolonging the service life of the memory, and on the other hand, parallelization can be realized to a greater extent, thereby improving the stability of data writing; in addition, by using the time stamp and the tile data index information, data reading speed and stability may be improved.

Embodiments three and four of the present invention respectively provide a data consistency check method, which can perform consistency check on data written into a storage medium by the data writing method described in embodiment one or embodiment two of the present invention.

Referring to fig. 10, the third embodiment of the present invention includes:

step 1001, respectively checking whether a tail data page of a tail data block last written through each of a plurality of channels for writing the plurality of data blocks into a storage medium, where each of the plurality of data blocks includes a plurality of data pages, at least one of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and there is a sequential circular correspondence between the number of the plurality of data blocks and the plurality of channels. The sequence cycle is specifically that a plurality of channels are allocated with data blocks in turn according to the sequence of the number of the data blocks from small to large, and one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the data blocks with the consecutive number are allocated from the first allocated channel to the previous channel in turn according to the previous sequence, which can be referred to as step 505 in fig. 8. In the data writing method described in the first embodiment or the second embodiment of the present invention, one reason for the consistency error of the written data is that the single data block is not completely written, that is, a single data block misses a partial data page. Since the plurality of data blocks in each channel are written block by block sequentially according to the number of the data blocks, and a plurality of data pages are usually written page by page sequentially in each data block, it is only necessary to check whether the trailing data page of the trailing data block last written by each channel is free of data (representation of no data such as data full FF).

Step 1002, when there is at least one tail data page of the tail data block without data, acquiring a minimum number S1 of the at least one tail data block. When there is no data in the tail data page of the tail data block, that is, the tail data blocks of all channels are full of data, the S1 is marked as null.

Step 1003, checking whether the numbers of the data blocks in the data blocks written through the channels are continuous according to the numbers of the tail data blocks written through the channels. In the data writing method described in the first embodiment or the second embodiment of the present invention, another reason that the written data has consistency errors is that the writing schedules of multiple channels are not matched, for example, when Block-6 is written through Channel2, the operation of writing Block-0 through Channel0 is still not completed, because the numbering between multiple data blocks written through multiple channels represents the chronological order of the timestamps of the multiple data blocks, if Block-0 is not completely written, all data blocks numbered later are still not meaningful to use even if they are completely written.

In step 1004, the problem of the number discontinuity may exist in one or more places, and when there is at least one discontinuity in the numbers of the plurality of data blocks, the at least one discontinuity start number S2 is determined. The start number S2 can also be understood as the minimum value of the number of the missing partial data blocks when there is at least one discontinuity in the number of the plurality of data blocks due to the missing partial data blocks. When the numbers of the data blocks in the data blocks written through the channels are all consecutive, marking the S2 as null value.

In step 1005, the smaller value between S1 and S2 is determined. It should be noted that, when there is no tail data block of the tail data page without data, the smaller value between S1 and S2 is S2; when the numbers of the data blocks are consecutive, the smaller value between S1 and S2 is S1.

Step 1006, erasing the data block with the number greater than or equal to the smaller value in the storage medium. Note that the erasure refers to physically deleting data from the storage medium.

Referring to fig. 11, an embodiment of the present invention includes:

step 1101, the memory comprises four channels, namely Channel0, Channel1, Channel2 and Channel3, the tail data blocks of the four channels are Block-12, Block-9, Block-6 and Block-15, respectively, and whether the tail data pages of the Block-12, Block-9, Block-6 and Block-15 are free of data (namely whether the tail data pages are full FF) is checked.

In step 1102, after the check of step 1101, the trailer data pages of Block-9 and Block-15 are found to have no data, and S1 is determined to be the smaller of 9 and 15, that is, the flag S1 is 9.

Step 1103, checking whether the numbers of the data blocks written by Channel0, Channel1, Channel2 and Channel3 are consecutive according to the numbers of Block-12, Block-9, Block-6 and Block-15. The specific checking means is to divide the numbers of Block-12, Block-9, Block-6 and Block-15 by the number of channels 4, and the values of the four quotients are not equal, thereby determining the number discontinuity of the data blocks among the data blocks written through Channel0, Channel1, Channel2 and Channel 3.

At step 1104, the maximum of the four quotients is 3, the quotient of Channel1 divided by Channel number 4, and the quotient of Channel2 divided by Channel number 4 are all less than 3, so that it can be determined that the data blocks causing the number discontinuity of the data blocks are Block-13 (corresponding to Channel1), Block-10 (corresponding to Channel2), and Block-14 (corresponding to Channel2), thereby determining S2 as the smaller value of 13, 10, and 14, i.e., the flag S2 is 10.

Step 1105, determine S₁And S₂ Smaller value therebetweenIs 9;

and step 1106, erasing Block-9, Block-11, Block-12 and Block-15 in the storage medium.

In the data consistency check method according to the third and fourth embodiments of the present invention, by performing the data consistency check operation in the data block, the data consistency check in a smaller storage unit can be implemented compared to the prior art, so as to further optimize the performance of the memory.

An embodiment of the present invention provides a data reading method, referring to fig. 12, including:

step 1201, determining a correspondence between a plurality of channels and numbers of a plurality of data blocks by scanning block data index information of a last written data block through each of the plurality of channels, the plurality of data blocks being written to a storage medium through the plurality of channels, each of the plurality of data blocks including the block data index information, the block data index information including the number of the data block. The correspondence represents the channels to which each data block is allocated, and the first and last data blocks and the channels corresponding to the first and last data blocks can be determined through step 1201.

Step 1202, sequentially reading block data index information of at least some data blocks of the plurality of data blocks according to the corresponding relationship. The at least part of the data blocks comprises part or all of the data blocks, and step 1202 is to read the block data index information of each data block in the part or all of the data blocks according to the number sequence of the data blocks.

Step 1203, determining one or more data blocks of the plurality of data blocks that meet a preset requirement according to the block data index information of the at least part of data blocks. The block data index information includes time information of a plurality of data packets within each data block, the time information including but not limited to a start time, an end time or a duration of data within the plurality of data packets, and topic type information for identifying a source of each of the plurality of data packets, such as a sensor acquiring data or a module processing data, or a content, such as audio data, video data or image data. By marking the time information of the data stored in the data block, block-level time management of the data content in the memory is facilitated, such as control of reading and analysis of the data content in a specified time range; by marking the theme type information of the data stored in the data block, the management of the theme type of the data content in the memory is facilitated, such as controlling to read and analyze the data content of the specified theme type.

Step 1204, sequentially parsing and reading the storage contents in the one or more data blocks according to the numbers of the one or more data blocks. The parsing includes, but is not limited to, decapsulating the data block, obtaining payload data in the data block, decoding or decompressing, and the reading includes, but is not limited to, playing audio and video, rendering image, rendering text, or transferring to other digital signal processing devices.

An embodiment of the present invention provides a data reading method, configured to read data from a millimeter wave radar stored in a memory, and the specific steps are as shown in fig. 13, where the method includes:

step 1301, scanning Block data index information of a data Block (Block-m, Block-n, Block-p and Block-q respectively) written in each Channel of Channel0, Channel1, Channel2 and Channel3 at last, and determining corresponding relations between the four channels and data Block numbers of { Block-0, Block-1, … … and Block-q }. Each data Block of { Block-0, Block-1, … …, Block-q } includes Block data index information including the data Block number, { Block-0, Block-1, … …, Block-q } writes to the storage medium via Channel0, Channel1, Channel2 and Channel3, which have a sequential round-robin correspondence, as { Block-0, Block-4, Block-8, … …, Block-m } corresponds to Channel0, { Block-1, Block-5, Block-9, … …, Block-n } corresponds to Channel1, { Block-2, Block-6, Block-10, … …, Block-p } corresponds to Channel2, { Block-3, Block-7, Block-11, … …, Block-q } corresponds to Channel 3.

And step 1302, sequentially reading the Block data index information of each data Block in { Block-0, Block-1, … …, Block-q } in the sequence from Block-0 or in the reverse sequence from Block-q according to the number of the data Block.

Step 1303, since the Block data index information includes the acquisition source of the data record in each data Block, it can be determined that { Block-0, Block-1, … …, Block-q } includes data blocks acquired from the millimeter wave radar, that is, data blocks { Block-2, Block-4, Block-5, Block-7, Block-9, … …, Block-m, Block-n, Block-q }.

Step 1304, sequentially analyzing and reading storage contents in the data blocks { Block-2, Block-4, Block-5, Block-7, Block-9, … …, Block-m, Block-n and Block-q }, and restoring a data packet sequence { B-B acquired by the millimeter wave radar₀，B ₁… … }.

The seventh embodiment of the invention provides a data reading method for reading data stored in a memory with the data acquisition time T₁～T ₂Data within the range, see fig. 14 for specific steps, includes:

step 1401, scanning Block data index information of a data Block (Block-m, Block-n, Block-p and Block-q, respectively) written last by each of four channels of Channel0, Channel1, Channel2 and Channel3, and determining a corresponding relationship between the four channels and a data Block number of { Block-0, Block-1, … …, Block-q }. Each data Block of { Block-0, Block-1, … …, Block-q } includes Block data index information including the data Block number, { Block-0, Block-1, … …, Block-q } writes to the storage medium via Channel0, Channel1, Channel2 and Channel3, which have a sequential round-robin correspondence, as { Block-0, Block-4, Block-8, … …, Block-m } corresponds to Channel0, { Block-1, Block-5, Block-9, … …, Block-n } corresponds to Channel1, { Block-2, Block-6, Block-10, … …, Block-p } corresponds to Channel2, { Block-3, Block-7, Block-11, … …, Block-q } corresponds to Channel 3.

Step 1402, reading Block data index information of each data Block in { Block-0, Block-1, … …, Block-q } according to the numbering sequence of the data blocks from Block-0, wherein the Block data index information includes the acquisition time of the data record in each data Block, and finding that the latest acquisition time of the data record in the data Block is later than T when the Block data index information of Block-9 is read₂Because the serial numbers of the data blocks can show the sequence of the acquisition time of the data records in the blocks, the acquisition time of the data records in the data blocks with the serial numbers larger than 9 is certainly later than T₂Therefore, reading the Block data index information of the data blocks after Block-9, i.e. reading the Block data index information of a part of the data blocks { Block-0, Block-1, … …, Block-q }, can be abandoned.

Step 1403, determining that the acquisition time in the Block-5, Block-6, Block-7, Block-8 and Block-9 is T in the Block data index information by reading the Block data index information₁～T ₂Data within the range.

Step 1404, sequentially analyzing and reading the storage contents in the data blocks { Block-5, Block-6, Block-7, Block-8 and Block-9}, and restoring the acquisition time T₁～T ₂Data content within the range.

In the fifth, sixth and seventh embodiments of the present invention, since one or more data blocks meeting the preset requirement can be determined only by reading the block data index information of at least some data blocks, reading all data in the entire data block is avoided, so that the reading speed and stability can be improved, and data can be selectively read through the block data index information, thereby reducing the complexity of file management and being more suitable for the application scenario of automatic driving.

An eighth embodiment of the present invention provides a data writing apparatus, referring to fig. 15, including:

a timestamp marking module 1501 for N ₁Each of the data packets is time stamped, where N₁Is a positive integer. The time stamp is used to mark the time information of the data packet, such as the start of the collection time of the data in the data packet, the end of the collection time of the data in the data packet, or the time point of the data packet transmitted to the input interface of the memory.

A sorting module 1502 for sorting the N₁Data packet according to said N₁The time sequence of the time stamps of the data packets is ordered.

A data page organization module 1503 for organizing the sorted N₁The storage granularity of the data packets based on the pages is sequentially organized into a plurality of data pages, and each data page in the plurality of data pages comprises N₃A data packet of which N₃Is a positive integer.

A data block organizing module 1504, configured to sequentially organize the plurality of data pages into a plurality of data blocks based on the storage granularity of the blocks.

The storage granularity of the page and the storage granularity of the block are inherent properties of the memory, for example, the storage granularity of one page is usually 4KB or 8KB, and the storage granularity of one block is 64 pages or 128 pages.

Each of the plurality of data chunks includes chunk data index information and N₂A data packet of which N₂Is a positive integer and N₁≥N ₂≥N ₃。

The block data index information includes the number of the data blocks and exists for each data page in each data block. Each data Page further includes Page data index information, and whether the Header information is tile data index information or Page data index information is distinguished by a "Page Header Type" field in fig. 9.

There is a sequential cyclical correspondence between the number of the plurality of data blocks and a plurality of channels used to write the plurality of data blocks to the storage medium. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the number of the data blocks from small to large, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the channels are continuously allocated with the data blocks with the serial numbers from the first allocated channel to the previous channels according to the alternate sequence.

The block data index information further includes the N₂Time information of each packet. The time information includes but is not limited to the N₂The start time, end time, or duration of the data within each data packet. By marking the time information of the data stored in the data block, block-level time management of the data content in the memory is facilitated, such as controlling reading and analyzing the data content in a specified time range.

Each data packet corresponds to a topic type, and the block data index information further includes the N₂Topic type information of each data packet. The topic type is used to identify the source or content of each data packet. Such as sensors that collect data or modules that process data. The content is, for example, audio data, video data, or image data. By marking the theme type information of the data stored in the data block, the management of the theme type of the data content in the memory is facilitated, such as controlling to read and analyze the data content of the specified theme type.

The channel writing module 1505 is configured to write at least one data block corresponding to each of the plurality of channels into the storage medium sequentially through each of the channels according to the number of the at least one data block. The method comprises the steps of executing serial writing operation on a plurality of data pages in each data block through each channel, and writing a plurality of block data through a plurality of channels in parallel.

According to the data writing device, the multiple data blocks reorganized based on the timestamps are written into the storage medium in parallel through the multiple channels according to the sequence circulation sequence block by block, on one hand, compared with a page-by-page writing mode in the prior art, the multiple blocks in the memory can be aged uniformly, the service life of the memory is prolonged, on the other hand, parallelization can be achieved to a greater extent, and the stability of data writing is improved; in addition, by using the time stamp and the tile data index information, data reading speed and stability may be improved.

An embodiment of the present invention provides a data consistency checking apparatus, referring to fig. 16, including:

a first checking module 1601, configured to check whether a tail data page of a tail data block is free of data for a last written tail data block through each of a plurality of channels, respectively, where the plurality of channels are used to write a plurality of data blocks into a storage medium, each of the plurality of data blocks includes a plurality of data pages, at least one of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and there is a sequential cyclic correspondence between the number of the plurality of data blocks and the plurality of channels. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the number of the data blocks from small to large, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the channels are continuously allocated with the data blocks with the serial numbers from the first allocated channel to the previous channels according to the alternate sequence.

The first number obtaining module 1602, configured to obtain the minimum number S1 of at least one tail data block when there is no data in a tail data page of the at least one tail data block.

A second checking module 1603, configured to check whether the numbers of the data blocks in the data blocks written through the channels are consecutive according to the numbers of the tail data blocks written through the channels.

The second number obtaining module 1604 is configured to determine a starting number S2 of at least one discontinuity when there is at least one discontinuity in the numbers of the plurality of data blocks.

A comparison module 1605 for determining the lesser value between S1 and S2. When the tail data block without data of the tail data page is not stored, the smaller value between S1 and S2 is S2; when the numbers of the data blocks are consecutive, the smaller value between S1 and S2 is S1.

A data block erasing module 1606, configured to erase the data block with the number greater than or equal to the smaller value in the storage medium. The erasure refers to physically deleting data from the storage medium.

In the ninth embodiment of the present invention, by performing the data consistency check operation in the data block, compared with the prior art, the data consistency check of a smaller storage unit can be implemented, so as to further optimize the performance of the memory.

An embodiment of the present invention provides a data reading apparatus, see fig. 17, including:

a correspondence determining module 1701 for determining correspondence between the plurality of channels and numbers of a plurality of data blocks written to the storage medium through the plurality of channels by scanning block data index information of a data block last written through each of the plurality of channels, each of the plurality of data blocks including the block data index information including the number of the data block.

A block data index information reading module 1702, configured to sequentially read block data index information of at least some data blocks in the plurality of data blocks according to the corresponding relationship. Wherein the at least part comprises part or all.

A to-be-read data block determining module 1703, configured to determine one or more data blocks, which meet a preset requirement, of the multiple data blocks according to the block data index information of the at least part of data blocks.

The block data index information includes time information or topic type information of a plurality of data packets in each data block, and the preset requirement includes a requirement on the time information or a requirement on the topic type information. The time information includes, but is not limited to, a start time, an end time, or a duration of data within the plurality of data packets. The topic type is used to identify the source or content of each of the plurality of data packets. Such as sensors that collect data or modules that process data. The content is, for example, audio data, video data, or image data. By marking the time information of the data stored in the data block, block-level time management of the data content in the memory is facilitated, such as controlling reading and analyzing the data content in a specified time range. By marking the theme type information of the data stored in the data block, the management of the theme type of the data content in the memory is facilitated, such as controlling to read and analyze the data content of the specified theme type.

A content analysis module 1704, configured to sequentially analyze and read the storage content in the one or more data blocks according to the number of the one or more data blocks.

The one or more data blocks meeting the preset requirement can be determined only by reading the block data index information of at least part of the data blocks, and the embodiment of the invention avoids reading all data in the whole data block, so that the reading speed and the stability can be improved, and the data can be selectively read through the block data index information, thereby reducing the complexity of file management and being more suitable for the application scene of automatic driving.

Only one or more of the various modules in the above eight, nine and ten embodiments may be implemented in software, hardware, firmware or a combination thereof. The software or firmware includes, but is not limited to, computer program instructions or code and may be executed by a hardware processor. The hardware includes, but is not limited to, various types of integrated circuits such as the aforementioned master CPU402, master CPU602, data processing chip 404, data processing chip 504, or master chip 604.

An apparatus according to an eleventh embodiment of the present invention, referring to fig. 18, includes a memory 1801 and a processor 1802, where the memory 1801 stores computer program instructions, and the processor 1802 executes the computer program instructions to perform operations according to any one or more of the first to seventh embodiments of the present invention. The processor in the eleventh embodiment of the present invention may be the aforementioned main control CPU402, main control CPU602, data processing chip 404, data processing chip 504, or main control chip 604.

When the present invention is applied to an automatic driving scenario, the modules or devices in the apparatus may be implemented by original devices, devices or modules in a vehicle-mounted system, for example, the memory 1801 may be the main control CPU402, the main control CPU602, the data processing chip 404, the data processing chip 504 or a memory device in the main control chip 604, the memory 1801 may also be the storage medium 405, the storage medium 505 or the flash memory chip 605, and the processor 1802 may be the main control CPU402, the main control CPU602, the data processing chip 404, the data processing chip 504 or the main control chip 604. In addition, the memory 1801 and the processor 1802 may be implemented in the same device or may be implemented by two different devices or apparatuses.

It is clear to a person skilled in the art that the descriptions of the embodiments provided in the present application may be referred to each other, and for convenience and brevity of the description, for example, the functions and the steps performed by the devices and the apparatuses provided in the embodiments of the present application may be referred to the relevant descriptions of the method embodiments of the present application, and the method embodiments and the device embodiments may be referred to each other.

Those skilled in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs all or part of the steps comprising the method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways without departing from the scope of the application. For example, the above-described embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement without inventive effort.

Additionally, the apparatus and methods described, as well as the illustrations of various embodiments, may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present application. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some interfaces, and may be in an electronic, mechanical or other form.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A method of writing data, comprising:

is N₁Each of the data packets is time stamped, where N₁Is a positive integer;

the N is₁Each data packet is according to the N₁Sequencing the time sequence of the timestamps of the data packets;

the N after the sorting is carried out₁The storage granularity of the data packets based on the pages is sequentially organized into a plurality of data pages, and each data page in the plurality of data pages comprises N₃A data packet of which N₃Is a positive integer;

sequentially organizing the plurality of data pages into a plurality of data blocks based on the storage granularity of the blocks, wherein each data block in the plurality of data blocks comprises block data index information and N₂A data packet of which N₂Is a positive integer and N₁≥N ₂≥N ₃The block data index information comprises the data areaThe serial numbers of the blocks, and the serial numbers of the data blocks and a plurality of channels for writing the data blocks into the storage medium have sequential cyclic correspondence;

and respectively writing at least one data block corresponding to each channel in the plurality of channels into the storage medium sequentially through each channel according to the serial number of the at least one data block.
The method of claim 1, wherein the chunk data index information further comprises the N₂Time information of each packet.
The method of claim 1 or 2, wherein each data page further comprises page data index information, the page data index information comprising the N₃Time information of each packet.
The method according to any one of claims 1 to 3, wherein each data packet corresponds to a topic type, and the chunk data index information further includes the N₂Subject type information of each packet.
The method of claim 4, wherein the topic type is used to identify a source or content of the each data packet.
A data consistency checking method is characterized by comprising the following steps:

respectively checking whether a tail data page of the tail data block is free of data aiming at the tail data block last written in through each channel of a plurality of channels, wherein the plurality of channels are used for writing a plurality of data blocks into a storage medium, each data block of the plurality of data blocks comprises a plurality of data pages, at least one data page of the plurality of data pages comprises block data index information, the block data index information comprises the number of the data block, and a sequential cyclic corresponding relation exists between the number of the plurality of data blocks and the plurality of channels;

when the tail data page of at least one tail data block has no data, acquiring the minimum number S of the at least one tail data block₁；

Checking whether the numbers of the plurality of data blocks among the plurality of data blocks written through the plurality of channels are consecutive according to the numbers of the plurality of tail data blocks written through the plurality of channels;

when at least one discontinuity exists in the numbers of the plurality of data blocks, determining a starting number S of the at least one discontinuity₂；

Determination of S₁And S₂The smaller value in between;

and erasing the data blocks with the numbers larger than or equal to the smaller value in the storage medium.
A data reading method, comprising:

determining a correspondence between a plurality of channels and numbers of a plurality of data blocks by scanning block data index information of a data block last written through each of the plurality of channels, the plurality of data blocks being written to a storage medium through the plurality of channels, each of the plurality of data blocks including the block data index information, the block data index information including the number of the data block;

sequentially reading block data index information of at least part of the data blocks in the plurality of data blocks according to the corresponding relation;

determining one or more data blocks meeting preset requirements in the plurality of data blocks according to the block data index information of at least part of the data blocks;

and sequentially analyzing and reading the storage contents in the one or more data blocks according to the serial numbers of the one or more data blocks.
The method of claim 7, wherein the block data index information comprises at least one of time information or topic type information of a plurality of data packets within each data block, and wherein the preset requirement comprises at least one of a requirement regarding the time information or a requirement regarding the topic type information.
The method of claim 8, wherein the topic type is used to identify a source or content of each of the plurality of data packets.
A data writing apparatus, comprising:

a timestamp marking module for N₁Each of the data packets is time stamped, where N₁Is a positive integer;

a sorting module for sorting the N₁Each data packet is according to the N₁Sequencing the time sequence of the timestamps of the data packets;

a data page organization module for organizing the sorted N₁The storage granularity of the data packets based on the pages is sequentially organized into a plurality of data pages, and each data page in the plurality of data pages comprises N₃A data packet of which N₃Is a positive integer;

a data block organization module, configured to sequentially organize the multiple data pages into multiple data blocks based on a storage granularity of the blocks, where each data block of the multiple data blocks includes block data index information and N₂A data packet of which N₂Is a positive integer and N₁≥N ₂≥N ₃The block data index information comprises the number of the data block, and a sequential cyclic correspondence exists between the number of the data blocks and a plurality of channels for writing the data blocks into a storage medium;

and a channel writing module, configured to write at least one data block corresponding to each of the multiple channels into the storage medium sequentially through each channel according to the number of the at least one data block.
The apparatus of claim 10, wherein the block data index information further comprises the N₂Time information of each data packet.
The apparatus of claim 10 or 11, wherein each data page further comprises page data index information, the page data index information comprising the N₃Time information of each packet.
The apparatus according to any one of claims 10-12, wherein each data packet corresponds to a topic type, and the chunk data index information further comprises the N₂Subject type information of each packet.
The apparatus of claim 13, wherein the topic type is used to identify a source or content of the each data packet.
A data consistency check apparatus, comprising:

a first checking module, configured to check whether a tail data page of a tail data block is free of data for a tail data block last written through each of a plurality of channels, respectively, where the plurality of channels are used to write a plurality of data blocks into a storage medium, each of the plurality of data blocks includes a plurality of data pages, at least one of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and a sequential cyclic correspondence exists between the number of the plurality of data blocks and the plurality of channels;

a first number obtaining module, configured to obtain a minimum number S of at least one tail data block when there is no data in a tail data page of the at least one tail data block₁；

A second checking module, configured to check whether numbers of the plurality of data blocks written through the plurality of channels are consecutive according to numbers of the plurality of tail data blocks written through the plurality of channels;

a second number obtaining module, configured to determine, when at least one discontinuity exists in the numbers of the multiple data blocks, a starting number S of the at least one discontinuity₂；

A comparison module for determining S₁And S₂The smaller value in between;

and the data block erasing module is used for erasing the data blocks with the numbers larger than or equal to the smaller value in the storage medium.
A data reading apparatus, comprising:

a correspondence determining module configured to determine a correspondence between a plurality of channels and numbers of a plurality of data blocks by scanning block data index information of a data block last written through each of the plurality of channels, the plurality of data blocks being written to a storage medium through the plurality of channels, each of the plurality of data blocks including the block data index information, the block data index information including the number of the data block;

the block data index information reading module is used for sequentially reading the block data index information of at least part of the data blocks in the plurality of data blocks according to the corresponding relation;

a to-be-read data block determining module, configured to determine, according to the block data index information of the at least part of data blocks, one or more data blocks that meet a preset requirement among the multiple data blocks;

and the content analysis module is used for sequentially analyzing and reading the storage contents in the one or more data blocks according to the serial numbers of the one or more data blocks.
The apparatus of claim 16, wherein the block data index information comprises at least one of time information or topic type information of a plurality of data packets in each data block, and wherein the preset requirement comprises at least one of a requirement regarding the time information or a requirement regarding the topic type information.
The apparatus of claim 17, wherein the topic type is used to identify a source or content of each of the plurality of data packets.
A data writing apparatus comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform operations of any one of claims 1 to 5.
A data consistency checking apparatus comprising a memory and a processor, the memory storing computer program instructions, the processor executing the computer program instructions to perform the operations of any of claim 6.
A data reading apparatus comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform the operations of any one of claims 7 to 9.
A computer storage medium comprising computer instructions which, when executed by a processor, cause the data writing apparatus to perform the method of any one of claims 1 to 5.
A computer storage medium comprising computer instructions that, when executed by a processor, cause the data consistency checking apparatus to perform the method of claim 6.
A computer storage medium comprising computer instructions which, when executed by a processor, cause the data reading apparatus to perform the method of any one of claims 7 to 9.
A computer program product, characterized in that it causes the data writing means to execute the method according to any of claims 1-5, when the computer program product is run on a processor.
A computer program product, characterized in that it causes the data consistency checking means to carry out the method as claimed in claim 6, when the computer program product is run on a processor.
A computer program product, characterized in that it causes the data reading apparatus to carry out the method according to any one of claims 7 to 9, when the computer program product is run on a processor.