CN114556283B - Method and device for data writing, consistency checking and reading - Google Patents

Method and device for data writing, consistency checking and reading Download PDF

Info

Publication number
CN114556283B
CN114556283B CN201980101436.0A CN201980101436A CN114556283B CN 114556283 B CN114556283 B CN 114556283B CN 201980101436 A CN201980101436 A CN 201980101436A CN 114556283 B CN114556283 B CN 114556283B
Authority
CN
China
Prior art keywords
data
block
blocks
index information
data blocks
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201980101436.0A
Other languages
Chinese (zh)
Other versions
CN114556283A (en
Inventor
张丰伟
陈鹏
周玉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of CN114556283A publication Critical patent/CN114556283A/en
Application granted granted Critical
Publication of CN114556283B publication Critical patent/CN114556283B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing For Digital Recording And Reproducing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A method and device for writing, checking and reading data, by writing a plurality of data blocks reorganized based on time stamps into a storage medium through a plurality of channels in parallel in a sequential circulation order, on one hand, compared with the page-by-page writing mode in the prior art, the method and device can enable the plurality of blocks in a memory to be aged uniformly, the service life of the memory is prolonged, on the other hand, parallelization can be realized to a greater extent, and the stability of data writing is improved; in addition, by using the time stamp and the block data index information, the data reading speed and stability can be improved, the use efficiency of the solid state disk can be improved, and the application requirements of an automatic driving scene can be met.

Description

Method and device for data writing, consistency checking and reading
Technical Field
The present disclosure relates to storage technologies, and in particular, to a method and apparatus for writing, checking and reading data.
Background
During the running of the vehicle, it is highly necessary to record various running-related data because: on one hand, the recorded data can be used for positioning the problem, and can be input into an algorithm or a service module for simulation verification and model training, so that the development efficiency and the test efficiency of automatic driving are greatly improved; on the other hand, in the case of an accident in the commercial operation of automatic driving, various vehicle-mounted data before and after the accident are required as a part of an accident evidence chain to perform accident qualification.
A mainstream data storage technology in the prior art is a Solid State Disk (Solid State Disk or Solid State Drive, abbreviated as SSD) storage technology. The SSD memory is mainly composed of a controller, a firmware algorithm and flash memory particles: the controller is a small processor in nature, is a control center of the whole SSD memory, and is used for executing various instructions and algorithms for managing scheduling, reading and writing, deleting data and the like; the firmware algorithm is a program for driving the controller, and the controller uses the control program in the firmware algorithm to execute operations such as automatic signal processing, wear-leveling, error checking and correcting, bad block management, garbage collection, encryption and the like; flash memory grain is a storage medium in SSD memory, and the controller operates flash memory grain for multiple blocks (blocks) in parallel through several channels (channels), one block comprising 64 or 128 pages (pages), and the storage capacity of one page is typically 4KB. SSD memory has the characteristics of "read-write by page" and "erase by block", specifically, SSD memory has pages as basic access units, but blocks must be the smallest unit of erase when erasing data.
Fig. 1 to 3 illustrate write operations of an SSD memory having 8 channels in the prior art: in fig. 1 to 3, only one block corresponding to each channel is shown, and actually, each channel corresponds to a plurality of blocks; the SSD controller controls 8 channels to be sequentially circulated (sequential circulation means that data is sequentially allocated in order of arrival of channel numbers from small, and the channel with the largest number is allocated and then is continuously allocated from the channel with the smallest number), and controls 8 channels to perform writing operation in parallel; when 4KB data is to be written initially, only one page of memory space is occupied, in FIG. 1, channel 0 is taken as a starting channel (other channels can be taken as starting channels), and 4KB data is written into one page in a block corresponding to the channel 0 (see light gray squares in FIGS. 1-3); if 16KB data need to be written continuously, and the storage space of four pages need to be occupied again, 4KB data are written into one page in a block corresponding to each channel by using channels 1 to 4 in parallel (see dark gray squares in figures 2 to 3); if 20KB of data needs to be written continuously, and the storage space of five pages needs to be occupied, 4KB of data is written into one page in a block corresponding to each channel by using the channel 5, the channel 6, the channel 7, the channel 0 and the channel 1 in parallel (see black blocks in fig. 2-3).
In the automatic driving process, various equipment data such as a plurality of cameras, laser radars, ultrasonic sensors and the like need to be recorded, complex algorithm and business module data need to be recorded, the data volume is very large, the requirements of data from different sources on the read-write speed of a memory are also greatly different, the number of files involved in data reading is large, and the requirements on a file management system are very high. The prior art has low data read-write efficiency for an automatic driving scene and poor stability, and cannot meet the requirements of an automatic driving system, so that the storage method of the prior art needs to be optimized.
Disclosure of Invention
In order to overcome the above problems in the prior art, the embodiments of the present invention provide a method and apparatus for writing, checking and reading data.
In a first aspect, a data writing method is provided, including: marking each data packet in the N1 data packets with a time stamp, wherein N1 is a positive integer; sequencing the N1 data packets according to the time sequence of the time stamps of the N1 data packets; sequentially organizing the sequenced N1 data packets into a plurality of data pages based on the storage granularity of the pages, wherein each data page in the plurality of data pages comprises N3 data packets, and N3 is a positive integer; sequentially organizing the storage granularity of the plurality of data pages based on blocks into a plurality of data blocks, wherein each data block in the plurality of data blocks comprises block data index information and N2 data packets, N2 is a positive integer, N1 is more than or equal to N2 is more than or equal to N3, the block data index information comprises the serial numbers of the data blocks, and a corresponding relation of sequential circulation exists between the serial numbers of the plurality of data blocks and a plurality of channels for writing the plurality of data blocks into a storage medium; and respectively writing at least one data block corresponding to each channel in the plurality of channels into the storage medium through each channel according to the serial number of the at least one data block. Further, the time stamp is used to mark time information of the data packet, such as a start point of acquisition time of data in the data packet, an end point of acquisition time of data in the data packet, or a time point when the data packet is transmitted to a memory input interface. Further, the storage granularity of the pages and the storage granularity of the blocks are inherent properties of memory, for example, typically one page has a storage granularity of 4KB or 8KB, and one block has a storage granularity of 64 or 128 pages. Further, the block data index information exists in at least one data page in each data block, for example, in each data page in each data block, or only in the first data page in each data block. Further, writing the plurality of block data through the plurality of channels is a parallel operation. Further, serial read and write operations are performed on multiple pages of data within each data block through each channel. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the serial numbers of the data blocks from small to small, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the serial numbers of the data blocks continuing to be allocated according to the previous alternating sequence from the first allocated channel. The reorganized multiple data blocks are written into the storage medium through multiple channels in parallel according to the sequential circulation order, so that on one hand, the multiple blocks in the memory can be uniformly aged, the service life of the memory is prolonged, on the other hand, parallelization can be realized to a greater extent, and the stability of data writing is improved; in addition, by using the time stamp and the block data index information, data reading speed and stability can be improved.
In a first possible implementation manner of the data writing method according to the first aspect, the block data index information further includes the N 2 Time information of individual data packets. The time information includes, but is not limited to, the N 2 The start time, end time or duration of the data within the data packet. By marking time information of data stored in a data block within the data block, block-level time management of data content within a memory is facilitated, e.g., control read and parse specificationData content of the inter-range.
In a second possible implementation manner of the data writing method according to the first aspect or the first possible implementation manner of the first aspect, the each data page further includes page data index information, the page data index information includes the N 3 Time information of individual data packets. The time information includes, but is not limited to, the N 3 The start time, end time or duration of the data within the data packet. By marking the time information of the data stored in the data page, the time management of the page level is convenient for the data content in the memory, such as controlling the reading and analyzing the data content of the appointed time range.
In a third possible implementation manner of the data writing method according to the first aspect or any implementation manner of the first aspect, each data packet corresponds to a topic type, and the block data index information further includes the N 2 Theme type information of the individual packets. By marking the topic type information of the data stored in the data block, the topic type management of the data content in the memory is facilitated, for example, the reading and analysis of the data content with the specified topic type are controlled.
In a fourth possible implementation form of the data writing method according to the first aspect as such or any of the implementation forms of the first aspect, the topic type is used to identify a source or content of each data packet. Such as sensors that collect data or modules that process data. Such as audio data, video data or image data.
In a second aspect, a data consistency check method is provided, including: checking whether a tail data page of a tail data block is free of data or not according to tail data blocks written last through each of a plurality of channels, wherein the channels are used for writing the data blocks into a storage medium, each data block in the data blocks comprises a plurality of data pages, at least one data page in the data pages comprises block data index information, the block data index information comprises a serial number of the data block, and a corresponding relation of sequential circulation exists between the serial numbers of the data blocks and the channels; when there is no data in the tail data page of at least one tail data block, acquiring the minimum number S1 of the at least one tail data block; checking whether the numbers of the plurality of data blocks written through the plurality of channels are continuous or not according to the numbers of the plurality of tail data blocks written through the plurality of channels; when at least one discontinuity exists in the numbers of the plurality of data blocks, determining a start number S2 of the at least one discontinuity; determining a smaller value between S1 and S2; and erasing the data blocks with the numbers greater than or equal to the smaller value in the storage medium. Further, the erasing refers to physically deleting data from the storage medium. Further, when there is no tail data block of the tail data page without data, the smaller value between S1 and S2 is S2; when the numbers of the plurality of data blocks are all continuous, the smaller value between S1 and S2 is S1. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the serial numbers of the data blocks from small to small, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the serial numbers of the data blocks continuing to be allocated according to the previous alternating sequence from the first allocated channel. By performing the data consistency check operation described above within a data block, a smaller unit of memory data consistency check can be achieved relative to the prior art, thereby further optimizing the performance of the memory.
In a third aspect, there is provided a data reading method, comprising: determining a correspondence between a plurality of channels and numbers of a plurality of data blocks written into a storage medium through the plurality of channels by scanning block data index information of the data block written last through each of the plurality of channels, each of the plurality of data blocks including the block data index information, the block data index information including the numbers of the data blocks; according to the corresponding relation, sequentially reading block data index information of at least part of the data blocks in the plurality of data blocks; determining one or more data blocks meeting preset requirements in the plurality of data blocks according to the block data index information of at least part of the data blocks; and sequentially analyzing and reading the storage content in the one or more data blocks according to the numbers of the one or more data blocks. Wherein the at least part comprises part or all of the components. The one or more data blocks meeting the preset requirements can be determined only by reading the block data index information of at least part of the data blocks, so that the reading of all data in the whole data block is avoided, the reading speed and the reading stability can be improved, the data can be selectively read through the block data index information, the complexity of file management is reduced, and the method is more suitable for the application scene of automatic driving.
According to a third aspect, in a first possible implementation manner of the data reading method, the block data index information includes at least one of time information or topic type information of a plurality of data packets in each data block, and the preset requirement includes at least one of a requirement about the time information or a requirement about the topic type information. The time information includes, but is not limited to, a start time, an end time, or a duration of data within the plurality of data packets. By marking the time information of the data stored in the data block, the time management of the block level is facilitated for the data content in the memory, for example, the reading and analysis of the data content of the specified time range are controlled. By marking the topic type information of the data stored in the data block, the topic type management of the data content in the memory is facilitated, for example, the reading and analysis of the data content with the specified topic type are controlled.
In a second possible implementation manner of the data reading method according to the third aspect or the first possible implementation manner of the third aspect, the theme type is used to identify a source or content of each data packet of the plurality of data packets. Such as sensors that collect data or modules that process data. Such as audio data, video data or image data.
In a fourth aspect, there is provided a data writing apparatus comprising: a time stamp marking module for N 1 Each of the individual packets is time stamped, where N 1 Is a positive integer; a sorting module for sorting the N 1 The data packets are according to the N 1 Time sequence ordering of time stamps of the data packets; a data page organization module for organizing the N ordered 1 The data packets are orderly organized into a plurality of data pages based on the storage granularity of the pages, and each data page in the plurality of data pages comprises N 3 Data packets, where N 3 Is a positive integer; a data block organization module for sequentially organizing the plurality of data pages into a plurality of data blocks based on a block-based storage granularity, wherein each data block of the plurality of data blocks comprises block data index information and N 2 Data packets, where N 2 Is a positive integer and N 1 ≥N 2 ≥N 3 The block data index information comprises the numbers of the data blocks, and a correspondence relationship exists between the numbers of the plurality of data blocks and a plurality of channels for writing the plurality of data blocks into a storage medium in a sequential cycle; and the channel writing module is used for respectively writing at least one data block corresponding to each channel in the plurality of channels into the storage medium through each channel according to the serial number of the at least one data block. Further, the time stamp is used to mark time information of the data packet, such as a start point of acquisition time of data in the data packet, an end point of acquisition time of data in the data packet, or a time point when the data packet is transmitted to a memory input interface. Further, the storage granularity of the pages and the storage granularity of the blocks are inherent properties of memory, for example, typically one page has a storage granularity of 4KB or 8KB, and one block has a storage granularity of 64 or 128 pages. Further, the block data index information exists in at least one data page in each data block The face, for example, is present in each data page in each data block, or only in the first data page in each data block. Further, writing the plurality of block data through the plurality of channels is a parallel operation. Further, serial read and write operations are performed on multiple pages of data within each data block through each channel. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the serial numbers of the data blocks from small to small, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the serial numbers of the data blocks continuing to be allocated according to the previous alternating sequence from the first allocated channel. The reorganized multiple data blocks are written into the storage medium through multiple channels in parallel according to the sequential circulation order, so that on one hand, the multiple blocks in the memory can be uniformly aged, the service life of the memory is prolonged, on the other hand, parallelization can be realized to a greater extent, and the stability of data writing is improved; in addition, by using the time stamp and the block data index information, data reading speed and stability can be improved.
In a first possible implementation manner of the data writing device according to the fourth aspect, the block data index information further includes the N 2 Time information of individual data packets. The time information includes, but is not limited to, the N 2 The start time, end time or duration of the data within the data packet. By marking the time information of the data stored in the data block, the time management of the block level is facilitated for the data content in the memory, for example, the reading and analysis of the data content of the specified time range are controlled.
In a second possible implementation manner of the data writing device according to the fourth aspect or the first possible implementation manner of the fourth aspect, the each data page further includes page data index information, the page data index information includes the N 3 Time information of individual data packets. The time information includes, but is not limited to, the N 3 Start time, end time or duration of data in individual packetsTime. By marking the time information of the data stored in the data page, the time management of the page level is convenient for the data content in the memory, such as controlling the reading and analyzing the data content of the appointed time range.
In a third possible implementation manner of the data writing device according to the fourth aspect or any implementation manner of the fourth aspect, each data packet corresponds to a topic type, and the block data index information further includes the N 2 Theme type information of the individual packets. By marking the topic type information of the data stored in the data block, the topic type management of the data content in the memory is facilitated, for example, the reading and analysis of the data content with the specified topic type are controlled.
In a fourth possible implementation form of the data writing apparatus according to the fourth aspect as such or any one of the implementation forms of the fourth aspect, the theme type is used to identify a source or content of each data packet. Such as sensors that collect data or modules that process data. Such as audio data, video data or image data.
In a fifth aspect, there is provided a data consistency check apparatus including: a first checking module, configured to check whether a tail data page of a tail data block is free of data for a tail data block written last through each of a plurality of channels, where the plurality of channels are used to write a plurality of data blocks into a storage medium, each of the plurality of data blocks includes a plurality of data pages, at least one of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and a correspondence of sequential cycles exists between the number of the plurality of data blocks and the plurality of channels; the first number acquisition module is used for acquiring the minimum number S1 of at least one tail data block when the tail data page of the at least one tail data block has no data; a second checking module, configured to check whether numbers of the plurality of data blocks written through the plurality of channels are consecutive according to numbers of the plurality of tail data blocks written through the plurality of channels; a second number acquisition module, configured to determine a start number S2 of at least one discontinuity when the numbers of the plurality of data blocks have the at least one discontinuity; a comparison module for determining a smaller value between S1 and S2; and the data block erasing module is used for erasing the data blocks with the numbers larger than or equal to the smaller value in the storage medium. Further, the erasing refers to physically deleting data from the storage medium. Further, when there is no tail data block of the tail data page without data, the smaller value between S1 and S2 is S2; when the numbers of the plurality of data blocks are all continuous, the smaller value between S1 and S2 is S1. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the serial numbers of the data blocks from small to small, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the serial numbers of the data blocks continuing to be allocated according to the previous alternating sequence from the first allocated channel. By performing the data consistency check operation described above within a data block, a smaller unit of memory data consistency check can be achieved relative to the prior art, thereby further optimizing the performance of the memory.
In a sixth aspect, there is provided a data reading apparatus comprising: the corresponding relation determining module is used for determining the corresponding relation between the plurality of channels and the serial numbers of the plurality of data blocks by scanning the block data index information of the data block which is finally written in through each of the plurality of channels, wherein the plurality of data blocks are written in the storage medium through the plurality of channels, each data block in the plurality of data blocks comprises the block data index information, and the block data index information comprises the serial numbers of the data blocks; the block data index information reading module is used for sequentially reading the block data index information of at least part of the data blocks in the plurality of data blocks according to the corresponding relation; the data block to be read determining module is used for determining one or more data blocks meeting preset requirements in the plurality of data blocks according to the block data index information of the at least part of data blocks; and the content analysis module is used for sequentially analyzing and reading the storage content in the one or more data blocks according to the numbers of the one or more data blocks. Wherein the at least part comprises part or all of the components. The one or more data blocks meeting the preset requirements can be determined only by reading the block data index information of at least part of the data blocks, so that the reading of all data in the whole data block is avoided, the reading speed and the reading stability can be improved, the data can be selectively read through the block data index information, the complexity of file management is reduced, and the method is more suitable for the application scene of automatic driving.
According to a sixth aspect, in a first possible implementation manner of the data reading apparatus, the block data index information includes at least one of time information or topic type information of a plurality of data packets in each data block, and the preset requirement includes at least one of a requirement about the time information or a requirement about the topic type information. The time information includes, but is not limited to, a start time, an end time, or a duration of data within the plurality of data packets. By marking the time information of the data stored in the data block, the time management of the block level is facilitated for the data content in the memory, for example, the reading and analysis of the data content of the specified time range are controlled. By marking the topic type information of the data stored in the data block, the topic type management of the data content in the memory is facilitated, for example, the reading and analysis of the data content with the specified topic type are controlled.
In a second possible implementation manner of the data reading device according to the sixth aspect or the first possible implementation manner of the sixth aspect, the theme type is used to identify a source or content of each data packet of the plurality of data packets. Such as sensors that collect data or modules that process data. Such as audio data, video data or image data.
In a seventh aspect, a data writing apparatus is provided, comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform the operations of the first aspect or any implementation of the first aspect above.
In an eighth aspect, there is provided a data consistency check apparatus comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform the operations of the second aspect.
In a ninth aspect, there is provided a data reading apparatus comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform operations according to the third aspect or any implementation of the third aspect above.
In a tenth aspect, there is provided a computer storage medium comprising computer instructions which, when executed by a processor, cause the data writing apparatus to perform the method according to the first aspect or any implementation of the first aspect above.
In an eleventh aspect, there is provided a computer storage medium comprising computer instructions which, when executed by a processor, cause the data consistency check apparatus to perform the method according to the second aspect.
In a twelfth aspect, there is provided a computer storage medium comprising computer instructions which, when executed by a processor, cause the data reading apparatus to perform the method of the third aspect or any implementation of the third aspect above.
In a thirteenth aspect, a computer program product is provided which, when run on a processor, causes the data writing device to perform the method according to the first aspect or any implementation of the first aspect above.
In a fourteenth aspect, there is provided a computer program product for, when run on a processor, causing the data consistency check apparatus to perform the method as described in the second aspect.
In a fifteenth aspect, there is provided a computer program product which, when run on a processor, causes the data reading apparatus to perform the method according to the third aspect or any one of the implementations of the third aspect above.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of writing 4KB of data in SSD memory in the prior art.
FIG. 2 is a diagram of writing 4KB data in an SSD memory and then writing 16KB data in the prior art.
FIG. 3 is a diagram of writing 20KB of data in an SSD memory and then writing 20KB of data in the prior art.
Fig. 4 is a hardware configuration diagram of a first exemplary application scenario of the present invention.
Fig. 5 is a hardware configuration diagram of a second exemplary application scenario of the present invention.
Fig. 6 is a hardware configuration diagram of a third exemplary application scenario of the present invention.
Fig. 7 is a flowchart of a data writing method according to an embodiment of the invention.
Fig. 8 is a schematic diagram of a data writing method according to a second embodiment of the invention.
Fig. 9 is a schematic diagram of a block data index information composition in a data writing method according to a second embodiment of the present invention.
Fig. 10 is a flowchart of a data consistency check method according to a third embodiment of the present invention.
Fig. 11 is a schematic diagram of a data consistency check method according to a fourth embodiment of the present invention.
Fig. 12 is a flowchart of a data reading method according to a fifth embodiment of the present invention.
Fig. 13 is a schematic diagram of a data reading method according to a sixth embodiment of the invention.
Fig. 14 is a schematic diagram of a data reading method according to a seventh embodiment of the invention.
Fig. 15 is a block diagram showing a configuration of a data writing apparatus according to an embodiment of the present invention.
Fig. 16 is a block diagram showing a structure of a data consistency check apparatus according to a ninth embodiment of the present invention.
Fig. 17 is a block diagram showing a structure of a data reading apparatus according to a tenth embodiment of the present invention.
Fig. 18 is a block diagram showing the structure of an apparatus according to an eleventh embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 4-6 are three typical application scenarios of the present invention when applied to an intelligent vehicle to implement driving data recording. It should be noted that, the application scenario of the present invention is not limited to driving data recording, and includes all application scenarios involving data writing, consistency checking or reading. The MDC (Multi Domain controller) in the intelligent vehicle can access signals of different sensors, analyze and process the signals, finally send out control commands, and the recording of driving data is realized based on the MDC in fig. 4-6. It should be noted that, in practical application, the recording of driving data may be realized based on other components with control functions in the vehicle besides MDC.
In fig. 4, a separate driving data recording subsystem 403 is provided in the MDC401 and dedicated for driving data recording, and the driving data recording subsystem 403 may be packaged separately as an aircraft black box to improve the security of data. The original main control CPU402 in the MDC401 receives the driving data collected by the plurality of sensors (including the millimeter wave radar 406, the ultrasonic radar 407, the laser radar 408, the GPS module 409 and the camera 410), and sends the driving data to the driving data recording subsystem 403, and finally the data is stored in the storage medium 405 of the driving data recording subsystem 403. The storage medium 405 is preferably implemented using flash memory particles (or referred to as solid state storage particles). The driving data recording subsystem 403 is further configured with a data processing chip 404, where the data processing chip 404 may include, but is not limited to, a CPU (central processing unit ), FPGA (Field Programmable Gate Array, field programmable gate array), ASIC (Application Specific Integrated Circuit ), DSP (Digital Signal Processor, digital signal processor) or MCU (Microcontroller Unit, micro control unit), etc. for implementing a data processing or control function, and the data processing chip 404 may further include a storage device for implementing storage of part of data, control software or driving firmware.
The internal structure of the MDC501 in fig. 5 is the same as the MDC401 in fig. 4, except that the main control CPU502 in fig. 5 does not participate in the recording of the driving data, and the driving data is received by the driving data recording subsystem 503 directly from a plurality of sensors (including the millimeter wave radar 506, the ultrasonic radar 507, the laser radar 508, the GPS module 509, and the camera 510).
Unlike fig. 4 and fig. 5, an independent driving data recording subsystem is provided in the MDC for driving recording, in fig. 6, the MDC601 is externally connected with an external memory 603, and the external memory 603 is preferably an SSD solid state disk. Similar to the internal structure of a typical solid state disk, the external memory 603 includes a host chip 604 and a flash memory chip 605. The main control CPU602 in the MDC601 receives the driving data collected by the plurality of sensors (including the millimeter wave radar 606, the ultrasonic radar 607, the laser radar 608, the GPS module 609 and the camera 610), and sends the driving data to the external memory 603, and finally the data is stored in the flash memory chip 605 of the external memory 603.
An embodiment of the present invention provides a data writing method, referring to fig. 1, including:
step 701, N 1 Each of the individual packets is time stamped, where N 1 Is a positive integer.
In this embodiment, the N is 1 Each of the data packets is from a data acquisition device, said N 1 The data packets are from a plurality of different data acquisition devices (such as cameras, laser radars, millimeter wave radars, ultrasonic sensors or man-machine interaction devices) and each have a frame structure, so that attribute information, particularly time information, of each data packet can be identified, an acquisition time starting point of data in each data packet is acquired, and a time stamp is marked for each data packet by the acquisition time starting point. Furthermore, the time stamp can also be used to mark other time information, such as the end of the acquisition time of the data in the data packet or the point in time when the data packet is transferred to the memory input interface.
Step 702, converting the N 1 The data packets are according to the N 1 The time stamps of the individual data packets are ordered in chronological order.
Due to the N 1 The data packets relate to different topic types, so the step is to uniformly sort the data packets of a plurality of topic types based on time information.
Step 703, sorting the N 1 The data packets are orderly organized into a plurality of data pages based on the storage granularity of the pages, and each data page in the plurality of data pages comprises N 3 Data packets, where N 3 Is a positive integer.
Each data page comprises page data index information for carrying the N 3 Time information and topic type information of each data packet, wherein the topic type information is used for identifying the N 3 The data acquisition equipment from which the data in the data packets come, wherein the time information is N 3 Initiation of recording of content integrity within individual packetsTime and end time, and start time and end time of recorded content for each topic type.
Step 704, organizing the plurality of data pages into a plurality of data blocks sequentially based on a block-based storage granularity, each data block of the plurality of data blocks including block data index information and N 2 Data packets, where N 2 Is a positive integer and N 1 ≥N 2 ≥N 3 The block data index information includes numbers of the data blocks, and there is a correspondence of sequential cycles between the numbers of the plurality of data blocks and a plurality of channels for writing the plurality of data blocks to a storage medium. Further, the block data index information may further include the N 2 Time information of individual data packets. Each data packet corresponds to a topic type, and the block data index information further includes the N 2 Theme type information of the individual packets. The topic type is used to identify the source or content of each data packet.
In this embodiment, the memory granularity of the page is 4KB, and one block includes 128 pages, so that the memory granularity of the block is 512KB. The block data index information exists only in the first data page of each data block. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the serial numbers of the data blocks from small to small, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the serial numbers of the data blocks continuing to be allocated according to the previous alternating sequence from the first allocated channel.
Step 705, writing at least one data block corresponding to each channel of the plurality of channels into the storage medium through each channel in turn according to the number of the at least one data block. The writing of the plurality of block data through the plurality of channels is parallel operation, and the writing of the plurality of data pages in each data block through each channel is serial operation.
The second embodiment of the present invention also provides a data writing method, referring to fig. 8, including:
Step 801, data packet sequence A { A } 0 ，A 1 ，……}、B{B 0 ，B 1 ，……}、C{C 0 ，C 1 ，……}、D{D 0 ，D 1 … … and E { E } 0 ，E 1 Each packet in … … is time stamped.
The laser radar, millimeter wave radar, camera, robot operating system and map manager are vehicle-mounted data acquisition devices that respectively package real-time acquired data into a data packet sequence A { A } 0 ，A 1 ，……}、B{B 0 ，B 1 ，……}、C{C 0 ，C 1 ，……}、D{D 0 ，D 1 … … and E { E } 0 ，E 1 … …, and the above-mentioned data packet sequences have different generation rates. The data packet is stored for facilitating future test analysis and problem location. Since each data packet contains data acquisition time information, each data packet may be time stamped t in step 801 according to the data acquisition time information i 。
Step 802, sorting the data packets in the data packet sequences A-E according to the time sequence of the time stamps to obtain a data packet sequence X { B } including data packets from a plurality of data acquisition devices 0 ，C 0 ，B 1 ，D 0 ，E 0 ，C 1 ，A 0 ，A 1 ，E 1 ，D 1 ，……}。
In step 803, the data packet sequence X is organized into a plurality of data pages in turn based on the storage granularity of the pages. In this embodiment, the storage granularity of the page is 4K, and after considering the storage space occupied by the header information of the data packet and the data page, each data page may bear N 2 Payload information of individual data packets.
In step 804, the plurality of data pages obtained in step 803 are sequentially organized into a plurality of data blocks based on the storage granularity of the blocks, and the first data page of each data block includes the block data index information. Fig. 9 illustrates an exemplary composition of the tile data index information, including the following fields:
the Page Header Type specifies whether the Header information is block data index information or Page data index information, and each data Page in implementation II also comprises Page data index information, so that the first data Page of each data block comprises both the block data index information and the Page data index information, and the two Header information are distinguished by using the field;
channel Num: number of channels;
channel ID: the channel number of the block/page;
topic Num: the number of Topic types involved in the recording of the present block/page;
topic0 Start offset, the first recording offset of the 0 th Topic type record in the block/page;
topic0 End offset, the last recording offset of the 0 th theme type record in the block/page;
start Time: the start time of all records of the block/page;
end Time: the end time of all records of the block/page;
Serial Num: the number of the data blocks is globally unique, the number is restarted from 0 after reaching the maximum value, and the number of the plurality of data blocks obtained in the step 504 is { Block-0, block-1, block-2, … …, block-m, … … }, and the number can also be started from other numbers according to the situation;
dlen: the effective data length (including header information) in one block/page.
In step 805, the data blocks { Block-0, block-1, block-2, … …, block-m, … … } are written to the storage medium in a channel parallel write manner by sequentially passing the channels corresponding to each data Block according to the numbers of the data blocks.
Wherein there is a correspondence between the numbers of the plurality of data blocks and the plurality of channels in a sequential cycle, the meaning of the sequential cycle is described below by taking step 805 in fig. 8 as an example:
in fig. 8, 4 channels are used to perform writing operations, namely, channel0, channel1, channel2, and Channel3, { Block-0, block-4, block-8, … … } are written to the storage medium through Channel0, { Block-1, block-5, block-9, … … } are written to the storage medium through Channel1, { Block-2, block-6, block-10, … … } are written to the storage medium through Channel2, { Block-3, block-7, block-11, … … } are written to the storage medium through Channel3, and Block-by-Block writing operations are performed in each Channel according to the number of the data Block, for example, the Block-1 writing operation through Channel0 is performed first after the Block-0 writing is completed.
In addition to the sequential loop, the 4 channels follow the provision of parallel writing, e.g. write Block-0 through Channel0, write Block-1 through Channel1, write Block-2 through Channel2 and write Block-3 through Channel3 are performed simultaneously, although there may be a difference in progress of the write operation between the channels, e.g. when Block-6 is written through Channel2, the write of Block-0 through Channel0 is not yet completed.
According to the data writing method disclosed by the embodiment I and the embodiment II of the invention, the blocks of the data reorganized based on the time stamp are written into the storage medium through the channels in parallel in the order of sequential circulation block by block, so that the blocks in the memory can be uniformly aged in a page-by-page writing mode compared with the prior art, the service life of the memory is prolonged, the parallelization can be realized to a greater degree, and the stability of data writing is improved; in addition, by using the time stamp and the block data index information, data reading speed and stability can be improved.
The third and fourth embodiments of the present invention provide a data consistency check method, respectively, which can perform consistency check on data written in a storage medium by the data writing method as described in the first or second embodiment of the present invention.
Referring to fig. 10, an embodiment of the present invention includes:
step 1001, checking whether a tail data page of the tail data block has no data for a tail data block written last through each of a plurality of channels, where the plurality of channels are used for writing a plurality of data blocks into a storage medium, each data block of the plurality of data blocks includes a plurality of data pages, at least one data page of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and a correspondence of sequential cycles exists between the number of the plurality of data blocks and the plurality of channels. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the numbers of the data blocks from small to small, and one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the data blocks with consecutive numbers are sequentially allocated according to the previous alternating sequence from the first channel allocated before, which can refer to step 505 in fig. 8. In the data writing method described in the first embodiment or the second embodiment of the present invention, one reason why the written data has consistency errors is that the single data block is not completely written, i.e. a situation that a partial data page is missed by the single data block occurs. Since the plurality of data blocks in each channel are written in block-by-block order according to the number of the data blocks, and the plurality of data pages are generally written in page-by-page order in each data block, it is only necessary to check whether the tail data page of the last written tail data block of each channel has no data (no data representation such as data full FF).
Step 1002, when there is no data in the tail data page of at least one tail data block, obtaining the minimum number S1 of the at least one tail data block. And when the tail data page of the tail data block does not exist, namely, the tail data blocks of all channels are full of data, marking the S1 as null value.
Step 1003, checking whether the numbers of the plurality of data blocks written through the plurality of channels are consecutive according to the numbers of the plurality of tail data blocks written through the plurality of channels. In the data writing method described in the first embodiment or the second embodiment of the present invention, another reason why the written data has consistency errors is that the writing progress of the multiple channels is not matched, for example, when the Block-6 is written through Channel2, the operation of writing the Block-0 through Channel0 is not completed yet, and because the number between the multiple data blocks written through the multiple channels reflects the time sequence of the time stamps of the multiple data blocks, if the Block-0 is not completely written, all the data blocks of the following number have no meaning even if the Block-0 is completely written.
In step 1004, the problem of discontinuity in the number may exist at one or more locations, and when there is at least one discontinuity in the number of the plurality of data blocks, determining a start number S2 of the at least one discontinuity. The start number S2 may also be understood as the minimum value of the number of the missing partial data block when there is at least one discontinuity in the number of the plurality of data blocks due to the missing partial data block. And when the numbers of the plurality of data blocks written through the plurality of channels are continuous, marking the S2 as a null value.
In step 1005, a smaller value between S1 and S2 is determined. It should be noted that, when there is no tail data block of the tail data page without data, the smaller value between S1 and S2 is S2; when the numbers of the plurality of data blocks are all continuous, the smaller value between S1 and S2 is S1.
At step 1006, data blocks with numbers greater than or equal to the smaller value are erased in the storage medium. It should be noted that the erasure refers to physically deleting data from the storage medium.
Referring to fig. 11, an embodiment of the present invention includes:
in step 1101, the memory includes four channels, namely Channel0, channel1, channel2 and Channel3, and tail data blocks of the four channels are Block-12, block-9, block-6 and Block-15, respectively, and it is checked whether tail data pages of Block-12, block-9, block-6 and Block-15 have no data (i.e. are full FF).
Step 1102, after checking in step 1101, finds that the tail data pages of Block-9 and Block-15 have no data, and determines S1 to be the smaller value of 9 and 15, i.e., the flag s1=9.
Step 1103, checking whether the numbers of the plurality of data blocks written by Channel0, channel1, channel2 and Channel3 are consecutive according to the numbers of the above blocks-12, 9, 6 and 15. The specific checking means is to divide the numbers of the blocks-12, the blocks-9, the blocks-6 and the blocks-15 by the number of channels 4, and the values of the four quotients are unequal, so that the number discontinuity of the plurality of data blocks written by the channels 0, 1, 2 and 3 is determined.
In step 1104, the maximum value of the four quotients is 3, the quotient of Channel1 divided by Channel number 4 and the quotient of Channel2 divided by Channel number 4 are all less than 3, so it can be determined that the data blocks causing the number discontinuity of the data blocks are Block-13 (corresponding to Channel 1), block-10 (corresponding to Channel 2) and Block-14 (corresponding to Channel 2), and thus the smaller value of S2 is 13, 10 and 14, namely, the sign s2=10.
Step 1105, determining S 1 And S is equal to 2 A smaller value of 9;
step 1106, erase Block-9, block-11, block-12 and Block-15 in the storage medium.
The data consistency check method according to the third and fourth embodiments of the present invention can further optimize the performance of the memory by performing the data consistency check operation described above in the data block, and thus realizing data consistency check with a smaller unit of memory than in the prior art.
The fifth embodiment of the present invention provides a data reading method, referring to fig. 12, including:
step 1201, determining a correspondence between numbers of a plurality of channels and a plurality of data blocks written into a storage medium through the plurality of channels by scanning block data index information of the data block written last through each of the plurality of channels, wherein each of the plurality of data blocks includes the block data index information, and the block data index information includes the number of the data block. The correspondence represents the channel to which each data block is allocated, and the first and last data blocks and the channels corresponding to the first and last data blocks may be determined in step 1201.
Step 1202, sequentially reading block data index information of at least some of the plurality of data blocks according to the correspondence. The at least part includes a part or all of the data blocks, and step 1202 is to read the block data index information of each data block in the part or all of the data blocks according to the serial number sequence of the data blocks.
Step 1203, determining one or more data blocks meeting a preset requirement from the plurality of data blocks according to the block data index information of the at least part of data blocks. The tile data index information includes time information of a plurality of data packets within each data tile, including but not limited to a start time, an end time, or a duration of data within the plurality of data packets, and a topic type for identifying a source of each data packet of the plurality of data packets, such as a sensor that collects data or a module that processes data, and content, such as audio data, video data, or image data. By marking the time information of the data stored in the data block, the time management of the block level is convenient for the data content in the memory, such as controlling the reading and analyzing the data content of the appointed time range; by marking the topic type information of the data stored in the data block, the topic type management of the data content in the memory is facilitated, for example, the reading and analysis of the data content with the specified topic type are controlled.
Step 1204, sequentially parsing and reading the storage contents in the one or more data blocks according to the numbers of the one or more data blocks. The parsing includes, but is not limited to, operations of unpacking the data block, acquiring payload data in the data block, decoding or decompressing, etc., and the reading includes, but is not limited to, operations of audio-video playing, image presenting, text presenting, or transferring to other digital signal processing devices, etc.
The sixth embodiment of the present invention provides a data reading method, which is used for reading data from a millimeter wave radar stored in a memory, and specifically includes the steps of:
in step 1301, the corresponding relationship between the four channels and the data Block numbers of { Block-0, block-1, … …, block-q } is determined by scanning the Block data index information of the last written data Block (Block-m, block-n, block-p, and Block-q, respectively) of each of the four channels of Channel0, channel1, channel2, and Channel3. Each data Block of { Block-0, block-1, … …, block-q } includes Block data index information including the number of the data Block, { Block-0, block-1, … …, block-q } is written to the storage medium through Channel0, channel1, channel2, and Channel3, which have a sequentially cyclic correspondence, and are represented as { Block-0, block-4, block-8, … …, block-m } corresponding to Channel0, { Block-1, block-5, block-9, … …, block-n } corresponding to Channel1, { Block-2, block-6, block-10, … …, block-p } corresponding to Channel2, { Block-3, block-7, block-23, block-q } corresponding to Channel3.
In step 1302, the Block data index information of each data Block { Block-0, block-1, … …, block-q } is sequentially read from Block-0 or from Block-q in reverse order according to the number of the data Block.
In step 1303, since the Block data index information includes the acquisition source of the data record in each data Block, it can be determined that { Block-0, block-1, … …, block-q } includes the data Block acquired from the millimeter wave radar, i.e., the data Block { Block-2, block-4, block-5, block-7, block-9, … …, block-m, block-n, block-q }.
Step 1304, sequentially analyzing and reading the memory contents in the data blocks { Block-2, block-4, block-5, block-7, block-9, … …, block-m, block-n, block-q }, and recovering the data packet sequence { B } acquired by the millimeter wave radar 0 ，B 1 … ….
The seventh embodiment of the invention provides a data reading method for reading the number stored in the memoryAccording to the acquisition time at T 1 ～T 2 Data within the range, see fig. 14 for specific steps, including:
in step 1401, the corresponding relationship between the four channels and the data Block numbers of { Block-0, block-1, … …, block-q } is determined by scanning the Block data index information of the last written data Block (Block-m, block-n, block-p, and Block-q, respectively) of each of the four channels of Channel0, channel1, channel2, and Channel3. Each data Block of { Block-0, block-1, … …, block-q } includes Block data index information including the number of the data Block, { Block-0, block-1, … …, block-q } is written to the storage medium through Channel0, channel1, channel2, and Channel3, which have a sequentially cyclic correspondence, and are represented as { Block-0, block-4, block-8, … …, block-m } corresponding to Channel0, { Block-1, block-5, block-9, … …, block-n } corresponding to Channel1, { Block-2, block-6, block-10, … …, block-p } corresponding to Channel2, { Block-3, block-7, block-23, block-q } corresponding to Channel3.
Step 1402, starting from Block-0 and reading the Block data index information of each data Block { Block-0, block-1, … …, block-q } according to the number sequence of the data blocks, wherein the Block data index information contains the acquisition time of the data record in each data Block, and when the Block data index information of Block-9 is read, finding that the latest acquisition time of the data record in the data Block is later than T 2 Because the serial numbers of the data blocks can reflect the acquisition time sequence of the data records in the blocks, the acquisition time of the data records in the data blocks with serial numbers larger than 9 is inevitably later than T 2 Therefore, the Block data index information of the data Block after the Block-9 is read, i.e., the Block data index information of the partial data Block in the { Block-0, block-1, … …, block-q } portion can be discarded.
Step 1403, determining that the acquisition time included in { Block-5, block-6, block-7, block-8, block-9} is located at T by reading the Block data index information 1 ～T 2 Data within the range.
Step 1404, sequentially analyzing and reading the memory contents in the data blocks { Block-5, block-6, block-7, block-8, block-9}, and recovering the acquisition time to be at T 1 ～T 2 Data content within the range.
In the fifth, sixth and seventh embodiments of the present invention, since one or more data blocks satisfying the preset requirement can be determined only by reading the block data index information of at least some data blocks, reading of all data in the whole data block is avoided, so that the reading speed and stability can be improved, and the data can be selectively read through the block data index information, the complexity of file management is reduced, and the present invention is more suitable for the application scenario of automatic driving.
An eighth embodiment of the present invention provides a data writing apparatus, referring to fig. 15, including:
a timestamp marking module 1501 for N 1 Each of the individual packets is time stamped, where N 1 Is a positive integer. The time stamp is used for marking time information of the data packet, such as a collection time starting point of data in the data packet, a collection time ending point of data in the data packet or a time point of transmission of the data packet to a memory input interface.
A sorting module 1502 for sorting the N 1 The data packets are according to the N 1 The time stamps of the individual data packets are ordered in chronological order.
A data page organizing module 1503 for organizing the N ordered 1 The data packets are orderly organized into a plurality of data pages based on the storage granularity of the pages, and each data page in the plurality of data pages comprises N 3 Data packets, where N 3 Is a positive integer.
The data block organizing module 1504 is configured to sequentially organize the plurality of data pages into a plurality of data blocks based on a storage granularity of the blocks.
The memory granularity of the pages and the memory granularity of the blocks are inherent properties of memory, for example, typically one page has a memory granularity of 4KB or 8KB, and one block has a memory granularity of 64 or 128 pages.
Each data block of the plurality of data blocks includes block data index information and N 2 Data packets, where N 2 Is a positive integer and N 1 ≥N 2 ≥N 3 。
The block data index information includes a number of the data block and is present in each data page in each data block. Each data Page further includes Page data index information, and whether the Header information is block data index information or Page data index information is distinguished by a "Page Header Type" field in fig. 9.
There is a correspondence of sequential cycles between the numbers of the plurality of data blocks and a plurality of channels for writing the plurality of data blocks to the storage medium. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the serial numbers of the data blocks from small to small, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the serial numbers of the data blocks continuing to be allocated according to the previous alternating sequence from the first allocated channel.
The block data index information further includes the N 2 Time information of individual data packets. The time information includes, but is not limited to, the N 2 The start time, end time or duration of the data within the data packet. By marking the time information of the data stored in the data block, the time management of the block level is facilitated for the data content in the memory, for example, the reading and analysis of the data content of the specified time range are controlled.
Each data packet corresponds to a topic type, and the block data index information further includes the N 2 Theme type information of the individual packets. The topic type is used to identify the source or content of each data packet. Such as sensors that collect data or modules that process data. Such as audio data, video data or image data. Facilitating access to memory by marking in a data block subject type information of data stored in the data blockThe data content in the content management system manages the topic type, such as controlling the reading and analyzing of the data content of the specified topic type.
The channel writing module 1505 is configured to write at least one data block corresponding to each channel of the plurality of channels into the storage medium through each channel in sequence according to the number of the at least one data block. And performing serial writing operation on a plurality of data pages in each data block through each channel, and writing a plurality of block data into the plurality of channels through the plurality of channels to perform parallel operation.
According to the data writing device, the blocks of the data reorganized based on the time stamps are written into the storage medium through the channels in parallel in the order of sequential circulation block by block, so that the blocks in the memory can be uniformly aged in a page-by-page writing mode compared with the prior art, the service life of the memory is prolonged, parallelization can be realized to a greater degree, and the stability of data writing is improved; in addition, by using the time stamp and the block data index information, data reading speed and stability can be improved.
An embodiment of the present invention provides a data consistency checking apparatus, referring to fig. 16, including:
a first checking module 1601, configured to check, for each tail data block written last through each of a plurality of channels, whether a tail data page of the tail data block has no data, where the plurality of channels are used to write a plurality of data blocks into a storage medium, each data block of the plurality of data blocks includes a plurality of data pages, at least one data page of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and a correspondence exists between the number of the plurality of data blocks and the plurality of channels in a sequential cycle. The sequential cycle is specifically that a plurality of channels are alternately allocated with data blocks according to the sequence of the serial numbers of the data blocks from small to small, one data channel is allocated with one data block at a time, and after the last channel is allocated with one data block, the serial numbers of the data blocks continuing to be allocated according to the previous alternating sequence from the first allocated channel.
The first number obtaining module 1602 is configured to obtain a minimum number S1 of at least one tail data block when there is no data in a tail data page of the at least one tail data block.
The second checking module 1603 is configured to check whether the numbers of the plurality of data blocks written through the plurality of channels are consecutive according to the numbers of the plurality of tail data blocks written through the plurality of channels.
The second number obtaining module 1604 is configured to determine a start number S2 of at least one discontinuity when the numbers of the plurality of data blocks have the at least one discontinuity.
A comparison module 1605 for determining a smaller value between S1 and S2. When the tail data block of the tail data page without data does not exist, the smaller value between the S1 and the S2 is S2; when the numbers of the plurality of data blocks are all continuous, the smaller value between S1 and S2 is S1.
A data block erase module 1606 for erasing data blocks with numbers greater than or equal to the smaller value in the storage medium. The erasing refers to physically deleting data from the storage medium.
By executing the data consistency check operation in the data block, the data consistency check of a smaller storage unit can be realized compared with the prior art, thereby further optimizing the performance of the memory.
An embodiment of the present invention provides a data reading apparatus, referring to fig. 17, including:
the correspondence determining module 1701 is configured to determine a correspondence between numbers of a plurality of channels and a plurality of data blocks written to a storage medium through the plurality of channels by scanning block data index information of a data block written last through each of the plurality of channels, where each of the plurality of data blocks includes the block data index information, and the block data index information includes the number of the data block.
The block data index information reading module 1702 is configured to sequentially read the block data index information of at least some of the plurality of data blocks according to the correspondence. Wherein the at least part comprises part or all of the components.
And the to-be-read data block determining module 1703 is configured to determine one or more data blocks meeting a preset requirement from the plurality of data blocks according to the block data index information of the at least part of the data blocks.
The block data index information includes time information or topic type information of a plurality of data packets in each data block, and the preset requirement includes a requirement about the time information or a requirement about the topic type information. The time information includes, but is not limited to, a start time, an end time, or a duration of data within the plurality of data packets. The topic type is used to identify the source or content of each of the plurality of data packets. Such as sensors that collect data or modules that process data. Such as audio data, video data or image data. By marking the time information of the data stored in the data block, the time management of the block level is facilitated for the data content in the memory, for example, the reading and analysis of the data content of the specified time range are controlled. By marking the topic type information of the data stored in the data block, the topic type management of the data content in the memory is facilitated, for example, the reading and analysis of the data content with the specified topic type are controlled.
And the content parsing module 1704 is configured to parse and read the storage content in the one or more data blocks in sequence according to the numbers of the one or more data blocks.
Because one or more data blocks meeting the preset requirements can be determined only by reading the block data index information of at least part of the data blocks, the embodiment of the invention avoids reading all data in the whole data block, so that the reading speed and the reading stability can be improved, the data can be selectively read through the block data index information, the complexity of file management is reduced, and the method and the device are more suitable for application scenes of automatic driving.
Only one or more of the individual modules in the above embodiments eight, nine and ten may be implemented in software, hardware, firmware or a combination thereof. The software or firmware includes, but is not limited to, computer program instructions or code and may be executed by a hardware processor. The hardware includes, but is not limited to, various integrated circuits such as, for example, master CPU402, master CPU602, data processing chip 404, data processing chip 504, or master chip 604.
An eleventh embodiment of the present invention provides an apparatus, see fig. 18, comprising a memory 1801 and a processor 1802, the memory 1801 storing computer program instructions, the processor 1802 executing the computer program instructions to perform operations as described in any one or more of embodiments one to seven of the present invention. The processor in the eleventh embodiment of the present invention may be the aforementioned master CPU402, master CPU602, data processing chip 404, data processing chip 504, or master chip 604.
When the present invention is applied to an autopilot scenario, the modules or devices in the apparatus may be implemented by the original devices, devices or modules in the vehicle-mounted system, for example, the memory 1801 may be a memory device in the main control CPU402, the main control CPU602, the data processing chip 404, the data processing chip 504 or the main control chip 604, the memory 1801 may also be the memory medium 405, the memory medium 505 or the flash memory chip 605, and the processor 1802 may be the main control CPU402, the main control CPU602, the data processing chip 404, the data processing chip 504 or the main control chip 604. In addition, the memory 1801 and the processor 1802 may be implemented in the same device or may be implemented on two different devices or apparatuses.
It will be apparent to those skilled in the art that the descriptions of the embodiments of the present application may be provided with reference to each other, and that the descriptions of the devices, the functions of the apparatus and the steps performed in the embodiments of the present application may be provided for convenience and brevity, for example, with reference to the description of the embodiments of the method of the present application, and the reference may also be made to the descriptions of the embodiments of the method of the present application between the embodiments of the apparatus.
Those skilled in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs all or part of the steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways without exceeding the scope of the application. For example, the embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Wherein the units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
In addition, the described apparatus and methods, as well as illustrations of various embodiments, may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the application. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electronic, mechanical or other forms.
The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (24)

1. A data writing method, comprising:
is N 1 Each of the individual packets is time stamped, where N 1 Is a positive integer;
the N is set to 1 The data packets are according to the N 1 Time-first of time-stamp of individual data packetsSequencing in sequence;
sorting the N 1 The data packets are orderly organized into a plurality of data pages based on the storage granularity of the pages, and each data page in the plurality of data pages comprises N 3 Data packets, where N 3 Is a positive integer;
sequentially organizing the plurality of data pages into a plurality of data blocks based on a block-based storage granularity, wherein each data block in the plurality of data blocks comprises block data index information and N 2 Data packets, where N 2 Is a positive integer and N 1 ≥N 2 ≥N 3 The block data index information comprises the numbers of the data blocks, and a correspondence relationship exists between the numbers of the plurality of data blocks and a plurality of channels for writing the plurality of data blocks into a storage medium in a sequential cycle;
And respectively writing at least one data block corresponding to each channel in the plurality of channels into the storage medium through each channel according to the serial number of the at least one data block.
2. The method of claim 1, wherein the block data index information further comprises the N 2 Time information of individual data packets.
3. The method of claim 1 or 2, wherein each data page further comprises page data index information, the page data index information comprising the N 3 Time information of individual data packets.
4. The method according to claim 1 or 2, wherein each data packet corresponds to a topic type, and the block data index information further includes the N 2 Theme type information of the individual packets.
5. The method of claim 4, wherein the topic type is used to identify the source or content of each data packet.
6. A data consistency check method, comprising:
checking whether a tail data page of a tail data block is free of data or not according to tail data blocks written last through each of a plurality of channels, wherein the channels are used for writing the data blocks into a storage medium, each data block in the data blocks comprises a plurality of data pages, at least one data page in the data pages comprises block data index information, the block data index information comprises a serial number of the data block, and a corresponding relation of sequential circulation exists between the serial numbers of the data blocks and the channels;
When there is no data in the tail data page of at least one tail data block, acquiring the minimum number S of the at least one tail data block 1 ；
Checking whether the numbers of the plurality of data blocks written through the plurality of channels are continuous or not according to the numbers of the plurality of tail data blocks written through the plurality of channels;
when there is at least one discontinuity in the numbers of the plurality of data blocks, determining a start number S of the at least one discontinuity 2 ；
Determining S 1 And S is equal to 2 Smaller values in between;
and erasing the data blocks with the numbers greater than or equal to the smaller value in the storage medium.
7. A data reading method, comprising:
determining a correspondence between a plurality of channels and numbers of a plurality of data blocks written into a storage medium through the plurality of channels by scanning block data index information of the data block written last through each of the plurality of channels, each of the plurality of data blocks including the block data index information, the block data index information including the numbers of the data blocks;
According to the corresponding relation, sequentially reading block data index information of at least part of the data blocks in the plurality of data blocks;
determining one or more data blocks meeting preset requirements in the plurality of data blocks according to the block data index information of at least part of the data blocks;
and sequentially analyzing and reading the storage content in the one or more data blocks according to the numbers of the one or more data blocks.
8. The method of claim 7, wherein the block data index information includes at least one of time information or topic type information of a plurality of data packets within each data block, and the preset requirements include at least one of requirements regarding the time information or requirements regarding the topic type information.
9. The method of claim 8, wherein the topic type is used to identify a source or content of each of the plurality of data packets.
10. A data writing apparatus, comprising:
a time stamp marking module for N 1 Each of the individual packets is time stamped, where N 1 Is a positive integer;
A sorting module for sorting the N 1 The data packets are according to the N 1 Time sequence ordering of time stamps of the data packets;
a data page organization module for organizing the N ordered 1 The data packets are orderly organized into a plurality of data pages based on the storage granularity of the pages, and each data page in the plurality of data pages comprises N 3 Data packets, where N 3 Is a positive integer;
a data block organization module for organizing the plurality of data pages based on a block-based storage granularitySequentially organizing a plurality of data blocks, wherein each data block of the plurality of data blocks comprises block data index information and N 2 Data packets, where N 2 Is a positive integer and N 1 ≥N 2 ≥N 3 The block data index information comprises the numbers of the data blocks, and a correspondence relationship exists between the numbers of the plurality of data blocks and a plurality of channels for writing the plurality of data blocks into a storage medium in a sequential cycle;
and the channel writing module is used for respectively writing at least one data block corresponding to each channel in the plurality of channels into the storage medium through each channel according to the serial number of the at least one data block.
11. The apparatus of claim 10, wherein the block data index information further comprises the N 2 Time information of individual data packets.
12. The apparatus of claim 10 or 11, wherein each data page further comprises page data index information, the page data index information comprising the N 3 Time information of individual data packets.
13. The apparatus according to claim 10 or 11, wherein each data packet corresponds to a topic type, and the block data index information further includes the N 2 Theme type information of the individual packets.
14. The apparatus of claim 13, wherein the topic type is used to identify a source or content of each of the data packets.
15. A data consistency check device, comprising:
a first checking module, configured to check whether a tail data page of a tail data block is free of data for a tail data block written last through each of a plurality of channels, where the plurality of channels are used to write a plurality of data blocks into a storage medium, each of the plurality of data blocks includes a plurality of data pages, at least one of the plurality of data pages includes block data index information, the block data index information includes a number of the data block, and a correspondence of sequential cycles exists between the number of the plurality of data blocks and the plurality of channels;
A first number obtaining module, configured to obtain a minimum number S of at least one tail data block when there is no data in a tail data page of the at least one tail data block 1 ；
A second checking module, configured to check whether numbers of the plurality of data blocks written through the plurality of channels are consecutive according to numbers of the plurality of tail data blocks written through the plurality of channels;
a second number acquisition module for determining a start number S of at least one discontinuity when the numbers of the plurality of data blocks are at least one discontinuity 2 ；
A comparison module for determining S 1 And S is equal to 2 Smaller values in between;
and the data block erasing module is used for erasing the data blocks with the numbers larger than or equal to the smaller value in the storage medium.
16. A data reading apparatus, comprising:
the corresponding relation determining module is used for determining the corresponding relation between the plurality of channels and the serial numbers of the plurality of data blocks by scanning the block data index information of the data block which is finally written in through each of the plurality of channels, wherein the plurality of data blocks are written in the storage medium through the plurality of channels, each data block in the plurality of data blocks comprises the block data index information, and the block data index information comprises the serial numbers of the data blocks;
The block data index information reading module is used for sequentially reading the block data index information of at least part of the data blocks in the plurality of data blocks according to the corresponding relation;
the data block to be read determining module is used for determining one or more data blocks meeting preset requirements in the plurality of data blocks according to the block data index information of the at least part of data blocks;
and the content analysis module is used for sequentially analyzing and reading the storage content in the one or more data blocks according to the numbers of the one or more data blocks.
17. The apparatus of claim 16, wherein the block data index information includes at least one of time information or topic type information for a plurality of data packets within each data block, and wherein the preset requirements include at least one of requirements regarding the time information or requirements regarding the topic type information.
18. The apparatus of claim 17, wherein the topic type is used to identify a source or content of each of the plurality of data packets.
19. A data writing device comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform the method of any of claims 1-5.
20. A data consistency check device comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform the method of claim 6.
21. A data reading apparatus comprising a memory storing computer program instructions and a processor executing the computer program instructions to perform the method of any of claims 7-9.
22. A computer storage medium comprising computer instructions which, when executed by a processor, cause a data writing apparatus to perform the method of any of claims 1-5.
23. A computer storage medium comprising computer instructions which, when executed by a processor, cause a data consistency check apparatus to perform the method of claim 6.
24. A computer storage medium comprising computer instructions which, when executed by a processor, cause a data reading apparatus to perform the method of any of claims 7-9.
CN201980101436.0A 2019-11-13 2019-11-13 Method and device for data writing, consistency checking and reading Active CN114556283B (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/117854 WO2021092784A1 (en) 2019-11-13 2019-11-13 Data writing, consistency checking and reading method and apparatus

Publications (2)

Publication Number Publication Date
CN114556283A CN114556283A (en) 2022-05-27
CN114556283B true CN114556283B (en) 2024-04-12

Family

ID=75911307

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201980101436.0A Active CN114556283B (en) 2019-11-13 2019-11-13 Method and device for data writing, consistency checking and reading

Country Status (2)

Country Link
CN (1) CN114556283B (en)
WO (1) WO2021092784A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113703668B (en) * 2021-07-15 2023-06-23 苏州浪潮智能科技有限公司 Information query method and system of solid state disk, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103226965A (en) * 2013-03-29 2013-07-31 浙江大学 Time-bitmap-based audio/video data access method
CN106970935A (en) * 2017-01-20 2017-07-21 朗坤智慧科技股份有限公司 Real-time data memory structure, method for writing data and method for reading data
CN108256076A (en) * 2018-01-18 2018-07-06 广州大学 Distributed mass data processing method and processing device
CN110134333A (en) * 2019-05-07 2019-08-16 深圳忆联信息系统有限公司 A kind of method and its system reset write-in data flow and promote SSD reading splicing rate

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101727402B (en) * 2009-10-23 2012-07-04 深圳市江波龙电子有限公司 Read and write control method and system of data of nonvolatile storage
CN102955743A (en) * 2011-08-25 2013-03-06 建兴电子科技股份有限公司 Solid state drive and wear leveling control method for same

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103226965A (en) * 2013-03-29 2013-07-31 浙江大学 Time-bitmap-based audio/video data access method
CN106970935A (en) * 2017-01-20 2017-07-21 朗坤智慧科技股份有限公司 Real-time data memory structure, method for writing data and method for reading data
CN108256076A (en) * 2018-01-18 2018-07-06 广州大学 Distributed mass data processing method and processing device
CN110134333A (en) * 2019-05-07 2019-08-16 深圳忆联信息系统有限公司 A kind of method and its system reset write-in data flow and promote SSD reading splicing rate

Also Published As

Publication number Publication date
CN114556283A (en) 2022-05-27
WO2021092784A1 (en) 2021-05-20

Similar Documents

Publication Publication Date Title
CN110764706B (en) Storage system, data management method, and storage medium
US8472449B2 (en) Packet file system
CN100407123C (en) Data storage device, reconstruction controlling device, reconstruction controlling method, and storage medium
CN104484427B (en) A kind of video file storage device and method
TW201937361A (en) Storage device
US20160034201A1 (en) Managing de-duplication using estimated benefits
CN108399050B (en) Data processing method and device
US11250888B1 (en) Flash memory and method for storing and retrieving embedded audio video data
US8572307B2 (en) Memory controller, memory card, and nonvolatile memory system
CN109408050A (en) The code generating method and device of appliance applications interface for rail vehicle
CN114556283B (en) Method and device for data writing, consistency checking and reading
CN113448946B (en) Data migration method and device and electronic equipment
US20140244699A1 (en) Apparatus and Methods for Selective Location and Duplication of Relevant Data
CN117156172B (en) Video slice reporting method, system, storage medium and computer
CN107577549A (en) It is a kind of to store the method for testing for deleting function again
CN103970512A (en) Multi-core processor and parallel replay method thereof
CN107967118A (en) A kind of method and apparatus realized hard disc data high speed dump and checked
CN110297811B (en) Data processing method and device, computer system and computer readable storage medium
CN110737389A (en) Method and device for storing data
CN109416617A (en) Duplication between heterogeneous storage system
CN111625502B (en) Data reading method and device, storage medium and electronic device
CN111309526B (en) File backup and recovery method and device
US20190354306A1 (en) Available-space management method for nonvolatile memory, access device which stores data to information storage device with nonvolatile memory, information storage device, and information storage system
CN105827681A (en) Backup data transmission method under SAN (Storage Area Network)
CN106991021A (en) The method and system of new data file are built from available data file

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant