CN116301602A - Data recording or reading method and device, acquisition equipment, vehicle and medium - Google Patents

Data recording or reading method and device, acquisition equipment, vehicle and medium Download PDF

Info

Publication number
CN116301602A
CN116301602A CN202310139703.1A CN202310139703A CN116301602A CN 116301602 A CN116301602 A CN 116301602A CN 202310139703 A CN202310139703 A CN 202310139703A CN 116301602 A CN116301602 A CN 116301602A
Authority
CN
China
Prior art keywords
data
file
cache
preset
writing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310139703.1A
Other languages
Chinese (zh)
Inventor
王宽
任凡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing Changan Automobile Co Ltd
Original Assignee
Chongqing Changan Automobile Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing Changan Automobile Co Ltd filed Critical Chongqing Changan Automobile Co Ltd
Priority to CN202310139703.1A priority Critical patent/CN116301602A/en
Publication of CN116301602A publication Critical patent/CN116301602A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0661—Format or protocol conversion arrangements
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present application relates to the field of vehicle data processing technologies, and in particular, to a data recording or reading method, a device, an acquisition device, a vehicle, and a medium, where the method includes: identifying an actual capacity of target data to be recorded; if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating the written cache index of the preset cache; if the actual capacity is greater than or equal to the residual storage capacity of the preset cache, partial data of the target data are written into the preset cache, when the preset cache is full, the data of the preset cache are written into a record file of the storage device through an operation interface of the operating system, the residual data of the target data are copied to a first address of the preset cache, and the written cache index of the preset cache is updated. Therefore, the problems that the system cache is easy to clean and cause jamming when a large amount of high-frequency data is recorded in the related technology, the safety of automatic driving of the vehicle is reduced and the like are solved.

Description

Data recording or reading method and device, acquisition equipment, vehicle and medium
Technical Field
The present disclosure relates to the field of vehicle data processing technologies, and in particular, to a data recording or reading method, a device, an acquisition apparatus, a vehicle, and a medium.
Background
With the development of technology, the automatic driving technology is widely applied. The automatic driving technology is a technology for simulating the behavior of a driver through an automatic driving system on a vehicle so as to control the vehicle to normally run, and through the automatic driving technology, the operation required to be executed by the driver when the driver drives the vehicle can be reduced, and the driving experience of the driver is improved. The automatic driving system not only needs to acquire the operation required to be executed by a driver when driving the automobile, but also needs to acquire and collect related data information around the automobile, such as lane lines, crowd, front-rear automobile distances, obstacles and the like, and executes corresponding operation according to the related data information, thereby improving safety.
The vehicle can record the related data based on the data recording system of the distributed server network of the network management server and the data recording server, thereby effectively reducing the concentrated load of the server and facilitating the grouping management; or, the data recording of the data recorder is completed through an RS422 interface and an RS232 interface; or a data recording device which reduces the writing process to the data recording unit and increases the writing data amount per time to reduce the writing cost, and uses high-speed sampling to ensure the large-scale integrated recording point sampling data.
In the related art, from the aspects of data recording and unloading of a data recording device, a distributed data recording system and a high-speed sampling method, practical tests show that when the system cache is too high, sporadic jamming can occur in the whole system, and the safety of an automatic driving system is seriously influenced.
Disclosure of Invention
The application provides a data recording or reading method, a device, acquisition equipment, a vehicle and a medium, so as to solve the problems that the system cache is cleaned easily to cause a jam and the safety of automatic driving of the vehicle is reduced and the like when a large amount of high-frequency data is recorded in the related technology.
An embodiment of a first aspect of the present application provides a data recording method, including the steps of: identifying an actual capacity of target data to be recorded; if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating a write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing part of the data of the target data into the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the remaining data of the target data to a head address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a kernel cache of the operating system is forbidden by using a no-buffer mark.
According to the technical means, the embodiment of the application identifies the actual capacity of the target data to be recorded, carries out different processing on the actual capacity of different target data, writes the target data into the preset cache if the actual capacity is smaller than the remaining storage capacity of the preset cache, and updates the written cache index; if the actual capacity is greater than or equal to the residual storage capacity of the preset cache, partial data are written into the preset cache, when the preset cache is full, the cache data are written into a record file of the storage device through an operating system interface, the residual data are copied to a cache head address, and the written cache index of the cache is updated, so that the problem of stuck caused by system cache cleaning during high-frequency large-volume data recording can be effectively solved, and the safety of automatic driving of a vehicle is ensured.
Optionally, the head address of the preset buffer is a multiple of the data block in the storage device, and the maximum capacity of the buffer is a multiple of the data page in the storage device.
It can be understood that when the first address of the preset cache is a multiple of the data block in the storage device and the maximum capacity of the cache is a multiple of the data page in the storage device, the cache can be directly written into the storage device without the cache of the operating system, so that the problem of sporadic cartoon of the operating system when the cache of the operating system is too high is solved.
Optionally, the writing, by the operation interface of the operating system, the preset cached data into the record file of the storage device includes: identifying whether the file capacity of the recorded file is larger than a preset capacity; if the file capacity is smaller than or equal to the preset capacity, writing the preset cache into the storage device, and entering a next writing period; and if the file capacity is larger than the preset capacity, or an abnormal signal is received, calling an abnormal signal processing function to write the preset cached data into the storage device, closing the current record file after writing, generating a new record file, and restarting recording the data under the new record file.
According to the technical means, in the embodiment of the application, whether the file capacity of the recorded file is larger than the set capacity is judged, so that the next operation is performed, and if the file capacity is smaller than or equal to the set capacity, the cache is written into the storage equipment and enters the next period; if the data is larger than the preset capacity or abnormal information is received, an abnormal signal processing function is called to write the cached data into the storage equipment, a record file is closed after writing, a new file record is generated, and the recorded data is restarted under the new file, so that the data is required to be cached and packetized when the data is recorded, loss caused by data deletion is avoided, and the safety of the data is ensured.
Optionally, the calling the exception signal processing function writes the preset cached data into the storage device, including: judging whether the current cache byte number of the preset cache is an integer multiple of a data page in the storage device or not; if the number of the current cache bytes is an integer multiple of the data page, writing the preset cached data into the storage device; if the current cache byte number is a non-integer multiple of the data page, calculating an excessive writing byte number according to the current cache byte number and a preset multiple of the data page, writing the excessive writing byte number and the current cache byte number into the storage device together, and deleting the excessive writing byte number after writing.
According to the technical means, the embodiment of the application judges whether the current cache byte number of the preset cache is an integer multiple of the data page, if yes, the cache data is written into the storage device, if not, the current cache byte number and a certain multiple of the data page are calculated, the redundant write byte numbers are written into the storage device together, and the redundant write byte numbers are deleted after writing, and because the data size of the write storage device is a multiple of the memory page size, the data can be directly written into the storage device without being cached by the operating system, thereby solving the problem of sporadic clamping when the cache of the operating system is too high.
Optionally, before identifying the remaining storage capacity of the predefined preset cache, the method further includes: defining a data recording interface based on a serialized data recording protocol; and calling the data recording interface to perform serialization processing on the received target data to obtain serialized data, and writing the serialized data into the preset cache.
According to the technical means, the data recording interface is defined based on the serialized data recording protocol, the interface is called to perform serialized processing on the received target data to obtain serialized data, the serialized data is written into the preset cache, and the data can be converted into a specific format by means of automatic and manual recording of the data and the like so as to facilitate subsequent writing and storage.
Optionally, the serialized data recording protocol includes a file header structure, file header content, data header structure, and data header content, the defining a data recording interface based on the serialized data recording protocol includes: reading a configuration file and acquiring a data type name list; traversing the data type name list to obtain a message descriptor; if the message descriptor is empty, judging that the record information is wrong, otherwise, acquiring a dependent file, and executing recursion operation on the dependent file until the dependent file is cleared; initializing the file header structure, writing the interface content length of the file header structure and the file header content into a record file, defining a data header structure when recording each frame of data, carrying out serialization processing on the data by using a protocol buffer area interface, and writing the updated data header content into the record file.
According to the technical means, all data type names to be recorded are predefined through the configuration file, text contents of all interfaces are obtained at one time based on the configuration file when a program is initialized, the interface text contents are filled into the file header content part of the data, so that when the subsequent data is played back, corresponding text contents of the interface file stored in the data file header are read, the interface text contents are loaded into a data pool after being processed, and further the subsequent reverse serialization for the serialized data is supported, and the interpretation of the recorded file is realized.
Optionally, the serialized data recording protocol includes a file header structure, file header content, data header structure, and data header content, the defining a data recording interface based on the serialized data recording protocol includes: creating a data type name list; defining a data head structure, carrying out serialization processing on data by using a protocol buffer area interface, and writing updated data head content into the record file; before stopping recording data, defining the file header structure, continuously arranging the interface content length of the file header structure and the file header content, and inserting the file header structure and the file header content into the recording file.
According to the technical means, the embodiment of the application does not need to configure a file, judges whether the interface text content with the type variable is processed and searched when data is recorded each time, if the interface text content with the type variable is not processed, creates a data type name list for recording the variable type with the type variable which is processed and searched and corresponds to the text content, defines a data header structure, uses a protocol buffer interface to perform serialization processing on the data, writes the updated data header content into a record file, defines a file header structure before stopping recording, arranges and inserts the interface content length and the file header content into the record file, so that when the subsequent data is played back, the corresponding text content of the interface file stored in the data file header is read, and the interface text content is loaded into a data pool after being processed, thereby further supporting subsequent anti-serialization for the serialized data and realizing the interpretation of the record file itself.
Optionally, while disabling kernel buffering of the operating system by using the unbuffered identifier, the method further includes: opening a file descriptor by using an opening function of the operating system; and opening the record file by using the file descriptor, and creating a target file.
According to the technical means, the file descriptor can be opened by using the opening function of the operating system, the record file is opened by using the file descriptor, the target file is created, and once the target file is created, the received data is indicated, so that the user can be prompted more intuitively.
An embodiment of a second aspect of the present application provides a data reading method, including the steps of: acquiring a data reading request; opening a record file with any recorded serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure; generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
An embodiment of a third aspect of the present application provides a data recording apparatus, including: the identification module is used for identifying the actual capacity of the target data to be recorded; the first updating module is used for writing the target data into the preset cache if the actual capacity is smaller than the residual storage capacity of the preset cache, and updating the written cache index of the preset cache; and the second updating module is used for writing partial data of the target data into the preset cache if the actual capacity is greater than or equal to the residual storage capacity of the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the residual data of the target data to a first address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a no-buffer mark is used for prohibiting kernel cache of the operating system.
An embodiment of a fourth aspect of the present application provides a data reading apparatus, including: the acquisition module is used for acquiring the data reading request; the filling module is used for opening a record file which is randomly recorded with the serialized data in the storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure; and the deserializing module is used for generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and deserializing the serialized data by utilizing the preset data template and the character string to obtain target data.
A fifth aspect of the present application provides a data acquisition device for: identifying an actual capacity of target data to be recorded; if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating a write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing part of the data of the target data into the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the remaining data of the target data to a head address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a kernel cache of the operating system is forbidden by using a no-buffer mark.
An embodiment of a sixth aspect of the present application provides a data reading apparatus for: acquiring a data reading request; opening a record file with any recorded serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure; generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
An embodiment of a seventh aspect of the present application provides a vehicle, including: a storage device, wherein one or more record files are created in the storage device; the data acquisition device is used for identifying the actual capacity of target data to be recorded; if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating a write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing part of the data of the target data into the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the remaining data of the target data to a head address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a kernel cache of the operating system is forbidden by using a no-buffer mark.
Optionally, the method further comprises: a reading device for acquiring a data reading request; opening a record file with any recorded serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure; generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
An eighth aspect of the present application provides a computer-readable storage medium having stored thereon a computer program for execution by a processor for implementing the data recording method or the data reading method as described in the above embodiments.
Therefore, the application has at least the following beneficial effects:
(1) The embodiment of the application identifies the actual capacity of target data to be recorded, carries out different processing aiming at the actual capacity of different target data, writes the target data into a preset cache if the actual capacity is smaller than the residual storage capacity of the preset cache, and updates the written cache index; if the actual capacity is greater than or equal to the residual storage capacity of the preset cache, partial data are written into the preset cache, when the preset cache is full, the cache data are written into a record file of the storage device through an operating system interface, the residual data are copied to a cache head address, and the written cache index of the cache is updated, so that the problem of stuck caused by system cache cleaning during high-frequency large-volume data recording can be effectively solved, and the safety of automatic driving of a vehicle is ensured.
(2) According to the method and the device, when the head address of the preset cache is the multiple of the data block in the storage device and the maximum capacity of the cache is the multiple of the data page in the storage device, the cache can be directly written into the storage device without being cached by the operating system, so that the problem of sporadic cartoon of the operating system when the cache of the operating system is too high is solved.
(3) According to the embodiment of the application, whether the file capacity of the recorded file is larger than the set capacity is judged, so that the next operation is performed, and if the file capacity is smaller than or equal to the set capacity, the cache is written into the storage equipment and enters the next period; if the data is larger than the preset capacity or abnormal information is received, an abnormal signal processing function is called to write the cached data into the storage equipment, a record file is closed after writing, a new file record is generated, and the recorded data is restarted under the new file, so that the data is required to be cached and packetized when the data is recorded, loss caused by data deletion is avoided, and the safety of the data is ensured.
(4) According to the method and the device, whether the current cache byte number of the preset cache is an integer multiple of a data page or not is judged, if yes, cache data are written into the storage device, if not, the current cache byte number and a certain multiple of the data page are calculated, the redundant write byte number is jointly written into the storage device, and after writing, the redundant write byte number is deleted, and because the data size of the write storage device is a multiple of the memory page size, the data can be directly written into the storage device without being cached by an operating system, so that the problem of sporadic cartoon of the operating system when the cache of the operating system is too high is solved.
(5) According to the embodiment of the application, the data recording interface is defined based on the serialized data recording protocol, the interface is called to conduct serialized processing on the received target data to obtain serialized data, the serialized data are written into the preset cache, and the data can be converted into a specific format through automatic and manual recording modes of the data, so that subsequent writing and storage are facilitated.
(6) According to the method and the device for processing the interface text content, all data type names needing to be recorded are predefined through the configuration file, text content of all interfaces is obtained at one time based on the configuration file when a program is initialized, the interface text content is filled into a file header content part of data, so that when subsequent data playback is conducted, corresponding text content of the interface file stored in the data file header is read, the interface text content is processed and then is loaded into a data pool, further subsequent reverse serialization for serialized data is supported, and interpretation of the recorded file is achieved.
(7) According to the embodiment of the application, a configuration file is not needed, whether the interface text content with the type variable is processed is judged when data is recorded each time, if the interface text content with the type variable is processed, a data type name list is created for recording the variable type of the text content corresponding to the interface which is processed and searched, a data head structure is defined, a protocol buffer area interface is used for carrying out serialization processing on the data, updated data head content is written into a record file, the file head structure is defined before recording is stopped, the interface content length and the file head content are arranged and inserted into the record file, so that when the subsequent data is played back, the corresponding text content of the interface file stored in the data file head is read, and the interface text content is loaded into a data pool after being processed, so that subsequent reverse serialization aiming at the serialized data is further supported, and the self-interpretability of the record file is realized.
(8) According to the embodiment of the application, the file descriptor can be opened by using the opening function of the operating system, the record file is opened by using the file descriptor, the target file is created, and once the target file is created, the received data is indicated, so that the user can be prompted more intuitively.
Additional aspects and advantages of the application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flow chart of a data recording method according to an embodiment of the present application;
FIG. 2 is a flow chart of a data reading method according to an embodiment of the present application;
FIG. 3 is an overall process flow diagram provided in accordance with an embodiment of the present application;
FIG. 4 is a flow chart of exception handling provided according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a data structure provided according to an embodiment of the present application;
fig. 6 is a schematic diagram of data packetization according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a protocol provided according to an embodiment of the present application;
FIG. 8 is a flowchart of a data recording scheme according to an embodiment of the present application;
FIG. 9 is a second flowchart of a data recording scheme according to an embodiment of the present application;
FIG. 10 is a flow chart of data parsing reinjection provided according to an embodiment of the present application;
FIG. 11 is a block diagram of a data recording apparatus according to an embodiment of the present application;
FIG. 12 is a block diagram of a data reading apparatus according to an embodiment of the present application;
fig. 13 is a schematic structural view of a vehicle according to an embodiment of the present application.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended for the purpose of explaining the present application and are not to be construed as limiting the present application.
Different modes can be adopted aiming at the data recording modes, each mode has the corresponding effect, the following embodiments are respectively innovated from the aspects of data recording and unloading of a data recording device, a distributed data recording system and a high-speed sampling method, and can not effectively solve the problem of stuck caused by system cache cleaning during high-frequency large-volume data recording.
In the related art (1), a data recording and unloading method of a micro airborne data recording device, the data recording device comprises a data recorder and a data adapter plate, the data recorder is connected with an airborne system through a micro interface plate, the data recorder is connected with the data adapter plate through the micro interface plate, and the data adapter plate is connected with an external computer through a standard interface. It includes a data recording step and a data unloading step. The data recording step comprises two substeps of a default working mode and an interrupt buffer mode, and the data unloading step comprises three substeps of external computer control data unloading, airborne system control data unloading and data unloading. The data recorder is simple in structure and small in size, data recording of the data recorder can be completed through the RS422 interface and the RS232 interface, and high-speed data unloading of the data recorder can be achieved through the USB3.0 interface.
In related art (2), a network data recording system includes: the system comprises a plurality of remote measurement and control units, a convergence routing device, a network control server and a data recording server, wherein the convergence routing device is connected with the plurality of remote measurement and control units and provides a remote network access function for the plurality of remote measurement and control units, the user application is used for monitoring and managing the remote measurement and control units and the convergence routing device, the network control server comprises a remote measurement and control unit network control server, a convergence routing device network control server and a user application network control server, and the remote measurement and control unit data recording server, the convergence routing device data recording server and the user application data recording server. The application provides a data recording system of a distributed server network based on a network management server and a data recording server, so that the centralized load of the server can be effectively reduced, and the grouping management is convenient; and a dynamic key technology is adopted to improve the data security of the system.
In the related art (3), by the data recording apparatus and the inspection apparatus and the control apparatus having the data recording apparatus, a data recording apparatus is realized which reduces the writing process to the data recording unit and increases the amount of data written each time to reduce the writing overhead, and ensures large-scale sampling data of the total recorded points by high-speed sampling.
Because of the characteristics of the linux operating system, data must be cached in the kernel of the operating system from the user space and then transferred to the hard disk. In daily use, the cache can effectively improve IO performance, but in an automatic driving data acquisition scene, the cache of an operating system can be quickly raised and kept high due to high-frequency writing of a large amount of data into a hard disk. Actual tests show that when the system cache is too high, sporadic jamming occurs in the whole system, and the safety of an automatic driving system is seriously affected.
Therefore, the embodiment of the application provides a zero-copy data recording method, which can be directly written into a hard disk without system buffering, and effectively solves the problem of clamping.
In the related art (4), a method, a system and a vehicle for storing and reading data of a heterogeneous sensor comprise: and (3) data storage: carrying out serialization processing on the sensor data; respectively endowing the serialized byte size, the enumerated values of the data source and the data type, the current time of the acquired data and the predefined verification code to corresponding variables in the data head structure body; writing a data head; writing the serialized data; reading data: reading the size bytes of the data head, generating a data head variable, and acquiring a corresponding variable stored in the data head; checking the verification code, if the verification code fails, throwing out an exception, and if the verification code fails, reading and acquiring continuous bytes with the length of the specified data size in the data head; and according to the data source and the data type in the data head, combining a predefined deserialization mode, deserializing the read serialized data, and obtaining the original real sensor data. The invention provides a data recording protocol supporting efficient multi-element sensor data recording, indexing and reinjection.
The above application provides a method for storing serialized data, which can accurately obtain the serialized data of the corresponding data according to the time sequence when the data is recorded, but the anti-sequence part needs to be combined with a predefined anti-serialization method, and the predefined part is not embodied in the recorded data and needs additional files for description, so that the independently recorded data cannot be self-interpreted, and if the improperly managed anti-serialization method or description file corresponding to the appointed version is lost or cannot be corresponding to the improperly managed, the data is equivalent to being used in an resolving way, and the management and maintenance cost of the data and the risk of the unusable data are increased.
Another solution to this problem is to replace the serialized data with structured data, where the binary file recorded contains the name and value of each field of the data, but this typically results in an increased cpu load when recording the data, and an increased file size.
Therefore, the embodiment of the application provides a self-explanatory serialized binary file record reinjection method, which can realize the self-explanation of data while adopting an efficient serialized recording scheme.
Data recording or reading methods, apparatuses, acquisition devices, vehicles, and media according to embodiments of the present application are described below with reference to the accompanying drawings. Specifically, fig. 1 is a flow chart of a data recording method according to an embodiment of the present application.
As shown in fig. 1, the data recording method includes the steps of:
in step S101, the actual capacity of the target data to be recorded is identified.
The target data may be data to be recorded, and is not particularly limited herein.
It can be appreciated that the embodiment of the application identifies the actual capacity of the data to be recorded by the data recording device, so that the corresponding operation can be performed according to different capacities.
In step S102, if the actual capacity is smaller than the remaining storage capacity of the preset buffer, the target data is written into the preset buffer, and the write buffer index of the preset buffer is updated.
The preset cache can be a raw_data_buf (cache space) which can be selected according to actual requirements; the buffer size is defined as KRawDataBufferMaxSize (maximum buffer size of buffer space), and is not particularly limited herein.
Wherein the cache index may record the location of each data in the cache and provide a pointer to the data value stored in the designated column of the table based on the index, finding the relevant data record; the buffer index rawdata_buf_cur_size (the size of the cell in the matrix buffer being operated) =0 written at the time of initialization is not specifically limited herein.
It can be understood that, in the embodiment of the present application, when the actual capacity of the target data to be recorded is smaller than the remaining storage capacity of the preset cache, the target data is written into the cache, and the written cache index is updated, so that the relevant data location can be quickly located when the data is acquired subsequently.
In step S103, if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing part of the data of the target data into the preset cache, when the preset cache is full, writing the data of the preset cache into the record file of the storage device through the operation interface of the operating system, copying the remaining data of the target data to the first address of the preset cache, and updating the write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, the kernel cache of the operating system is forbidden by using the no-buffer identifier.
The first byte address in the memory area is used as the first address of the memory area, which is not specifically limited herein.
The head address of the preset cache is a multiple of a data block in the storage device, and the maximum capacity of the cache is a multiple of a data page in the storage device.
The unbuffered Input/Output (I/O) may be a standard I/O (Input/Output) that does not buffer the character, and is not specifically limited herein.
It can be understood that, in the embodiment of the present application, when the actual capacity of the target data to be recorded is greater than or equal to the remaining storage capacity of the preset cache, part of the data is written into the preset cache, and when the preset cache is full, the cached data is written into the record file of the storage device through the operating system interface, the remaining data is copied to the cache head address and the written cache index of the cache is updated, which can effectively solve the problem of stuck caused by system cache cleaning during high-frequency large-scale data recording, and ensure the safety of automatic driving of the vehicle.
In this embodiment of the present application, before identifying the remaining storage capacity of the predefined preset cache, the method further includes: defining a data recording interface based on a serialized data recording protocol; and calling a data recording interface to perform serialization processing on the received target data to obtain serialized data, and writing the serialized data into a preset cache.
The serialized data recording protocol includes a header structure, header content, a header structure, and header content, which are not specifically limited herein.
It can be understood that, before the remaining storage capacity of the preset buffer is identified, the embodiment of the application needs to define the data recording interface through the serialization data recording protocol, call the interface to perform serialization processing on the received data, write the data into the preset buffer, and convert the data into a specific format through modes of automatic and manual recording of the data, so as to facilitate subsequent writing and storage.
It should be noted that, in the embodiment of the present application, the data recording interface is defined based on the serialized data recording protocol, not only all the data type names to be recorded can be predefined through the configuration file, but also the text content of all the interfaces is obtained once based on the configuration file when the program is initialized; the interface text content of which the type of the variable is searched is judged whether to be processed or not and the next operation is carried out without a configuration file when the data is recorded each time, and the specific limitation is not carried out.
As one possible implementation, defining a data recording interface based on a serialized data recording protocol includes: reading a configuration file and acquiring a data type name list; traversing the data type name list to obtain a message descriptor; if the message descriptor is empty, judging that the recorded information is wrong, otherwise, acquiring the dependent file, and executing recursion operation on the dependent file until the dependent file is cleared; initializing a file header structure, writing the interface content length and the file header content of the file header structure into a record file, defining a data header structure when recording each frame of data, carrying out serialization processing on the data by using a protocol buffer interface, and writing the updated data header content into the record file.
The configuration file may be some computer program or a system configuration parameter and an initialization setting, and may perform different operations according to different situations, which is not limited herein specifically.
The message descriptor may be a feature describing the message, such as a priority, a life cycle, a name, etc. of the message, and is specifically described according to practical situations, which is not limited herein.
Wherein, the dependent file may be that each interface file depends on other interface files, which is not specifically limited herein.
The recursive operation can define a wireless set of objects by using limited sentences, convert large complex problem rubbing into a small-scale problem similar to the original problem to solve, repeat calculation for many times, reduce the workload, and is not particularly limited.
The protocol buffers provide a language neutral, platform neutral, extensible mechanism for serializing structured data in a forward and backward compatible manner, among other things, without limitation.
It can be understood that, in the embodiment of the present application, all data type names to be recorded are predefined through a configuration file, when a program is initialized, text contents of all interfaces are obtained at one time based on the configuration file, and interface text contents are filled into a file header content part of data, so that when subsequent data playback is performed, corresponding text contents of an interface file stored in a data file header are read, and the interface text contents are loaded into a data pool after being processed, thereby further supporting subsequent anti-serialization for serialized data, and realizing interpretation of the record file itself.
As another possible implementation, defining a data recording interface based on a serialized data recording protocol includes: creating a data type name list; defining a data head structure, carrying out serialization processing on data by using a protocol buffer area interface, and writing updated data head content into a record file; before stopping recording data, defining file header structure, continuously arranging interface content length and file header content of file header structure, and inserting them into recorded file.
It can be understood that, in the embodiment of the present application, a configuration file is not required, it is determined whether the interface text content with the type variable is processed and searched for is performed when data is recorded each time, if not, a data type name list is created for recording the variable type with the type variable corresponding to the interface and defining a data header structure, the data is serialized by using a protocol buffer interface, the updated data header content is written into a recording file, the file header structure is defined before recording is stopped, the interface content length and the file header content are arranged and inserted into the recording file, so that when the subsequent data is played back, the corresponding text content of the interface file stored in the data file header is read, and the interface text content is processed and then loaded into a data pool, thereby further supporting the subsequent anti-serialization for the serialized data, and realizing the interpretation of the recording file itself.
In this embodiment of the present application, writing, by an operation interface of an operating system, data of a preset cache to a record file of a storage device includes: identifying whether the file capacity of the recorded file is larger than a preset capacity; if the file capacity is smaller than or equal to the preset capacity, writing the preset cache into the storage device, and entering the next writing period; if the file capacity is larger than the preset capacity, or an abnormal signal is received, calling an abnormal signal processing function to write the preset cached data into the storage device, closing the current record file after writing, generating a new record file, and restarting recording the data under the new record file.
The preset capacity may be a capacity set by a user, and may be specifically set according to actual situations, which is not specifically limited herein.
The exception signal processing function may be Fulsh And Close File (refreshing and closing a file), which is not specifically limited herein.
It can be understood that, in the embodiment of the present application, whether the file capacity of the record file is greater than the set capacity is identified to determine the next operation, if the file capacity is less than or equal to the set capacity, the preset cache is written into the storage device and the next writing period is entered; if the data is larger than the set capacity or an abnormal signal is received, an abnormal signal processing function is called to write the cached data into the storage equipment, a record file is closed after writing, a new file record is generated, and the recorded data is restarted under the new file, so that the data is required to be cached and packetized when the data is recorded, the loss caused by data deletion is avoided, and the safety of the data is ensured.
In this embodiment of the present application, calling an exception signal processing function to write preset cached data into a storage device includes: judging whether the number of the current cache bytes of the preset cache is an integer multiple of a data page in the storage device; if the number of the current cache bytes is an integer multiple of the number of the data pages, writing the data of the preset cache into the storage device; if the current cache byte number is a non-integer multiple of the data page, calculating the redundant write byte number according to the current cache byte number and the preset multiple of the data page, writing the redundant write byte number and the current cache byte number into the storage device together, and deleting the redundant write byte number after writing.
The data page includes a header, a maximum and minimum record, a User record (User Records), a free space, a page directory, and a tail, which are not specifically limited herein.
The preset multiple may be a multiple set by a user, and may be set according to actual situations, which is not specifically limited herein.
It can be understood that, in the embodiment of the present application, whether the current buffer byte number of the preset buffer memory is an integer multiple of the data page is determined, if yes, the buffer memory data is written into the storage device, if not, the current buffer byte number and a certain multiple of the data page are calculated, the redundant write byte number is written into the storage device together, and after writing, the redundant write byte number is deleted, and because the size of the write storage device is a multiple of the size of the memory page, the data volume must be directly written into the storage device without the buffer memory of the operating system, thereby solving the problem of occasional stuck in the case of overhigh buffer memory of the operating system.
In this embodiment of the present application, while disabling kernel buffering of an operating system by using a bufferless identifier, the method further includes: opening the file descriptor by using an opening function of the operating system; and opening the record file by using the file descriptor to create the target file.
It can be understood that the embodiment of the application can open the file descriptor by using the opening function of the operating system, open the record file by using the file descriptor, create the target file, and once the target file is created, the received data is indicated, so that the user can be prompted more intuitively.
According to the data recording method provided by the embodiment of the application, the actual capacity of target data to be recorded is identified, different processing is carried out aiming at the actual capacity of different target data, if the actual capacity is smaller than the residual storage capacity of the preset cache, the target data is written into the preset cache, and the written cache index is updated; if the actual capacity is greater than or equal to the residual storage capacity of the preset cache, partial data are written into the preset cache, when the preset cache is full, the cache data are written into a record file of the storage device through an operating system interface, the residual data are copied to a cache head address, and the written cache index of the cache is updated, so that the problem of stuck caused by system cache cleaning during high-frequency large-volume data recording can be effectively solved, and the safety of automatic driving of a vehicle is ensured.
Next, a data reading method according to an embodiment of the present application will be described with reference to the accompanying drawings.
Fig. 2 is a flowchart of a data reading method according to an embodiment of the present application.
As shown in fig. 2, the data reading method includes the steps of:
in step S201, a data read request is acquired.
It can be appreciated that the embodiment of the present application facilitates subsequent opening of the record file according to the read request by obtaining the data read request.
In step S202, a record file with any serialized data recorded in the storage device is opened according to the data reading request, a header of the record file is read, header contents corresponding to the header are filled into a corresponding header structure, and an interface content length is filled into an interface content variable of the header structure.
It can be understood that, in the embodiment of the present application, according to a data reading request, a record file in which serialized data is arbitrarily recorded in a storage device is opened, a header of the record file is read, header content corresponding to the header is filled into a corresponding header structure, and an interface content length is filled into an interface content variable of the header structure, so that a data pool is generated based on the filled header structure.
In step S203, a data pool is generated based on the filled file header structure, the data header in the data pool is read, the data content corresponding to the data header is filled into the character string, the data type of the data content is obtained, the preset data template is matched according to the data type name, and the serialized data is deserialized by using the preset data template and the character string, so as to obtain the target data.
The data pool may be a data pool set by a user, and may be set according to actual needs of the user, which is not limited herein.
The preset data template may be a data template set by a user, and may be selected according to actual needs of the user, which is not limited herein.
It can be understood that, in the embodiment of the present application, a data pool is generated based on the structure of the file header after filling, the data header is read, the corresponding data content is filled into the character string, the serialized data is deserialized by using the set data template and the character string, so as to obtain the target data, the interpretation of the record file itself is realized, and the record is the serialized binary data, and the acquisition and loading of the interface are performed during the operation, no matter any interface is newly added or changed in the later stage, and no modification is required to be made to the program for recording and playing back the data.
According to the data reading method provided by the embodiment of the application, by acquiring a data reading request, opening a record file with any record of serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header content corresponding to the file header into a corresponding file header structure, filling interface content length into an interface content variable of the file header structure, generating a data pool based on the filled file header structure, reading a data header, filling the corresponding data content into a character string, and performing deserialization on the serialized data by using a set data template and the character string to obtain target data, so that the record file can be interpreted, and because the record file is the serialized binary data, the acquisition and loading of the interface are performed during operation, no modification is required for any program of data record and playback no matter whether any interface is newly added or changed in the later period.
The data recording or reading method proposed in the present application will be described in detail with reference to fig. 3 to 10, and is mainly divided into a data recording part and a data reinjection part, specifically as follows:
1. data recording part
Step 1, initializing, defining data write cache raw_data_buf (cache space), defining cache size as KRawDataBufferMaxSize (maximum cache amount of cache space), and currently writing cache index raw_data_buf_cur_size (unit size being operated in matrix buffer) =0. In order to write directly to the hard disk without going through the operating system cache, two points must be guaranteed here: firstly, the operating system is ensured to allocate a first address for the raw_data_buf (cache space) to be a multiple of the hard disk block size, and secondly, the KRAwDataBufferMaxSize (maximum cache amount of the cache space) is ensured to be a multiple of the memory page size. When recording data, the data is first stored in a write cache raw_data_buf (cache space).
Step 2, initializing, namely defining a variable file_name (variable file name), and finishing initialization by using the current time of the system, wherein the system time during data recording is used as the file name, such as 2022_06_01-12_31_23.Dat, and the dat file is the data file; defining a data packet size variable max_file_size (maximum size of a file), judging whether the recorded binary file size exceeds max_file_size (maximum size of the file) every time data is written into a hard disk, closing a current record file if the recorded binary file size exceeds max_file_size, and creating a new record file by taking the current system time as a file name.
Step 3, initializing, creating a data receiving socket and completing initializing link.
And step 4, initializing, creating a working thread and running periodically.
Step 5, initializing, namely binding an exception signal processing function FulshANdCloseFile, when a program receives an exception interrupt signal, executing cache safe writing hard disk operation and closing a file descriptor, as shown in fig. 4, in order to improve recording efficiency, when data is received, firstly storing the data in a cache raw_data_buf of a memory, and when the cache capacity is larger than KRAwDataBufferMaxSize, executing a hard disk writing instruction, wherein the zero copy mechanism requires that the data size of each hard disk to be written is a multiple of the memory page size, and the received data cannot be required to be a multiple of the memory page size, so that the received data must be split, and in most cases, the data of each hard disk to be written is incomplete, and part of bytes belonging to a certain packet of data are still in the memory cache and are not written into the hard disk; therefore, when no abnormal signal processing function exists, the data in the buffer memory can be lost, and the recorded data is incomplete and cannot be used.
And 6, receiving data, wherein when the data is received for the first time, an open function is used for opening a file descriptor fd in combination with an O_DIRECT (unbuffered input and output) parameter, and the fact that DIRECT I/O operation is adopted is indicated to an operating system kernel without buffering is indicated.
The file is opened when the data is received for the first time, so as to avoid creating an empty dat file when the data is not received; therefore, once the file is created, the data is clearly indicated to be received, and the user can be prompted more intuitively; meanwhile, since the initialization of the file_name is already completed in the step 2, the name of the file is still named by the time when the data recording module is started when the data is recorded.
Step 7, formatting the received data and adding necessary auxiliary information, wherein each frame of data is defined as rdata, and the data length is defined as len. The necessary auxiliary information is understood as a data header, and further describes information such as source, name, batch, length, etc. of the data, as shown in fig. 5. The data head is a user-defined structure body DataHead with continuous memory and fixed size.
Step 8, the following steps are performed on the data header data and the data rdata, respectively.
Step 9, as shown in fig. 6, judging whether the current writing cache space has enough residual space to store all received data, wherein the rawdata_buf_cur_size+len < = KRawDataBufferMaxSize, if True, copying the data to the data writing cache raw_data_buf, and updating the rawdata_buf_cur_size+ = len; if False, the copy-able byte size copy_len=krawdatabuffermaxsize-rawdata_buf_sur_size, and the remaining byte size eft _len=len-copy_len are calculated, the copy-able bytes are copied to raw_data_buf, and raw_data_buf is written to the hard disk as a whole, the remaining bytes are copied to raw_data_buf start position, and rawdata_buf_cur_size=left_len is updated.
And step 10, judging whether the current file size exceeds max_file_size, if not, entering the next period, and starting to execute the data from the step 6, if so, executing the following steps.
Step 11, calculating quotient and remainder of the division of the current cache byte number and the hard disk page size, judging whether the remainder is 0, and if True, writing the data in the current raw_data_buf into the hard disk; if False, calculating the size of the bytes to be written, calculating the size of the pages of the hard disk, namely, write_byte= (quoient+1), calculating the size of the bytes to be written, and more_byte = write_byte-rawdata_buf_cur_size, writing the size of the bytes to be written in the raw_data_buf into the hard disk, deleting the redundant bytes of the size of the file, and closing the current file descriptor; here, the extra bytes are added and then written into the hard disk, and the extra bytes are deleted again because the data size of the hard disk must be a multiple of the memory page size.
Step 12, updating file_name and opening new file
Step 13, periodically starting to execute from the step 6 receiving data.
Step 14, fulshANdCloseFile exception handling function execution scheme is consistent with step 10.
In the above description, the data recording module is used as an independent program, and continuously acquires data through the communication interface and records the received data.
The recording method still supports the user to call the related interface, so that the manual recording of the data is realized; the implementation steps are as follows:
step 15, an initialization step is performed, consistent with steps 1 and 2.
And step 16, filling the data head data, finishing serialization of the recorded data, calling a data recording interface, and transmitting the data head and the serialized data.
Step 17, repeating step 15 as needed, and performing the process in step 15 is consistent with steps 8 to 11.
And step 18, when the recording of the data is stopped, calling a FulshANdCloseFile function to write the data in the memory buffer into the hard disk and closing the file descriptor.
2. Data reinjection part
The data interface recorded by this method must be defined using protobuf. While there are two embodiments, specifically: one is to define all data type names needing to be recorded in advance through a configuration file, and when a program is initialized, text contents of all interfaces are obtained at one time based on the configuration file; and the other is to judge whether the interface text content of the type variable is processed or not when the data is recorded every time without configuration files, if so, the next step is continued, and if not, the interface text content of the corresponding variable type is acquired in the proto data pool.
The first scheme is shown in fig. 8:
step 1, reading the configuration file, and obtaining a list of all data type names to be recorded, namely topic Lists.
Step 2, traversing a list topic list, calling google:: protobuf:: descriptive pool:: generated_pool () - > FindMessageTypeByName () interface to obtain a message descriptor msg_descriptor, if msg_descriptor is empty, recording error information, and not loading the message type symbol when compiling the program; if msg_descriptor is not null, the interface calls text content to acquire and add the text content to the variable protoContents through msg_descriptor- > file () - > DebugString ().
Since each interface file may depend on other interface files, msg_descriptor- > file () - > dependency_count (), the number of file dependencies n is obtained, a loop is built based on the dependent file data n, and file_descriptor=msg_descriptor- > file () - > dependency (n). The recursively executing steps in the loop body until the dependent data is 0. The interface definition text content of all the data variable types to be recorded and the interface definition text content on which the data variable types to be recorded depend are completed.
Step 3, initializing header structure FileHead, fileHead is a predefined 1-byte aligned continuous structure, and includes at least version number version and proto interface content length proto Content Size. Other fields can be dynamically expanded according to actual requirements, and the update proto Content Size variable is the length size of the proto Contents.
And 4, creating a File descriptor, and writing the File Head content into the record File.
And 5, writing the proto Contents into the record file.
And 6, completing the writing of the content of the file header part.
Step 7, when each frame of Data needs to be recorded, defining a Data header structure Raw Data Head, wherein the structure is a 1-byte aligned custom structure, and at least comprises an array with a type of char for representing the message name, a variable size for representing the Data length and a time stamp for representing the Data time stamp. Other fields can be extended according to actual requirements.
Step 8, based on the recorded message, using the Serialize To String interface of the protobuf to sequence the Data, updating the Raw Data Head content and writing the Data into the file, and then writing the serialized Data into the file.
And 9, repeating the steps 7 to 8, and continuously recording each frame of data.
And step 10, closing the file descriptor and finishing data recording.
The second way is shown in fig. 9:
step 1, initializing, creating a searched data type name list handled Type List, which is used for recording the variable types of the text content corresponding to the interface which is processed and searched.
And 2, when each frame of Data needs to be recorded, defining a Data Head structure Raw Data Head, serializing the Data by using a Serialize To String interface of protobuf based on the recorded information, updating the content of the Raw Data Head, writing the content into a file, and then writing the serialized Data into the file.
Step 3, searching for the data type name at handled Type List, skipping if the data type name is found, and if the data type name is not found, indicating that the data type is recorded for the first time, and acquiring an interface text according to the following method is needed.
Calling the following interfaces to acquire a message descriptor msg_descriptor, a google:: protobuf:: descriptorpool:: generated_pool () - > FindMessageTypeByName (), and if msg_descriptor is empty, recording error information, wherein the message type symbol is not loaded when the program is compiled; if msg_descriptor is not null, the interface calls text content to acquire and add the text content to the variable protoContents through msg_descriptor- > file () - > DebugString (). Since each interface file may depend on other interface files, msg_descriptor- > file () - > dependency_count (), the number of file dependencies n is obtained, a loop is built based on the dependent file data n, and file_descriptor=msg_descriptor- > file () - > dependency (n). The recursively executing steps in the loop body until the dependent data is 0.
And 4, repeating the steps 2 to 3, and recording all data to be recorded.
Before stopping recording data, defining file head structure body FileHead, continuously arranging FileHead content and protoContents content, and inserting them into the beginning of recording file.
And 6, closing the file descriptor, and finishing data recording.
While the data reading and reinjection part is shown in fig. 10, the following is a specific embodiment:
step 1, opening a binary record file
Step 2, reading the byte stream with the size and the length of the FileHead structure body, filling the byte stream into the FileHead structure body, and reading the protoContentSize variable size
Step 3, reading the protoContentSize size byte stream and filling the byte stream into protoContents variables.
And 4, writing the content in the protoContents into an interface file according to a text coding mode, and creating a google:: protobuf:: compiler:: diskSourceTree object sourceTree. The interface. Proto file is loaded through the MapPath interface of sourceTree. And then loading sourcTree by google:: protobuf:: compiler:: importer, thereby completing the creation of the interface description file.
And 5, reading the byte with the size and the length of the RawDataHead structure body, filling the byte into the RawDataHead structure body, and obtaining the data length size and the message type name.
And 6, reading the byte stream with the size and filling the byte stream into the string.
Step 7, the message file descriptor msg_descriptor, google:: protobuf:: descriptor pool:: generated_pool () - > FindMessageTypeByName () is obtained mainly through the following interfaces.
If the msg_descriptor message file descriptor is empty, recording error information; if the msg_descriptor message file descriptor is not null, the parent pointer prototype of the protobuf variable structure corresponding to the message name can be obtained by google:: protobuf::: messagefactor::: generated_factor () - > GetPrototype (msg_descriptor).
Step 8, creating an empty message copy msg through a new method of a parent pointer protoypte.
And 9, finishing the reverse serialization of the message through msg- > ParseFromString (string).
Step 10, reading and deserializing the binary serialized data can be completed through the steps.
In summary, the embodiment of the application provides a data recording method suitable for high frequency and large data volume, and the hard disk can be directly written from the user space without being cached by the kernel of the operating system under the linux operating system, so that the problem of sporadic jamming of the operating system when the cache of the operating system is too high is solved; meanwhile, based on the serialization, anti-serialization and reflection functions provided by protobuf, a self-explanatory serialization data recording protocol (figure 7) is realized, and corresponding data recording and reinjection methods are provided.
Next, a data recording apparatus according to an embodiment of the present application will be described with reference to the accompanying drawings.
Fig. 11 is a block diagram of a data recording apparatus according to an embodiment of the present application.
As shown in fig. 11, the data recording apparatus 10 includes: an identification module 110, a first update module 120, and a second update module 130.
Wherein, the identification module 110 is used for identifying the actual capacity of the target data to be recorded; the first updating module 120 is configured to write the target data into the preset cache if the actual capacity is smaller than the remaining storage capacity of the preset cache, and update the write cache index of the preset cache; the second updating module 130 is configured to write part of data of the target data into the preset buffer if the actual capacity is greater than or equal to the remaining storage capacity of the preset buffer, write the data of the preset buffer into a record file of the storage device through an operation interface of the operating system when the preset buffer is full, copy the remaining data of the target data to a first address of the preset buffer, and update a write buffer index of the preset buffer, wherein when the record file is created and the data is received for the first time, the kernel buffer of the operating system is prohibited by using the no-buffer identifier.
It should be noted that the foregoing explanation of the embodiment of the data recording method is also applicable to the data recording apparatus of this embodiment, and will not be repeated here.
According to the data recording device provided by the embodiment of the application, the actual capacity of target data to be recorded is identified, different processing is carried out aiming at the actual capacity of different target data, if the actual capacity is smaller than the residual storage capacity of the preset cache, the target data is written into the preset cache, and the written cache index is updated; if the actual capacity is greater than or equal to the residual storage capacity of the preset cache, partial data are written into the preset cache, when the preset cache is full, the cache data are written into a record file of the storage device through an operating system interface, the residual data are copied to a cache head address, and the written cache index of the cache is updated, so that the problem of stuck caused by system cache cleaning during high-frequency large-volume data recording can be effectively solved, and the safety of automatic driving of a vehicle is ensured.
Next, a data reading apparatus according to an embodiment of the present application will be described with reference to the accompanying drawings.
Fig. 12 is a block diagram of a data reading apparatus according to an embodiment of the present application.
As shown in fig. 12, the data reading apparatus 20 includes: an acquisition module 210, a population module 220, and a de-serialization module 230.
The acquiring module 210 is configured to acquire a data reading request; the filling module 220 is configured to open a record file in which serialized data is arbitrarily recorded in the storage device according to the data reading request, read a header of the record file, fill header content corresponding to the header into a corresponding header structure, and fill an interface content length into an interface content variable of the header structure; the deserializing module 230 is configured to generate a data pool based on the filled file header structure, read the data header in the data pool, fill the data content corresponding to the data header into the character string, obtain the data type of the data content, match the preset data template according to the data type name, and deserialize the serialized data by using the preset data template and the character string to obtain the target data.
It should be noted that the foregoing explanation of the embodiment of the data reading method is also applicable to the data reading apparatus of this embodiment, and will not be repeated here.
According to the data reading device provided by the embodiment of the application, by acquiring a data reading request, opening a record file with any record of serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header content corresponding to the file header into a corresponding file header structure, filling interface content length into an interface content variable of the file header structure, generating a data pool based on the filled file header structure, reading a data header in the data pool, filling the corresponding data content into a character string, and performing deserialization on the serialized data by using a set data template and the character string to obtain target data, so that the record file can be interpreted, and because the record is the serialized binary data, the acquisition and loading of the interface are performed during operation, no modification is needed for programs of data record and playback no matter whether any interface is newly added or changed in the later period.
The embodiment of the application also provides data acquisition equipment, which is used for: identifying an actual capacity of target data to be recorded; if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating the written cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing partial data of the target data into the preset cache, when the preset cache is full, writing the data of the preset cache into a record file of the storage device through an operation interface of the operating system, copying the remaining data of the target data to a first address of the preset cache, and updating a write cache index of the preset cache, wherein when the record file is created and data is received for the first time, a no-buffer mark is used for prohibiting kernel cache of the operating system.
The embodiment of the application also provides a data reading device, which is used for: acquiring a data reading request; opening a record file with any recorded serialized data in the storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into corresponding file header structures, and filling interface content lengths into interface content variables of the file header structures; generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into the character string, acquiring the data type of the data content, matching a preset data template according to the name of the data type, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
Fig. 13 is a schematic structural diagram of a vehicle according to an embodiment of the present application. The vehicle 30 may include: a storage device 310 and a data acquisition device 320.
Wherein one or more record files are created in the storage device 310; the data acquisition device 320 is used for identifying the actual capacity of the target data to be recorded; if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating the written cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing partial data of the target data into the preset cache, when the preset cache is full, writing the data of the preset cache into a record file of the storage device through an operation interface of the operating system, copying the remaining data of the target data to a first address of the preset cache, and updating a write cache index of the preset cache, wherein when the record file is created and data is received for the first time, a no-buffer mark is used for prohibiting kernel cache of the operating system.
In the embodiment of the present application, the vehicle 30 further includes: a reading device.
The reading device is used for acquiring a data reading request; opening a record file with any recorded serialized data in the storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into corresponding file header structures, and filling interface content lengths into interface content variables of the file header structures; generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into the character string, acquiring the data type of the data content, matching a preset data template according to the name of the data type, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
The embodiments of the present application also provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data recording method as above, or the data reading method.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or N embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless explicitly defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more N executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present application.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As with the other embodiments, if implemented in hardware, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable gate arrays, field programmable gate arrays, and the like.
Those of ordinary skill in the art will appreciate that all or part of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a computer readable storage medium, where the program when executed includes one or a combination of the steps of the method embodiments.
Although embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives, and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.

Claims (16)

1. A data recording method, comprising the steps of:
identifying an actual capacity of target data to be recorded;
if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating a write cache index of the preset cache;
if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing part of the data of the target data into the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the remaining data of the target data to a head address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a kernel cache of the operating system is forbidden by using a no-buffer mark.
2. The method of claim 1, wherein the head address of the preset buffer is a multiple of the data blocks in the storage device, and the maximum capacity of the buffer is a multiple of the data pages in the storage device.
3. The method according to claim 1, wherein writing the preset cached data to a record file of a storage device through an operation interface of an operating system includes:
identifying whether the file capacity of the recorded file is larger than a preset capacity;
if the file capacity is smaller than or equal to the preset capacity, writing the preset cache into the storage device, and entering a next writing period;
and if the file capacity is larger than the preset capacity, or an abnormal signal is received, calling an abnormal signal processing function to write the preset cached data into the storage device, closing the current record file after writing, generating a new record file, and restarting recording the data under the new record file.
4. A method according to claim 3, wherein said calling an exception signal processing function to write said pre-cached data to said storage device comprises:
Judging whether the current cache byte number of the preset cache is an integer multiple of a data page in the storage device or not;
if the number of the current cache bytes is an integer multiple of the data page, writing the preset cached data into the storage device;
if the current cache byte number is a non-integer multiple of the data page, calculating an excessive writing byte number according to the current cache byte number and a preset multiple of the data page, writing the excessive writing byte number and the current cache byte number into the storage device together, and deleting the excessive writing byte number after writing.
5. The method of claim 1, further comprising, prior to identifying the remaining storage capacity of the predefined preset cache:
defining a data recording interface based on a serialized data recording protocol;
and calling the data recording interface to perform serialization processing on the received target data to obtain serialized data, and writing the serialized data into the preset cache.
6. The method of claim 5, wherein the serialized data recording protocol includes a header structure, header content, a header structure, and header content, the defining a data recording interface based on the serialized data recording protocol comprising:
Reading a configuration file and acquiring a data type name list;
traversing the data type name list to obtain a message descriptor;
if the message descriptor is empty, judging that the record information is wrong, otherwise, acquiring a dependent file, and executing recursion operation on the dependent file until the dependent file is cleared;
initializing the file header structure, writing the interface content length of the file header structure and the file header content into a record file, defining a data header structure when recording each frame of data, carrying out serialization processing on the data by using a protocol buffer area interface, and writing the updated data header content into the record file.
7. The method of claim 5, wherein the serialized data recording protocol includes a header structure, header content, a header structure, and header content, the defining a data recording interface based on the serialized data recording protocol comprising:
creating a data type name list;
defining a data head structure, carrying out serialization processing on data by using a protocol buffer area interface, and writing updated data head content into the record file;
before stopping recording data, defining the file header structure, continuously arranging the interface content length of the file header structure and the file header content, and inserting the file header structure and the file header content into the recording file.
8. The method of claim 1, wherein while disabling kernel caching of the operating system with the unbuffered identification, further comprising:
opening a file descriptor by using an opening function of the operating system;
and opening the record file by using the file descriptor, and creating a target file.
9. A data reading method, comprising the steps of:
acquiring a data reading request;
opening a record file with any recorded serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure;
generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
10. A data recording apparatus, comprising:
the identification module is used for identifying the actual capacity of the target data to be recorded;
the first updating module is used for writing the target data into the preset cache if the actual capacity is smaller than the residual storage capacity of the preset cache, and updating the written cache index of the preset cache;
and the second updating module is used for writing partial data of the target data into the preset cache if the actual capacity is greater than or equal to the residual storage capacity of the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the residual data of the target data to a first address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a no-buffer mark is used for prohibiting kernel cache of the operating system.
11. A data reading apparatus, comprising:
the acquisition module is used for acquiring the data reading request;
the filling module is used for opening a record file which is randomly recorded with the serialized data in the storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure;
And the deserializing module is used for generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and deserializing the serialized data by utilizing the preset data template and the character string to obtain target data.
12. A data acquisition device, the data acquisition device being configured to:
identifying an actual capacity of target data to be recorded;
if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating a write cache index of the preset cache;
if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing part of the data of the target data into the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the remaining data of the target data to a head address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a kernel cache of the operating system is forbidden by using a no-buffer mark.
13. A data reading apparatus, characterized in that the data reading apparatus is configured to:
acquiring a data reading request;
opening a record file with any recorded serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure;
generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
14. A vehicle, characterized by comprising:
a storage device, wherein one or more record files are created in the storage device;
the data acquisition device is used for identifying the actual capacity of target data to be recorded; if the actual capacity is smaller than the residual storage capacity of the preset cache, writing the target data into the preset cache, and updating a write cache index of the preset cache; if the actual capacity is greater than or equal to the remaining storage capacity of the preset cache, writing part of the data of the target data into the preset cache, writing the data of the preset cache into a record file of a storage device through an operation interface of an operating system when the preset cache is full, copying the remaining data of the target data to a head address of the preset cache, and updating a write cache index of the preset cache, wherein when the data is received for the first time after the record file is created, a kernel cache of the operating system is forbidden by using a no-buffer mark.
15. The vehicle of claim 14, further comprising:
a reading device for acquiring a data reading request;
opening a record file with any recorded serialized data in a storage device according to the data reading request, reading a file header of the record file, filling file header contents corresponding to the file header into a corresponding file header structure, and filling interface content length into interface content variables of the file header structure;
generating a data pool based on the filled file header structure, reading the data header in the data pool, filling the data content corresponding to the data header into a character string, acquiring the data type of the data content, matching a preset data template according to the data type name, and performing deserialization on the serialized data by using the preset data template and the character string to obtain target data.
16. A computer-readable storage medium having stored thereon a computer program, characterized in that the program is executed by a processor for realizing the data recording method according to any one of claims 1 to 8 or the data reading method according to claim 9.
CN202310139703.1A 2023-02-20 2023-02-20 Data recording or reading method and device, acquisition equipment, vehicle and medium Pending CN116301602A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310139703.1A CN116301602A (en) 2023-02-20 2023-02-20 Data recording or reading method and device, acquisition equipment, vehicle and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310139703.1A CN116301602A (en) 2023-02-20 2023-02-20 Data recording or reading method and device, acquisition equipment, vehicle and medium

Publications (1)

Publication Number Publication Date
CN116301602A true CN116301602A (en) 2023-06-23

Family

ID=86817822

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310139703.1A Pending CN116301602A (en) 2023-02-20 2023-02-20 Data recording or reading method and device, acquisition equipment, vehicle and medium

Country Status (1)

Country Link
CN (1) CN116301602A (en)

Similar Documents

Publication Publication Date Title
JP3138171B2 (en) How to download system features
EP3816783B1 (en) Method and device for data migration
US9892144B2 (en) Methods for in-place access of serialized data
CN110162306B (en) Advanced compiling method and device of system
JPH07104808B2 (en) Method and apparatus for dynamic volume tracking in an installable file system
US20230185480A1 (en) Ssd-based log data storage method and apparatus, device and medium
JPH0710048B2 (en) Method for detecting and compressing runs of the same continuous data unit
KR20060061204A (en) Computing device with relatively limited storage space and operating/file system thereof
US7711891B1 (en) Method, system, and computer-readable medium for updating memory devices in a computer system
US8589454B2 (en) Computer data file merging based on file metadata
US8196093B2 (en) Apparatus and method for componentizing legacy system
CN111562929A (en) Method, device and equipment for generating patch file and storage medium
US11720422B1 (en) Unified container for hardware and software binaries
US7213245B2 (en) Software on demand system
CN103970512A (en) Multi-core processor and parallel replay method thereof
CN111475150B (en) Cross-language binding method, device, equipment and storage medium
CN112395437B (en) 3D model loading method and device, electronic equipment and storage medium
WO2023169164A1 (en) Method and apparatus for restoring application program, and computer device and storage medium
CN116301602A (en) Data recording or reading method and device, acquisition equipment, vehicle and medium
US6813687B1 (en) Raid volume for sequential use that needs no redundancy pre-initialization
CN110716923B (en) Data processing method, data processing device, node equipment and storage medium
CN110058876B (en) Method, device, server and storage medium for expanding compatibility of wireshark software
US10860533B1 (en) File size as an indicator of file properties
CN113609124B (en) Data record storage method and system with expandability
CN111158858B (en) Cloning method and device of virtual machine and computer readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination