CN114168535A - Data processing and accessing method based on index cache - Google Patents

Data processing and accessing method based on index cache Download PDF

Info

Publication number
CN114168535A
CN114168535A CN202111314877.4A CN202111314877A CN114168535A CN 114168535 A CN114168535 A CN 114168535A CN 202111314877 A CN202111314877 A CN 202111314877A CN 114168535 A CN114168535 A CN 114168535A
Authority
CN
China
Prior art keywords
data
file
channel
parameter
index
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111314877.4A
Other languages
Chinese (zh)
Inventor
王德远
李申
常婉君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shaanxi Qianshan Avionics Co Ltd
Original Assignee
Shaanxi Qianshan Avionics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shaanxi Qianshan Avionics Co Ltd filed Critical Shaanxi Qianshan Avionics Co Ltd
Priority to CN202111314877.4A priority Critical patent/CN114168535A/en
Publication of CN114168535A publication Critical patent/CN114168535A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/172—Caching, prefetching or hoarding of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0661—Format or protocol conversion arrangements

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing For Digital Recording And Reproducing (AREA)

Abstract

The application provides a data processing and accessing method based on index cache, which belongs to the technical field of avionics and specifically comprises the steps of splitting a data packet in a flight parameter data stream according to a channel number to form a plurality of channel data files; writing a parameter configuration file and a channel data file index, and writing the channel data file according to the sequence of channel numbers to form a final processing file; reading a final processing file during data playback, acquiring parameter information, reading the final processing file according to the position of the final processing file recorded in the parameter information, analyzing parameters according to parameter configuration information in the parameter configuration file, and quickly reading data by combining the relative position of each channel and an index during reading; generating a cache file, wherein the cache file is used for storing the analyzed data; and reading the cache file when the data is played back again, and if the cache file does not exist, switching to the step of reading the final processing file when the data is played back. By the aid of the processing scheme, data processing speed is increased, and data access speed is increased.

Description

Data processing and accessing method based on index cache
Technical Field
The application relates to the technical field of avionics, in particular to a multimedia stream data synchronous playback system and method based on data fusion.
Background
The flight parameter data are collected and recorded by a flight parameter recording system, and have irreplaceable important roles in aviation accident investigation. In recent years, with the improvement of the number of acquired parameters, the sampling rate and the change of the data acquisition types, the data volume of airborne aviation data becomes large, the number of channels reaches hundreds due to the packet acquisition structure, and the influence of the reading and writing of a large number of small files on the data processing speed is increased. In order to rapidly process and access large-capacity multi-channel data, data processing methods and access methods need to be improved to improve data processing and access efficiency.
Disclosure of Invention
In view of this, embodiments of the present application provide a data processing and accessing method based on index cache, which processes and collates original data to generate a data file format convenient for access, so as to facilitate quick access to data during data playback.
The embodiment of the application provides a data processing and accessing method based on index cache, which comprises the following steps:
the method comprises the following steps that firstly, data packets in a flight parameter data stream are split according to channel numbers to form a plurality of channel data files;
writing a parameter configuration file and a channel data file index, and writing the channel data file according to the sequence of the channel numbers to form a final processing file;
reading the final processing file during data playback, acquiring parameter information, reading the final processing file according to the position of the final processing file recorded in the parameter information, analyzing parameters according to parameter configuration information in the parameter configuration file, and quickly reading data by combining the relative position and the index of each channel during reading;
generating a cache file, wherein the cache file is used for storing the analyzed data;
and step five, reading the cache file when the data playback is carried out again, and switching to the step three if the cache file does not exist.
According to a specific implementation manner of the embodiment of the application, when the data packet is split, each packet data only includes the total length of the current data packet, the relative timestamp and the original code data, and is stored in the channel data file corresponding to the corresponding data packet.
According to a specific implementation manner of the embodiment of the present application, generating the final processing file specifically includes: and sequentially writing index values of all parts of files, parameter configuration files, channel data file time indexes and index positions of all data packets in the files in the final processing file, and finally sequentially writing the channel data files according to the channel numbers.
According to a specific implementation manner of the embodiment of the present application, the index values of each part of the file in the final processing file include a parameter configuration file position, a channel data file time index position, and a data start position.
According to a specific implementation manner of the embodiment of the application, the cache file is stored in a folder with the same name as the directory of the analyzed file, and is generated by taking a resume name + a channel number + a parameter index number as a name, wherein the resume name is the name of the final processing file.
According to a specific implementation manner of the embodiment of the application, in the fifth step, if the cache file does not exist, the third step is performed, and the cache file is deleted first when the data is preprocessed again.
According to a specific implementation manner of the embodiment of the present application, the parameter configuration file includes: parameter name, system to which the parameter belongs, channel to which the parameter belongs, parameter position and parameter analysis formula information.
Advantageous effects
The data processing and accessing method based on the index cache in the embodiment of the application is designed for multi-channel data, and provides powerful technical support for data playback. The main beneficial effects are as follows:
1) speeding up data processing
The invention reads and writes the multi-channel data in sequence, and can effectively improve the processing efficiency.
2) Increasing data access speed
The data structure formed by the invention can improve the speed of repeatedly checking the data.
3) Improving user experience
By adopting the method, the processing and playback speed of the multi-channel data can be obviously improved, and the user experience is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flowchart of a data processing and accessing method based on index caching according to an embodiment of the present invention.
Detailed Description
The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
The following description of the embodiments of the present application is provided by way of specific examples, and other advantages and effects of the present application will be readily apparent to those skilled in the art from the disclosure herein. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. The present application is capable of other and different embodiments and its several details are capable of modifications and/or changes in various respects, all without departing from the spirit of the present application. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It is noted that various aspects of the embodiments are described below within the scope of the appended claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the present application, one skilled in the art should appreciate that one aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. Additionally, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present application, and the drawings only show the components related to the present application rather than the number, shape and size of the components in actual implementation, and the type, amount and ratio of the components in actual implementation may be changed arbitrarily, and the layout of the components may be more complicated.
In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details.
The embodiment of the application provides a data processing and accessing method based on index cache, and provides a data file format which is convenient for access and is generated by processing and sorting original data, so as to facilitate quick access to data during data playback, and specific method steps refer to fig. 1.
The method mainly comprises the following steps:
step one, splitting a data packet in the flight parameter data stream according to a channel number to form a plurality of channel data files. The original flight parameter data is split according to the type of the data packet, the data packet corresponding to each channel corresponds to one channel data file, each packet of data only comprises the total length of the current data packet, a relative timestamp and original code data during splitting, and channel data file indexes can be recorded every ten minutes.
Specifically, the data packets are classified according to the channel number marks in the packet headers of the data packets in the data streams, the data of the data packets in different channels are arranged according to the time sequence (namely the recorded data stream sequence), and the packet structure reserves the total length, the relative timestamp and the original data and stores the total length, the relative timestamp and the original data into temporary data files corresponding to the corresponding data packets, so that the original data are split into a plurality of channel data files according to the data type packets.
Writing a parameter configuration file and a channel data file index, and writing the channel data file according to the sequence of the channel numbers to form the final processing file.
Specifically, the generating of the final processing file is mainly writing index values of each part of files, each parameter configuration file, a channel data file time index, and an index position of each data packet in the file in the final processing file according to a sequence, and finally writing the channel data file generated in the first step according to the configuration sequence. The index values of all parts of files in the final processing file comprise the position of a parameter configuration file, the time index position of a channel data file and the data start position; the parameter configuration file mainly includes parameter names, parameter belonging systems, parameter belonging channels, parameter positions and parameter analysis formula information, and of course, the contents of the parameter configuration file can be adjusted according to actual requirements.
Reading the final processing file during data playback, acquiring parameter information, reading the final processing file according to the position of the final processing file recorded in the parameter information, analyzing parameters according to parameter configuration information in the parameter configuration file, and quickly reading data by combining the relative position and the index of each channel during reading. Specifically, parameter information is obtained through reading the parameter configuration part, corresponding channel information is obtained in the parameter information, the position of a final processing file where a channel corresponding to the parameter is located is determined through reading the data starting position and the channel index position, the position where the channel data is located in the specified time is accurately obtained through the channel time index, the parameter source code value is obtained through the parameter position information in the parameter information, and the parameter is analyzed through the parameter analysis formula information.
Specifically, the information such as parameter configuration and index is read according to the sequence of writing the file in the step two. And classifying the parameters selected by the user, wherein the data belonging to the same channel belongs to a class, reading and analyzing the original code data according to the class sequence (namely the channel sequence) and the index information, and inquiring and reading according to the index information when randomly reading. In the step, the multichannel data are sequentially read and written, so that the processing efficiency can be effectively improved.
And step four, generating a cache file, wherein the cache file is used for storing the analyzed data.
Specifically, a cache file is generated after data reading and analysis are completed, the cache file is stored in a file folder with the same name as the directory of the analyzed file, a resume name, a channel number and a parameter index number are used as names, the physical quantity after analysis is written, and the resume name is the name of the final processing file.
And step five, reading the cache file when the data playback is carried out again, and switching to the step three if the cache file does not exist. When the data is played back, the played back data is recorded in the form of a cache file, so that the time loss caused by multiple times of reading and writing can be reduced.
And when the data is played back again, whether the cached file exists needs to be checked, if the cached file exists, the cached file is read, and if the cached file does not exist, the step three is carried out for reading. When the data is preprocessed again, the cache file is deleted at first, so that the influence on reading is avoided.
The method splits the original flight parameter data according to the type of the data packet, and records the time index information of the channel data file. The configuration information, the index information and the channel data file are written into the file, so that the excessively low processing speed caused by reading and writing a large amount of small files is avoided. When the data is played back, the played back data is recorded in the form of a cache file, so that the time loss caused by multiple times of reading and writing is reduced.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (7)

1. A data processing and accessing method based on index cache is characterized by comprising the following steps:
the method comprises the following steps that firstly, data packets in a flight parameter data stream are split according to channel numbers to form a plurality of channel data files;
writing a parameter configuration file and a channel data file index, and writing the channel data file according to the sequence of the channel numbers to form a final processing file;
reading the final processing file during data playback, acquiring parameter information, reading the final processing file according to the position of the final processing file recorded in the parameter information, analyzing parameters according to parameter configuration information in the parameter configuration file, and quickly reading data by combining the relative position and the index of each channel during reading;
generating a cache file, wherein the cache file is used for storing the analyzed data;
and step five, reading the cache file when the data playback is carried out again, and switching to the step three if the cache file does not exist.
2. The method as claimed in claim 1, wherein when the data packets are divided, each packet data only contains the total length of the current data packet, the relative timestamp and the original code data, and is stored in the channel data file corresponding to the corresponding data packet.
3. The index cache-based data processing and accessing method of claim 1, wherein generating the final processing file specifically comprises: and sequentially writing index values of all parts of files, parameter configuration files, channel data file time indexes and index positions of all data packets in the files in the final processing file, and finally sequentially writing the channel data files according to the channel numbers.
4. The index cache-based data processing and accessing method of claim 3, wherein the index values of each part of the final processed file comprise a parameter configuration file position, a channel data file time index position and a data start position.
5. The method according to claim 1, wherein the cached files are stored in a folder with the same name as the directory of the parsed files, and are generated by using a resume name + a channel number + a parameter index number as a name, and the resume name is the name of the final processed file.
6. The index cache-based data processing and accessing method according to claim 1, wherein in the fifth step, if the cache file does not exist, the third step is performed, and when the data is preprocessed again, the cache file is deleted first.
7. The index cache based data processing and accessing method of any one of claims 1-6, wherein the parameter configuration file comprises: parameter name, system to which the parameter belongs, channel to which the parameter belongs, parameter position and parameter analysis formula information.
CN202111314877.4A 2021-11-08 2021-11-08 Data processing and accessing method based on index cache Pending CN114168535A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111314877.4A CN114168535A (en) 2021-11-08 2021-11-08 Data processing and accessing method based on index cache

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111314877.4A CN114168535A (en) 2021-11-08 2021-11-08 Data processing and accessing method based on index cache

Publications (1)

Publication Number Publication Date
CN114168535A true CN114168535A (en) 2022-03-11

Family

ID=80478180

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111314877.4A Pending CN114168535A (en) 2021-11-08 2021-11-08 Data processing and accessing method based on index cache

Country Status (1)

Country Link
CN (1) CN114168535A (en)

Similar Documents

Publication Publication Date Title
US8312051B2 (en) Incremental construction of search tree with signature pointers for identification of multimedia content
US9619512B2 (en) Memory searching system and method, real-time searching system and method, and computer storage medium
CN111209344A (en) Data synchronization method and device
US9570109B2 (en) Method and system for matching audio and video
CN104199888B (en) The data reconstruction method and device of Resilient file system
CA2905654C (en) Methods and systems for arranging and searching a database of media content recordings
CN104185088B (en) A kind of method for processing video frequency and device
CN107391769B (en) Index query method and device
DE112012006132T5 (en) Information processing apparatus and information processing method
CN107895391B (en) Flight parameter data packet processing method
CN114979314A (en) Automatic analysis method and system for automobile CAN data, computer readable storage medium and electronic equipment
CN114398217A (en) Massive simulation waveform data slicing method for FPGA parallel simulation
CN114168535A (en) Data processing and accessing method based on index cache
CN103530311A (en) Method and apparatus for prioritizing metadata
CN106791534B (en) Multichannel synchronous storage and playback method of cockpit audio and video recorder
CN109299345B (en) Project preview system and method for multi-channel human-computer environment test data synchronization platform
CN111045994A (en) KV database-based file classification retrieval method and system
CN113852857B (en) Sound playing method and system based on HTML5 video frame stepping
CN110427282A (en) The method, apparatus and computer-readable medium restored for log fragment
CN113507625B (en) Self-adaptive video restoration method
CN112131182A (en) Rapid alignment processing method for packet mining type flight parameter data
CN109633740A (en) Tape SEG-D Seismic data structure analysis method and solution compile device
CN111581942A (en) Data file comparison method
CN112752165A (en) Subtitle processing method, subtitle processing device, server and computer-readable storage medium
CN116933143B (en) Flight parameter data classification method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination