CN112084144A - Universal flight parameter data distributed storage method - Google Patents
Universal flight parameter data distributed storage method Download PDFInfo
- Publication number
- CN112084144A CN112084144A CN202010820218.7A CN202010820218A CN112084144A CN 112084144 A CN112084144 A CN 112084144A CN 202010820218 A CN202010820218 A CN 202010820218A CN 112084144 A CN112084144 A CN 112084144A
- Authority
- CN
- China
- Prior art keywords
- data
- flight parameter
- parameter data
- distributed storage
- storage method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000007619 statistical method Methods 0.000 abstract description 5
- 238000005065 mining Methods 0.000 abstract description 3
- 238000004364 calculation method Methods 0.000 abstract description 2
- 238000012545 processing Methods 0.000 abstract description 2
- 238000007726 management method Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/008—Registering or indicating the working of vehicles communicating information to a remotely located station
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C5/0841—Registering performance data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the disclosure provides a general flight parameter data distributed storage method, and belongs to the technical field of avionics. Specifically, the method comprises the steps of firstly, building a distributed storage basic platform; secondly, the formats of the data acquisition modes are unified; and step three, storing the acquired data in a branch system and a packet column. By the aid of the processing scheme, important flight parameter data can be stored in a distributed mode and backed up in a hot mode, formats of the flight parameter data are unified, time consumption of big data statistical analysis is reduced, big data calculation efficiency is improved, and a data base is laid for deep mining of the flight parameter data.
Description
Technical Field
The disclosure relates to the technical field of avionics, in particular to a general distributed storage method for flight parameter data.
Background
With the rapid development of aviation weaponry, the composition structures of various airplanes are increasingly complex, the task requirements are increasingly rich, and higher requirements are put forward on an airplane acquisition system. On one hand, the number of various measurement and control equipment is increased, the data acquisition range is enlarged, and the real-time acquisition, transmission and processing capabilities of test and measurement data are rapidly improved, so that the recorded data volume is exponentially increased. The new airplane can generate a data file of up to 20GB in one common trip, the traditional single-machine-based data analysis is far insufficient for the current flight parameter data, and the massive data provides new requirements and challenges for the data management of troops and the data application mode; on the other hand, in recent years, aviation accidents frequently occur, and various large airlines and army agencies develop application research of advanced flight data, and put forward the requirements of data statistical analysis based on aviation objective big data, fault diagnosis based on data driving, flight quality evaluation based on data driving, and the like, so that the application of aviation objective data is improved to a brand-new altitude. Therefore, a general flight parameter data storage mode needs to be designed, data in different data acquisition modes need to be compatible, and a data foundation is laid for application of upper-layer massive aviation objective data.
Disclosure of Invention
In view of this, the embodiments of the present disclosure provide a general distributed storage method for flight parameter data, which can effectively solve the problems that flight parameter data stored by a single machine is easy to lose, data structures in different data acquisition modes are inconsistent, and the problem that i/o time consumption of a data file is too high in statistical analysis of mass flight parameter data and mining of a machine learning algorithm.
In order to achieve the above purpose, the present disclosure provides the following technical solutions:
a general flight parameter data distributed storage method comprises the following three steps:
step one, building a distributed storage basic platform;
secondly, the formats of the data acquisition modes are unified;
and step three, storing the acquired data in a branch system and a packet column.
In a preferred embodiment, in the step one, a hadoop big data platform is adopted to construct a basic universal distributed file storage system.
In a preferred embodiment, the second step mainly comprises the unification of different data formats of the packet collection and the point collection.
In a preferred embodiment, an equal time axis manner is adopted for packet acquisition, packet acquisition data is remapped to the equal time axis with the nearest distance according to a data nearest principle and is converted into point acquisition data, and therefore format types of the packet acquisition and the point acquisition are unified.
In a preferred embodiment, in the third step, the acquisition parameters are stored according to a subsystem or a sub-subsystem, and only the required data columns are read, so that the time-consuming i/o operation of a large data system is reduced.
In a preferred embodiment, a file storage mode of the flight parameter data is defined as subsystem or packet set sub-file storage, a specific compressed storage format is a column queue format, and before an application software service at an upper layer reads data, the position of a parameter needs to be calculated, and then the parameter needs to be read.
In a preferred embodiment, a basic universal distributed file storage system is constructed by adopting hadoop hdfs components and yann components, and 6 servers are utilized, wherein 2 servers are respectively used as a NameNode and a Resourcemanager and are respectively used as a storage management main node and a resource scheduling management node.
In a preferred embodiment, 6 servers are used as datanodes, and node multiplexing is performed on 2 servers.
The general flight parameter data distributed storage method can be used for storing important flight parameter data in a distributed mode and performing hot backup, unifying formats of the flight parameter data, reducing time consumption of big data statistical analysis, improving efficiency of big data calculation and laying a data foundation for deep mining of the flight parameter data.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present disclosure, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a basic big data storage platform of the present invention;
FIG. 2 is a storage structure for flight parameter data point acquisition in accordance with the present invention;
FIG. 3 is a storage structure for flight parameter data packet collection according to the present invention.
Detailed Description
The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
The embodiments of the present disclosure are described below with specific examples, and other advantages and effects of the present disclosure will be readily apparent to those skilled in the art from the disclosure in the specification. It is to be understood that the described embodiments are merely illustrative of some, and not restrictive, of the embodiments of the disclosure. The disclosure may be embodied or carried out in various other specific embodiments, and various modifications and changes may be made in the details within the description without departing from the spirit of the disclosure. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the appended claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the disclosure, one skilled in the art should appreciate that one aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. Additionally, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present disclosure, and the drawings only show the components related to the present disclosure rather than the number, shape and size of the components in actual implementation, and the type, amount and ratio of the components in actual implementation may be changed arbitrarily, and the layout of the components may be more complicated.
In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details.
The embodiment of the disclosure provides a general distributed storage method for flight parameter data, referring to diagram 1, a specific underlying file storage system of the invention is based on hdfs (storage) and yann (scheduling) components of a Hadoop ecosphere mature in the field of big data, and utilizes 6 interconnected and intercommunicated servers, wherein 2 servers with better performance are respectively used as a NameNode and a ResourceManager and respectively used as a storage management main node and a resource scheduling management node, all 6 servers are used as datanodes, and node reuse is performed on 2 computers with better performance.
Referring to tables 1 and 2, data acquisition is shown for point acquisition and packet acquisition, respectively.
Table 1:
| parameter 1 | Parameter 2 | …… | Parameter n | |
| time 1 | ||||
| time 2 | ||||
| …… | ||||
| time n |
Table 2:
the invention unifies the data formats of packet collection and point collection. For upper layer data statistical analysis, the data structure stored in the point collection mode has inherent advantages because the point collection collects and stores all parameters at a certain time according to a fixed sampling rate, and because the storage structure of the point collection mode conforms to the table structure of a relational database, the point collection mode is convenient for aggregation analysis by various SQL standard statements on a large data platform and model construction by a related machine learning algorithm. The packet acquisition mode is to acquire a part of parameters at a certain moment, but not all the parameters, and the parameters acquired at each moment may be different, which causes the problems of different sampling rates, different acquisition intervals and different sampling moments of different parameters of the same parameter, and brings problems to subsequent data big data aggregation and big data machine learning algorithm.
The following acquisition parameters paralist [ para1, para2, para3] are assumed, and the acquisition at times t1, t2, t3, and t4 are shown in table 3 below.
Table 3:
the conversion to the dotted format is to shift the actual data bits to the nearest time axis, for example, shift the previous 100.800s collection value to the 100.750 s nearest to it and record, and fill null where there is no data, as shown in table 4 below.
Table 4:
| time(s) | Para1 | Para2 | Para3 |
| …… | …… | …… | …… |
| 100.0 | V | null | V |
| 100.25 | null | V | V |
| 100.50 | V | V | V |
| 100.75 | Null | Null | V |
Referring to fig. 2 and 3, in order to reduce i/o operations of massive large data, the present invention defines a file storage mode of flight parameter data as subsystem or packet set sub-file storage, specifically, a compressed storage format is a column queue format, and before reading data, application software services at an upper layer need to calculate positions of parameters, and then reduce i/o time of the large data under two-pipe conditions when reading the parameters.
The above description is only for the specific embodiments of the present disclosure, but the scope of the present disclosure is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present disclosure should be covered within the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (8)
1. A general flight parameter data distributed storage method is characterized by comprising the following three steps:
step one, building a distributed storage basic platform;
secondly, the formats of the data acquisition modes are unified;
and step three, storing the acquired data in a branch system and a packet column.
2. The distributed storage method for the general flight parameter data as claimed in claim 1, wherein in the first step, a hadoop big data platform is adopted to construct a basic general distributed file storage system.
3. The distributed storage method for the universal flight parameter data as claimed in claim 1, wherein in the second step, the unification of different data formats of packet collection and point collection is mainly included.
4. The distributed storage method for the general flight parameter data as claimed in claim 3, characterized in that an equal time axis manner is adopted for packet collection, the packet collected data is remapped to the equal time axis with the nearest distance according to the data nearest principle and converted into point collected data, and therefore format types of the packet collection and the point collection are unified.
5. The distributed storage method for the general flight parameter data as claimed in claim 1, wherein in the third step, the collected parameters are stored according to a subsystem or a sub-subsystem, only the required data columns are read, and the time-consuming i/o operation of a large data system is reduced.
6. The distributed storage method for the general flight parameter data as claimed in claim 5, wherein the file storage mode of the flight parameter data is defined as subsystem or packet set sub-file storage, the specific compressed storage format is a column partial format, and the position of the parameter needs to be calculated before the application software service at the upper layer reads the data, and then the parameter is read.
7. The distributed storage method for the general flight parameter data according to claim 2, characterized in that a basic general distributed file storage system is constructed by adopting hadoop hdfs component and yann component, and 6 servers are utilized, wherein 2 servers are respectively used as a NameNode and a Resourcemanager, and are respectively used as a storage management main node and a resource scheduling management node.
8. The distributed storage method for the general flight parameter data as claimed in claim 7, wherein 6 servers are used as datanodes, and the multiplexing of the nodes is performed on 2 servers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010820218.7A CN112084144A (en) | 2020-08-14 | 2020-08-14 | Universal flight parameter data distributed storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010820218.7A CN112084144A (en) | 2020-08-14 | 2020-08-14 | Universal flight parameter data distributed storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112084144A true CN112084144A (en) | 2020-12-15 |
Family
ID=73728245
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010820218.7A Pending CN112084144A (en) | 2020-08-14 | 2020-08-14 | Universal flight parameter data distributed storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112084144A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN107784103A (en) * | 2017-10-27 | 2018-03-09 | 北京人大金仓信息技术股份有限公司 | A kind of standard interface of access HDFS distributed memory systems |
| CN110457764A (en) * | 2019-07-17 | 2019-11-15 | 陕西千山航空电子有限责任公司 | A kind of helicopter flight data processing method and data processing platform (DPP) framework |
| CN111400326A (en) * | 2020-02-28 | 2020-07-10 | 深圳市赛为智能股份有限公司 | Smart city data management system and method thereof |
-
2020
- 2020-08-14 CN CN202010820218.7A patent/CN112084144A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN107784103A (en) * | 2017-10-27 | 2018-03-09 | 北京人大金仓信息技术股份有限公司 | A kind of standard interface of access HDFS distributed memory systems |
| CN110457764A (en) * | 2019-07-17 | 2019-11-15 | 陕西千山航空电子有限责任公司 | A kind of helicopter flight data processing method and data processing platform (DPP) framework |
| CN111400326A (en) * | 2020-02-28 | 2020-07-10 | 深圳市赛为智能股份有限公司 | Smart city data management system and method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105069703B (en) | A kind of electrical network mass data management method | |
| CN102955977A (en) | Energy efficiency service method and energy efficiency service platform adopting same on basis of cloud technology | |
| CN105117171A (en) | Energy SCADA massive data distributed processing system and method thereof | |
| CN110928740A (en) | Centralized visualization method and system for operation and maintenance data of cloud computing center | |
| CN107463706B (en) | Hadoop-based mass wave recording data storage and analysis method and system | |
| CN108804602A (en) | A kind of distributed spatial data storage computational methods based on SPARK | |
| CN105912601A (en) | Partition storage method for distributed real-time memory database of energy management system | |
| CN104778225A (en) | Method for synchronizing data in unstructured data multi-storage system | |
| Chen et al. | A big data analysis and application platform for civil aircraft health management | |
| CN108009265B (en) | Spatial data indexing method in cloud computing environment | |
| CN103678550A (en) | Mass data real-time query method based on dynamic index structure | |
| CN107798062A (en) | A kind of transformer station's historical data unifies storage method and system | |
| CN105577423A (en) | Real-time data center cluster management system | |
| CN112148578A (en) | IT fault defect prediction method based on machine learning | |
| CN112633632A (en) | Integrated short-term wind power cluster power prediction method based on signal decomposition technology | |
| CN117520470A (en) | Meteorological data management method and device based on GeoSOT grid coding and electronic equipment | |
| CN112084144A (en) | Universal flight parameter data distributed storage method | |
| CN106611012A (en) | Heterogeneous data real-time search method in big data environment | |
| CN110825744B (en) | Cluster environment-based air quality monitoring big data partition storage method | |
| CN102855278B (en) | A kind of emulation mode and system | |
| CN116523328A (en) | Intelligent decision-making method for cooperation of aviation equipment and manufacturing industry chain | |
| CN111127186A (en) | Application method of customer credit rating evaluation system based on big data technology | |
| CN104090895B (en) | Obtain the method for radix, device, server and system | |
| CN110377757A (en) | A kind of real time knowledge map construction system | |
| CN105677853A (en) | Data storage method and device based on big data technology framework |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201215 |