CN111030989B - Flight operation data message analysis system and method - Google Patents
Flight operation data message analysis system and method Download PDFInfo
- Publication number
- CN111030989B CN111030989B CN201911072617.3A CN201911072617A CN111030989B CN 111030989 B CN111030989 B CN 111030989B CN 201911072617 A CN201911072617 A CN 201911072617A CN 111030989 B CN111030989 B CN 111030989B
- Authority
- CN
- China
- Prior art keywords
- message
- data
- module
- analysis
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 146
- 238000000034 method Methods 0.000 title abstract description 24
- 238000012544 monitoring process Methods 0.000 claims abstract description 54
- 238000007781 pre-processing Methods 0.000 claims abstract description 33
- 230000007246 mechanism Effects 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 2
- 238000012795 verification Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 18
- 238000012423 maintenance Methods 0.000 abstract description 4
- 230000010006 flight Effects 0.000 abstract description 2
- 230000014509 gene expression Effects 0.000 description 32
- 238000005192 partition Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 5
- 239000003550 marker Substances 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000005111 flow chemistry technique Methods 0.000 description 3
- 239000000969 carrier Substances 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/40—Business processes related to the transportation industry
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- Computer Networks & Wireless Communication (AREA)
- Economics (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a flight operation data message analysis system, which comprises a message acquisition and identification module, a message analysis module, a message preprocessing and matching module and a message data monitoring and management module; the message analyzing method provided by the invention acquires, identifies and classifies message data, transmits the classified message data to a message analyzing module for analyzing and storing, processes and verifies the data by a message preprocessing matching module, and performs matching operation according to flight schedule information and analyzed flight information to obtain an optimal matching flight; the message data monitoring management module counts and marks data finally, generates a full-text index according to the message data, and realizes full-text retrieval of the message data, so that a unified analysis system is constructed, the message data is collected, classified, analyzed and matched with corresponding flights, the full-text index of the message data is generated, operation and maintenance personnel can manage and maintain the message data conveniently, and the management and monitoring of the flight operation process are realized.
Description
Technical Field
The invention relates to the technical field of flight message analysis, in particular to a flight operation data message analysis system and a flight operation data message analysis method.
Background
Flight operation message data (SITA, AFTN, ACARS, LDM, ADS-B and the like) are transmitted to each airline company by an airline company, an airport, an air traffic control office, a civil aviation office and other units through a network or a satellite. The airline company purchases various message data services according to the needs of the airline company or exchanges data with other organizations to obtain message data. Each airline company develops message analysis and primary processing according to the needs of the airline company, associates the message analysis and the primary processing with the flight, and finally, serves the data as industrial services. Flight message data has extremely high data value: the method has the advantages of influencing resource scheduling, reducing flight operation (delay and safety) risks, reducing operation cost, providing each environment monitoring of flight operation, adjusting production plans in time and improving production efficiency.
However, the existing message analysis can not form a uniform analysis system, the message analysis development is specific to a specific message, the message analysis development cannot be general, the redevelopment is needed for each change of a free-format message and an irregular message, and the efficiency is low; there is no unified management and no data processing warning for message data, when there is a problem in message parsing, it is inconvenient to investigate, and it cannot become a comprehensive solution for message processing in the field of aviation service. In the aspect of architecture, the existing messages have single-point faults, high availability and load balancing capability, the functions of message identification, classification and analysis cannot be dynamically added, the process of message acquisition, identification, classification, analysis, preprocessing, verification and matching to flight full-flow processing management is not available, each business unit repeatedly analyzes the messages, data of each link is inconsistent, a data island is formed, and operation and maintenance personnel are not convenient to manage the message data.
Disclosure of Invention
The invention provides a flight operation data message analysis system, aiming at overcoming the technical defects that the existing message analysis can not form a uniform analysis system, the message identification, classification and analysis functions can not be dynamically added, and the flight data full-flow processing management can not be realized.
The invention also provides a flight operation data message analysis method.
In order to solve the technical problems, the technical scheme of the invention is as follows:
the flight operation data message analysis system comprises a message acquisition module, a message analysis module, a message preprocessing matching module and a message data monitoring management module, wherein:
the message acquisition module is used for acquiring and classifying message data, transmitting the successful acquisition and classification to the message analysis module in a message form, and storing original data;
the message analysis module is used for analyzing message data and storing analysis results, sending an analysis success message to the message preprocessing matching module, and sending an analysis notification message to the message data monitoring management module;
the message preprocessing matching module carries out matching operation according to the information in the flight and the analyzed data to obtain the best flight and sends a matching result message to the message data monitoring management module;
the message data monitoring and management module performs data statistics and marking according to the collected and classified notification messages, the analyzed notification messages and the matching result messages, and generates a full-text index according to the message data to realize full-text retrieval of the message data.
The message acquisition module is deployed by adopting a cluster, performs message data acquisition and classification on a data source by coordinating a plurality of acquisition nodes through a distributed lock mechanism zookeeper, and stores the classified data; and finally, sending a successful acquisition and identification message containing a message data structure to the message analysis module.
In the scheme, in order to improve the concurrency performance of the system and the real-time performance of the system response, the message data identifies various classifications through the message acquisition module, each classification can be processed concurrently, and the classifications are processed sequentially, so that the processing performance is improved, and the data ordering is also ensured to a certain extent.
In the scheme, when the message middleware of the message collection module adopts Kafka, each message subcategory corresponds to one message partition by deploying Kafka clusters, and each message partition is configured with an independent disk under the permission of resources, so that high-speed sequential writing of data is ensured. When the message is consumed, each partition simultaneously directs a consumer to read the message, so that the message reading sequence processing is ensured, even if the adopted message middleware cannot ensure the data consumption sequence, the collected message data contains the timestamp for receiving the message data.
In the above scheme, the message data sources collected by the message collection module are different, and the communication mode protocols are different, so that the message collection module includes two modes of active collection and passive collection. The active acquisition mode supports database JDBC, MQ message, FTP, NFS network file sharing and the like, and the passive acquisition mode supports MQ message and Http request. The message acquisition module is deployed by adopting a cluster, and performs message data acquisition and classification on the data source by coordinating a plurality of acquisition nodes through a distributed lock mechanism zookeeper. The message data classification adopts a tree-shaped hierarchical structure, if the parent class does not identify the expression, the classification is matched by default, and the message data belongs to the leaf node message class until the final leaf node class identification is successful. Because the message data can belong to a plurality of categories simultaneously, a multi-message identification expression is adopted to calculate and identify messages concurrently, if the message data meets the requirement, the message data can belong to the corresponding category message and upload the message data original text to an FTP or a distributed file system, an uploading directory structure is created according to the category hierarchy of the father and son of the message and the time and date of receiving the message data, and finally, a successful acquisition and identification message containing a message data structure is sent to a message analysis module, wherein the message data structure comprises acquisition starting time, finishing time, message type and message tracking ID, and a unique UUID and a url address of the message original text or the stored message of which the message data is less than 1M are generated during acquisition.
In the scheme, when the message acquisition module successfully acquires the message data, a message of successful acquisition and identification notification is sent; and sending a result message to the monitoring management module no matter whether the acquisition and the identification are successful or not.
The message analysis module is provided with a rule engine QLExpress and a MongoDB database; the message analysis module loads a message analysis rule and updates regularly or updates manually when the system is started, when an analysis notification message is received, key value pairs of classified message data are analyzed according to the message analysis rule and a rule engine QLExpress, JSON data generated by the generated key value pairs and received message data are combined, and the JSON data and the received message data are stored in a MongoDB database of a message category set; and finally, adding message analysis starting time and analysis ending time to the received analysis notification message, and sending the message data to the message data monitoring and managing module.
In the scheme, the message analysis module loads a message analysis rule when the system is started, the message analysis rule calculates codes according to QLExpress expressions written by message classification before the system is on-line, the message type is taken as a key, the analysis rule is taken as a value and is stored in a cache, for example, redis, when the system is updated, the cache is updated simultaneously, a background thread is started, the latest analysis rule is loaded at regular time, and simultaneously, when the system is updated on-line or newly added with the message analysis rule, the cache data is updated synchronously.
When the message preprocessing matching module receives the analysis success notification message, reading analysis result data in the message or reading JSON data in the MongoDB database, inquiring actual flight information according to flight taking-off and landing addresses, flight numbers, carriers and calculated flight information contained in the JSON data, adopting range matching according to flight taking-off and landing time value classes as required, and calculating the optimal flight according to information in the flight and the analyzed data; and finally, adding the matching start time, the matching end time and the flight ID in the successfully analyzed message, and sending the message serving as a matching result message to the message data monitoring and management module.
In the above scheme, the message data monitoring and managing module subscribes the information of successful acquisition and failed acquisition sent by the message acquisition module, subscribes the information of successful analysis and failed analysis sent by the message analysis module, and subscribes the information of successful matching and failed matching sent by message preprocessing and matching. And when a message identification success or failure message is received, generating an index according to the original message content of the message so as to provide full-text retrieval of the message. And finally, counting the identification rate, the resolution rate, the matching rate, the efficiency of each stage and an error alarm according to the message tracking ID, the success and the failure of each node and the time information of each node.
The flight operation data message analysis method comprises the following steps:
s1: the message acquisition module acquires and classifies message data, classifies and stores original messages, and transmits the classified message data and analysis notification messages to the message analysis module and the data monitoring management module;
s2: the message analysis module analyzes and stores the message data, sends a matching notification message to the message preprocessing matching module, and sends the analysis notification message to the message data monitoring management module;
s3: the message preprocessing matching module carries out matching operation according to the information in the flight and the analyzed data to obtain the best flight and sends a matching result message to the message data monitoring management module;
s4: the message data monitoring management module carries out data statistics and marking according to the collected identification result message, the analysis result message and the matching result message, and generates a full-text index according to the message data, so as to realize full-text retrieval of the message data.
In step S1, deploying the message collection module by using a cluster, coordinating multiple collection nodes by using a distributed lock mechanism zookeeper to collect and classify message data from the data source, and storing the classified data; and finally, sending an analysis notification message containing the message data structure to a message analysis module.
In the scheme, in order to improve the concurrency performance of the system and the real-time performance of the system response, the message data identifies various classifications through the message acquisition module, each classification can be processed concurrently, and the classifications are processed sequentially, so that the processing performance is improved, and the data ordering is also ensured to a certain extent.
In the scheme, when the message middleware of the message collection module adopts Kafka, each message subcategory corresponds to one message partition by deploying Kafka clusters, and each message partition is configured with an independent disk under the permission of resources, so that high-speed sequential writing of data is ensured. When the message is consumed, only one consumer can read the message in each partition, so that the message reading sequence processing is ensured, even if the adopted message middleware cannot ensure the data consumption sequence, the collected message data contains the timestamp for receiving the message data.
In the above scheme, the message data sources collected by the message collection module are different, and the communication mode protocols are different, so that the message collection module includes two modes of active collection and passive collection. The active acquisition mode supports database JDBC, MQ message, FTP, NFS network file sharing and the like, and the passive acquisition mode supports MQ message and Http request. The message acquisition module is deployed by adopting a cluster, and performs message data acquisition and classification on the data source by coordinating a plurality of acquisition nodes through a distributed lock mechanism zookeeper. The message data classification adopts a tree-shaped hierarchical structure, if the father class does not identify the expression, the classification is matched by default, and the message data belongs to the leaf node message class until the final leaf node class identification is successful. Because the message data can belong to a plurality of categories simultaneously, a multi-message identification expression is adopted to calculate and identify messages concurrently, if the message data meets the requirement, the message data can belong to the corresponding category message and upload the message data original text to an FTP or a distributed file system, an uploading directory structure is created according to the category hierarchy of the father and son of the message and the time and date of receiving the message data, and finally, a successful acquisition and identification message containing a message data structure is sent to a message analysis module, wherein the message data structure comprises acquisition starting time, finishing time, message type and message tracking ID, and a unique UUID and a url address of the message original text or the stored message of which the message data is less than 1M are generated during acquisition.
In the scheme, when the message acquisition module successfully acquires the message data, the message acquisition module sends an acquisition identification success message; and sending a result message to the monitoring management module no matter whether the acquisition and the identification are successful or not.
In step S2, the message parsing module loads a message parsing rule and updates the rule at regular time when the system is started, when receiving a parsing notification message, parses a key value pair of the classified message data according to the message parsing rule and a rule engine QLExpress arranged on the message parsing module, combines JSON data generated by the generated key value pair with the received message data, and stores the JSON data and the received message data in a MongoDB database arranged on the message parsing module and used for storing a message category set; and finally, adding message analysis starting time and analysis ending time to the received analysis notification message, and sending the message analysis starting time and the analysis ending time to a message data monitoring and managing module.
In the scheme, the message analysis module loads a message analysis rule when the system is started, the message analysis rule calculates codes according to QLExpress expressions written by message classification before the system is on-line, the message type is taken as a key, the analysis rule is taken as a value and is stored in a cache, for example, redis, when the system is updated, the cache is updated simultaneously, a background thread is started, the latest analysis rule is loaded at regular time, and simultaneously, when the system is updated on-line or newly added with the message analysis rule, the cache data is updated synchronously.
In step S3, when the message preprocessing and matching module receives the message parsing success notification message, the parsing result data in the message is read, or the JSON data in the MongoDB database is read, the actual flight information is queried according to the flight takeoff and landing address, the flight number, the carrier and the calculated flight information included in the JSON data, the flight takeoff and landing time value class adopts range matching as required, and the optimal flight is calculated according to the information in the flight and the parsed data; and finally, adding the matching start time, the matching end time and the flight ID in the message analysis success notification message, and sending the message serving as a matching result message to a message data monitoring management module.
In the above scheme, the message data monitoring and managing module subscribes the information of successful acquisition and failed acquisition sent by the message acquisition module, subscribes the information of successful analysis and failed analysis sent by the message analysis module, and subscribes the information of successful matching and failed matching sent by message preprocessing and matching. And when a message identification success or failure message is received, generating an index according to the original message content of the message so as to provide full-text retrieval of the message. And finally, counting the identification rate, the resolution rate, the matching rate, the efficiency of each stage and an error alarm according to the message tracking ID, the success and the failure of each node and the time information of each node.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that:
the invention provides a flight operation data message analysis system and a flight operation data message analysis method, wherein a unified analysis system is constructed through a message acquisition module, a message analysis module, a message preprocessing matching module and a message data monitoring management module, and each node is deployed in a distributed mode, so that the acquisition, classification, analysis and matching of message data to corresponding flights are realized, a message data full-text index is generated, the management and maintenance of the message data of operation and maintenance personnel are facilitated, and the flow processing management of the flight data is realized.
Drawings
FIG. 1 is a schematic diagram of the system of the present invention;
FIG. 2 is a schematic flow diagram of the process of the invention;
wherein: 1. a message collection module; 2. a message parsing module; 3. a message preprocessing matching module; 4. and a message data monitoring and managing module.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the patent;
for the purpose of better illustrating the embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product;
it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.
Example 1
As shown in fig. 1, the flight operation data message parsing system includes a message collection module 1, a message parsing module 2, a message preprocessing matching module 3, and a message data monitoring management module 4, wherein:
the message collection module 1 is used for collecting and classifying message data, transmitting the classified message data and analysis notification message to the message analysis module 2, storing an original message and sending a collection and identification result to the message data monitoring and management module 4;
the message analysis module 2 is used for analyzing message data and storing analysis results, sending an analysis success message to the message preprocessing matching module 3, and sending an analysis result message to the message data monitoring management module 4;
the message preprocessing matching module 3 performs matching operation according to the information in the flight and the analyzed data to obtain the best flight and sends a matching result message to the message data monitoring management module 4;
the message data monitoring and management module 4 performs data statistics and marking according to the collected and identified notification message, the analyzed notification message and the matched notification message, and generates a full-text index according to the message data to realize full-text retrieval of the message data.
More specifically, the message acquisition module 1 is deployed by adopting a cluster, performs message data acquisition and classification on data sources by coordinating a plurality of acquisition nodes through a distributed lock mechanism zookeeper, and stores the classified data; and finally, sending an analysis notification message containing a message data structure to the message analysis module 2.
In the specific implementation process, in order to improve the concurrency performance of the system and the real-time performance of the system response, the message data identifies various classifications through the message acquisition module 1, each classification can be processed concurrently and sequentially, and the same classification is processed sequentially, so that the processing performance is improved, and the data order is also ensured to a certain extent.
In a specific implementation process, when the message middleware of the message collection module 1 adopts Kafka, each message sub-category corresponds to one message partition by deploying Kafka clusters, and each message partition is configured with an independent disk under resource permission, so that high-speed sequential writing of data is ensured. When the message is consumed, each partition can only command one consumer to read the message at the same time, so that the message reading sequence processing is ensured, even if the adopted message middleware cannot ensure the data consumption sequence, the collected message data contains the timestamp for receiving the message data.
In the specific implementation process, the message data acquired by the message acquisition module 1 has different communication mode protocols, so that the message acquisition module 1 includes two modes of active acquisition and passive acquisition. The active acquisition mode supports database JDBC, MQ message, FTP, NFS network file sharing and the like, and the passive acquisition mode supports MQ message and Http request. The message acquisition module 1 is deployed by adopting a cluster, and performs message data acquisition and classification on a data source by coordinating a plurality of acquisition nodes through a distributed lock mechanism zookeeper. The message data classification adopts a tree-shaped hierarchical structure, as shown in table 1. And if the parent class does not identify the expression, the class is matched by default until the final leaf node class identification is successful, and the message data belongs to the leaf node message class. Because the message data can belong to a plurality of categories simultaneously, a multi-message identification expression is adopted to calculate and identify messages concurrently, if the message data meets the requirement, the message data can belong to the corresponding category message and upload the message data original text to an FTP or a distributed file system, an uploading directory structure is created according to the category hierarchy of the father and son of the message and the time and date of receiving the message data, and finally, a message containing a message data structure is sent and sent to a message analysis module, wherein the message data structure comprises the collection starting time, the collection ending time, the message type and the message tracking ID, and a unique UUID and a url address of which the message data is smaller than 1M message original text or the stored message are generated during collection.
Table 1: packet classification data structure
In the implementation process, the packet categories are classified into upper and lower levels, and each level can have a plurality of sub-categories. When the top-level category has no PARENT, then PARTENT _ ID may be a null value. When the message is identified, only the identification of the parent category passes through the identification of the reentry sub-category, if the parent category has no identification expression, the identification is successful by default. The message identification expression data structure is specifically shown in table 2:
table 2: message recognition expression data structure
In a specific implementation process, each message class can be associated with a plurality of identification expressions, and the identification expressions are distinguished to belong to different message classes through CATEGORY _ CODE. The METHOD is an identification mode, namely the expression is identified by regular matching or text character position or QLExpress program expression, and an identification mode can be added subsequently according to the requirement. And NON is the expression calculation result, namely, when the calculation result is true, the final result of the expression is false. Wherein the value of LOGIC _ OPERATOR identifier is and, or extended form (and or form, the identifier may overlap more than one, such as: ((and, creating this way is convenient for multiple identifying expressions of message type to combine the logical OPERATOR and priority flexibly, i.e. multiple expressions may be composed:
(a expression or (b expression and c expression and (e expression or f expression)))
The expression storage mode is left bracket markers, each marker, a logic operator and/or and a calculation result-true/false for identifying the expression are respectively a calculation unit, and each calculation unit is respectively stored in a linked list node, for example:
| ( | a… | or | ( | b… | and | c… | and | ( | e… | or | f… | ) | ) | ) |
the above expression generation algorithm:
receiving an operational character, if the operational character starts with a non-a/o, acquiring the number of markers as i, and if no marker exists, setting i as zero;
the number of markers in the linked list is marked as j (wherein, ("the number of markers minus") "the number of markers is the value of j);
if i-j is n, n >0, then add logical operator in the linked list, then add n "(" marker node, if n <0 is less than zero, add n ")" marker node, if n is 0 then add logical operator (and/or) directly. If j >0 after the end, j) are added in the linked list ')' marker nodes;
finally, if the first node of the linked list is and or, deleting the first node;
and finally, evaluating the chain table through a double-stack arithmetic expression evaluation algorithm.
In the specific implementation process, when the message acquisition module successfully acquires the message data, a message of successful acquisition and identification is sent; and sending a result message to the monitoring management module no matter whether the acquisition and the identification are successful or not.
More specifically, a rule engine QLExpress and a MongoDB database are arranged on the message analysis module; the message analysis module loads a message analysis rule and updates the message analysis rule at regular time when the system is started, when a message which is successfully acquired and identified is received, key value pairs of classified message data are analyzed according to the message analysis rule and a rule engine QLExpress, JSON data generated by the generated key value pairs and the received message data are combined and stored in a MongoDB database of a message class set, and the preprocessing matching module is informed; and finally, adding message analysis starting time and analysis ending time to the received acquisition and identification success message, and sending the message to the message data monitoring and management module.
In the specific implementation process, the message analysis module loads a message analysis rule when the system is started, the message analysis rule calculates codes according to QLExpress expressions written by message classification before the system is on-line, the message classification is used as a key, the analysis rule is used as a value to be stored in a cache, if redis, the cache is updated when updating is performed, a background thread is started, the latest analysis rule is loaded at regular time, and the cache data is updated synchronously when the on-line updating or newly adding the message analysis rule is performed.
More specifically, when the message preprocessing matching module receives a successful analysis message, reading analysis result data in the message or reading JSON data in the MongoDB database, inquiring actual flight information according to flight taking-off and landing addresses, flight numbers, carriers and calculated flight information contained in a JSON file, adopting range matching according to flight taking-off and landing time value classes as required, and calculating the optimal flight according to information in the flight and the analyzed data; and finally, adding the matching start time, the matching end time and the flight ID in the matching notification message, and sending the matching start time, the matching end time and the flight ID as a matching result message to the message data monitoring and management module.
In a specific implementation process, the message data monitoring and managing module subscribes the information of successful acquisition and failed acquisition sent by the message acquisition module, subscribes the information of successful analysis and failed analysis sent by the message analysis module, and subscribes the information of successful matching and failed matching sent by the message preprocessing module. And when a message identification success or failure message is received, generating an index according to the original message content of the message so as to provide full-text retrieval of the message. And finally, counting the identification rate, the resolution rate, the matching rate, the efficiency of each stage and an error alarm according to the message tracking ID, the success and the failure of each node and the time information of each node.
Example 2
More specifically, on the basis of embodiment 1, as shown in fig. 2, the flight operation data packet parsing method includes the following steps:
s1: the message acquisition module 1 acquires and classifies message data, transmits the classified message data and the successfully acquired and identified message to the message analysis module 2, stores the original message data, and transmits the acquired and identified result message to the message data monitoring and management module 4;
s2: the message analysis module 2 analyzes and stores the message data, sends the message of successful analysis to the message preprocessing matching module 3, and sends the message analysis result message to the message data monitoring management module 4;
s3: the message preprocessing matching module 3 performs matching operation according to the information in the flight plan and the analyzed data to obtain the best flight and sends a matching result message to the message data monitoring management module 4;
s4: the message data monitoring management module 4 performs data statistics and marking according to the collected and identified notification message, the analyzed notification message and the matched notification message, and generates a full-text index according to the message data, thereby realizing full-text retrieval of the message data.
More specifically, in step S1, the message collection module 1 is deployed by using a cluster, collects and classifies message data from data sources by coordinating multiple collection nodes through a distributed lock mechanism zookeeper, and stores the classified data; and finally, sending the successful acquisition and analysis message containing the message data structure to the message analysis module 2.
In the specific implementation process, in order to improve the concurrency performance of the system and the real-time performance of the system response, the message data identifies various classifications through the message acquisition module 1, each classification can be processed concurrently and sequentially, and the same classification is processed sequentially, so that the processing performance is improved, and the data order is also ensured to a certain extent.
In a specific implementation process, when the message middleware of the message collection module 1 adopts Kafka, each message sub-category corresponds to one message partition by deploying Kafka clusters, and each message partition is configured with an independent disk under resource permission, so that high-speed sequential writing of data is ensured. When the message is consumed, each partition can only command one consumer to read the message at the same time, so that the message reading sequence processing is ensured, even if the adopted message middleware cannot ensure the data consumption sequence, the collected message data contains the timestamp for receiving the message data.
In the specific implementation process, the sources of the message data acquired by the message acquisition module 1 are different, and the communication mode protocols are different, so that the message acquisition module 1 comprises two modes of active acquisition and passive acquisition. The active acquisition mode supports database JDBC, MQ message, FTP, NFS network file sharing and the like, and the passive acquisition mode supports MQ message and Http request. The message acquisition module 1 is deployed by adopting a cluster, and performs message data acquisition and classification on a data source by coordinating a plurality of acquisition nodes through a distributed lock mechanism zookeeper. The message data classification adopts a tree-shaped hierarchical structure, if the parent class does not identify the expression, the classification is matched by default, and the message data belongs to the leaf node message class until the final leaf node class identification is successful. Because the message data can belong to a plurality of categories simultaneously, a multi-message identification expression is adopted to calculate and identify messages concurrently, if the message data meets the requirement, the message data can belong to the corresponding category message and upload the message data original text to an FTP or a distributed file system, an uploading directory structure is created according to the category hierarchy of the father and son of the message and the time and date of receiving the message data, and finally, a successful acquisition and identification message containing a message data structure is sent to a message analysis module, wherein the message data structure comprises acquisition starting time, finishing time, message type and message tracking ID, and a unique UUID and a url address of the message original text or the stored message of which the message data is less than 1M are generated during acquisition.
In the specific implementation process, when the message acquisition module 1 successfully acquires the message data, a message of successful acquisition and identification is sent; and sending a result message to the monitoring management module no matter whether the acquisition and the identification are successful or not.
More specifically, in step S2, the message parsing module 2 loads the message parsing rule and updates the rule at regular time when the system is started, and when a message with successful acquisition and identification is received, parses the key value pair of the classified message data according to the message parsing rule and the rule engine QLExpress arranged on the message parsing module 2, combines the JSON data generated by the generated key value pair with the received message data, and stores the JSON data into the MongoDB database arranged on the message parsing module and used for storing the message category set; and finally, adding message analysis starting time and analysis ending time to the received acquisition and identification success message, and sending the message to the message data monitoring and management module 4.
In a specific implementation process, the message parsing module 2 loads a message parsing rule when the system is started, the message parsing rule calculates codes according to QLExpress expressions compiled according to message classification before a new classification is online, the message classification is used as a key, the parsing rule is used as a value to be stored in a cache, if redis is used, the cache is updated when updating occurs, a background thread is started, the latest parsing rule is loaded at regular time, and meanwhile, cache data is updated synchronously when online updating or newly adding the message parsing rule occurs.
More specifically, in step S3, when the message preprocessing and matching module 3 receives the analysis success message, it reads the analysis result data in the message, or reads the JSON data in the montogdb database, queries the actual flight information according to the flight take-off and landing address, flight number, carrier and estimated flight information included in the JSON data, and calculates the optimal flight according to the information in the flight plan and the analyzed data; and finally, adding the matching start time, the matching end time and the flight ID in the successfully analyzed message, and sending the message serving as a matching result message to the message data monitoring and management module 4.
In a specific implementation process, the message data monitoring and managing module 4 subscribes the information of successful acquisition and failed acquisition sent by the message acquisition module 1, the information of successful analysis and failed analysis sent by the subscribed message analysis module 2, and the subscribed message preprocessing module 3 matches the information of successful matching and failed matching sent. And when a message identification success or failure message is received, generating an index according to the original message content of the message so as to provide full-text retrieval of the message. And finally, counting the identification rate, the resolution rate, the matching rate, the efficiency of each stage and an error alarm according to the message tracking ID, the success and the failure of each node and the time information of each node.
It should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims (2)
1. Flight operation data message analytic system, its characterized in that: the system comprises a message acquisition module (1), a message analysis module (2), a message preprocessing matching module (3) and a message data monitoring management module (4), wherein:
the message acquisition module (1) is used for acquiring, classifying and storing message data, transmitting the classified message data and the information of successful acquisition and identification to the message analysis module (2), and sending the information of success or failure of message acquisition and identification to the message data monitoring and management module (4);
the message analysis module (2) is used for analyzing message data, sending the analyzed data and the analyzed successful message to the message preprocessing matching module (3), and sending the message of whether the analysis is successful or not to the message data monitoring management module (4);
the message preprocessing matching module (3) carries out verification and matching operation according to information in the flight plan and the analyzed data to obtain the best flight and sends a matching result message to the message data monitoring management module (4);
the message data monitoring and management module (4) carries out data statistics and marking according to the message of whether the message acquisition and identification is successful or not, the message of whether the analysis is successful or not and the matching result message, generates a full-text index according to the message data, and realizes full-text retrieval of the message data;
the message acquisition module (1) is deployed by adopting a cluster, performs message data acquisition and classification on data sources by coordinating a plurality of acquisition nodes through a distributed lock mechanism zookeeper, and stores the classified data; finally, sending a successful acquisition and identification message containing a message data structure to the message analysis module (2); finally, whether the message acquisition and identification are successful or not, the acquisition and identification start time and the acquisition and identification end time information are used as messages of whether the message acquisition and identification are successful or not and are sent to the message data monitoring management module (4);
a rule engine QLExpress and a MongoDB database are arranged on the message analysis module (2); the message analysis module (2) loads a message analysis rule and updates regularly or triggers manually to load and update the analysis rule when the system is started, when a message with successful acquisition and identification is received, key value pairs of classified message data are analyzed according to the message analysis rule and a rule engine QLExpress, JSON format data formed by combining the key value pairs generated by successful analysis and the message with successful acquisition and identification is stored in a MongoDB database of a message category set, and meanwhile JSON data are sent to the message preprocessing matching module (3) as a message with successful analysis; finally, message analysis starting time and analysis ending time are added to the received acquisition and identification success message and are sent to the message data monitoring and management module (4) as the message of whether the analysis is successful or not;
when the message preprocessing matching module (3) receives the message of successful analysis, reading the analysis result in the message, wherein the flight taking-off and landing address, the flight number, the carrier and the message generation time contained in the message are used for calculating the flight date, the flight taking-off and landing time value class adopts range matching according to the requirement, the flight information in the flight plan is matched, and the optimal flight is calculated according to the information in the flight and the analyzed data; and finally, adding the matching start time, the matching end time and the flight ID in the successfully analyzed message, and sending the message serving as a matching result message to the message data monitoring and management module (4).
2. The flight operation data message analysis method is characterized by comprising the following steps:
s1: the message acquisition module (1) acquires message data, classifies and stores the message data, transmits the classified message data and the message of successful acquisition and identification to the message analysis module (2), and transmits the message of successful acquisition and identification to the message data monitoring and management module (4);
s2: the message analysis module (2) analyzes the message data and stores the result, and sends the analyzed data and the analyzed successful message to the message preprocessing matching module (3) and sends the message of whether the analysis is successful or not to the message data monitoring management module (4);
s3: the message preprocessing matching module (3) performs matching operation according to the information in the flight and the analyzed data to obtain the best flight and sends a matching result message to the message data monitoring management module (4);
s4: the message data monitoring and management module (4) carries out data statistics and marking according to the message of whether the message acquisition and identification succeeds, the message of whether the analysis succeeds and the matching result message, generates a full-text index according to the message data, and realizes full-text retrieval of the message data;
in the step S1, the message collection module (1) is deployed by using a cluster, collects and classifies message data from data sources by coordinating multiple collection nodes through a distributed lock mechanism zookeeper, and stores the classified data; finally, a message containing a message data structure is sent, if the message is a large message, a message containing a message storage address data structure is successfully acquired and identified to notify the message analysis module (2); finally, whether the message acquisition and identification are successful or not, the acquisition and identification start time and the acquisition and identification end time information are used as messages of whether the message acquisition and identification are successful or not and are sent to the message data monitoring management module (4);
in the step S2, the message parsing module (2) loads the message parsing rule when the system is started and updates the message parsing rule at regular time or triggers the loading parsing rule manually, when a message with successful acquisition and identification is received, key value pairs of the classified message data are parsed according to the message parsing rule and a rule engine QLExpress arranged on the message parsing module (2), and JSON-format data generated by combining the generated key value pairs and other related data are stored in a MongoDB database arranged on the message parsing module (2) and used for storing a message category set, and the JSON data are sent to the message preprocessing matching module (3) as a message with successful parsing; finally, message analysis starting time and analysis ending time are added to the received acquisition and identification success message and are sent to the message data monitoring and management module (4) as the message of whether the analysis is successful or not;
in the step S3, when the message preprocessing matching module (3) receives the message of successful parsing, if the message includes the address of the parsing result, the JSON file in the montogdb database is read, otherwise, the JSON file of the parsing result in the message is read, the flight date is calculated according to the flight take-off and landing address, the flight number, the carrier and the message generation time included in the JSON data, the flight take-off and landing time value class adopts range matching as required, the flight information in the flight plan is matched, and the optimal flight is calculated according to the information in the flight and the parsed data; and finally, adding the matching start time, the matching end time and the flight ID in the successfully analyzed message, and sending the message serving as a matching result message to a message data monitoring and managing module (4).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911072617.3A CN111030989B (en) | 2019-11-05 | 2019-11-05 | Flight operation data message analysis system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911072617.3A CN111030989B (en) | 2019-11-05 | 2019-11-05 | Flight operation data message analysis system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111030989A CN111030989A (en) | 2020-04-17 |
| CN111030989B true CN111030989B (en) | 2021-10-29 |
Family
ID=70200828
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911072617.3A Active CN111030989B (en) | 2019-11-05 | 2019-11-05 | Flight operation data message analysis system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111030989B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111950984B (en) * | 2020-08-13 | 2023-08-04 | 中国民航信息网络股份有限公司 | Method and device for automatically opening departure machine |
| CN112491463B (en) * | 2020-12-01 | 2022-02-25 | 中国商用飞机有限责任公司 | Method for identifying aircraft flight segment based on ACARS message |
| CN113205705B (en) * | 2021-03-26 | 2022-05-27 | 南京莱斯信息技术股份有限公司 | Flight telegram centralized processing system and method based on telegram label |
| CN113450600A (en) * | 2021-06-18 | 2021-09-28 | 成都民航空管科技发展有限公司 | System for joint operation of ATC system and iTWR system |
| CN114866487B (en) * | 2022-03-08 | 2024-03-05 | 国网江苏省电力有限公司南京供电分公司 | Massive power grid dispatching data acquisition and storage system |
| CN116720715B (en) * | 2023-08-08 | 2023-11-28 | 华航信航空科技(浙江)有限公司 | Electronic process single-tube control method applied to program control condition |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105448140A (en) * | 2015-12-30 | 2016-03-30 | 北京招通致晟科技有限公司 | Method and device for acquiring flight dynamic information |
| CN108241686A (en) * | 2016-12-26 | 2018-07-03 | 北京航管科技有限公司 | A kind of data integrating method and system |
| CN108694862A (en) * | 2018-07-19 | 2018-10-23 | 王立泽 | The method and system of blank pipe information automation system critical alarm verbal announcement |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040019509A1 (en) * | 2002-07-23 | 2004-01-29 | Bekkers Ivan H. | System and method for managing flight information |
-
2019
- 2019-11-05 CN CN201911072617.3A patent/CN111030989B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105448140A (en) * | 2015-12-30 | 2016-03-30 | 北京招通致晟科技有限公司 | Method and device for acquiring flight dynamic information |
| CN108241686A (en) * | 2016-12-26 | 2018-07-03 | 北京航管科技有限公司 | A kind of data integrating method and system |
| CN108694862A (en) * | 2018-07-19 | 2018-10-23 | 王立泽 | The method and system of blank pipe information automation system critical alarm verbal announcement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111030989A (en) | 2020-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111030989B (en) | Flight operation data message analysis system and method | |
| US20200026456A1 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| US20200084086A1 (en) | Management of computing system alerts | |
| CN107924406A (en) | Selection is used for the inquiry performed to real-time stream | |
| US10824647B2 (en) | Real-time prediction and explanation of sequences of abnormal events | |
| US10698935B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| CN105791417B (en) | A kind of intelligent deployment and process monitoring system and method based on cloud management platform | |
| CN110912757B (en) | Service monitoring method and server | |
| CN112199394A (en) | Alarm information pushing method and system, intelligent terminal and storage medium | |
| CN103190123A (en) | Method and apparatus for distributing published messages | |
| WO2014065115A1 (en) | Rule distribution server, as well as event processing system, method, and program | |
| CN102937984B (en) | A kind of collect the system of data, client and method | |
| Ge et al. | Adaptive analytic service for real-time internet of things applications | |
| EP3223201A1 (en) | Aircraft message management system | |
| CN110674231A (en) | Data lake-oriented user ID integration method and system | |
| US20120072589A1 (en) | Information Processing Apparatus and Method of Operating the Same | |
| CN113626447A (en) | Civil aviation data management platform and method | |
| JP2013045208A (en) | Data generation method, device and program, retrieval processing method, and device and program | |
| CN109308290A (en) | A kind of efficient data cleaning conversion method based on CIM | |
| CN112784113A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN116166735B (en) | Aviation data processing method and device, electronic equipment and storage medium | |
| CN116011972A (en) | Data processing system, method, electronic equipment and computer storage medium | |
| EP3380906A1 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| JP2016053976A (en) | Data generation method, device and program, retrieval processing method, and device and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Wang Yuzhong Inventor after: Liu Deyong Inventor before: Wang Yuzhong |
|
| CB03 | Change of inventor or designer information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |