CN114547376A - Airport message data intelligent processing method, device and medium based on big data - Google Patents

Airport message data intelligent processing method, device and medium based on big data Download PDF

Info

Publication number
CN114547376A
CN114547376A CN202210150040.9A CN202210150040A CN114547376A CN 114547376 A CN114547376 A CN 114547376A CN 202210150040 A CN202210150040 A CN 202210150040A CN 114547376 A CN114547376 A CN 114547376A
Authority
CN
China
Prior art keywords
data
message
big
messages
structured
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210150040.9A
Other languages
Chinese (zh)
Inventor
潘健
雷刚
伍岭
李向明
王鹏
林于群
孟庆斌
李静雯
吴玥
黄彪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Baiyun International Airport Co ltd
Original Assignee
Guangzhou Baiyun International Airport Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Baiyun International Airport Co ltd filed Critical Guangzhou Baiyun International Airport Co ltd
Priority to CN202210150040.9A priority Critical patent/CN114547376A/en
Publication of CN114547376A publication Critical patent/CN114547376A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
    • G06F16/81—Indexing, e.g. XML tags; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
    • G06F16/84—Mapping; Conversion
    • G06F16/86—Mapping to a database
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Software Systems (AREA)
  • Document Processing Apparatus (AREA)

Abstract

The invention discloses an airport message data intelligent processing method based on big data, which relates to the technical field of airport message data and comprises the steps of collecting message data of multiple data sources, carrying out priority sequencing on different message data of the data sources, storing the message data into a big data cluster, respectively carrying out format conversion on the message data of the multiple data sources in the big data cluster into messages in a text format or an XML format, carrying out data cleaning on the converted messages, analyzing the cleaned messages in the text format or the XML format into structured data, carrying out priority matching on the structured data according to the priorities of different data sources, covering the structured data with low priority by the structured data with high priority and carrying out fusion processing, and carrying out automatic auditing, modifying and confirming on the fused data. The method can solve the problems of single data source and low data processing speed, and has the effects of reducing airport data processing cost and improving accuracy and timeliness.

Description

Airport message data intelligent processing method, device and medium based on big data
Technical Field
The invention relates to the technical field of airport message data, in particular to an airport message data intelligent processing method, device and medium based on big data.
Background
The airport message data is a way for realizing information interaction between an airport and an airline company, is an important information source for realizing flight, passenger and freight guarantee, and is also a main evidence for settlement of both parties. At present, airport message data processing modes mainly include the following modes:
manual recording: for departure flights, most airports take printed airplane load message paper files from a flight loading department and manually input the printed airplane load message paper files into an airport statistical system; for inbound flights, most airports receive airplane payload message paper files from arriving flights and enter airport statistics manually.
And (3) system processing: the data files are downloaded through a single system, then the data files are imported into a statistical system to be matched with flight data of an operation department through program matching, basic data needs to be imported and exported manually, and matching of message data and flight data can be basically achieved.
However, the above method has the following disadvantages:
(1) data source singleness
The existing method only introduces paper cabin single data or a single data source derived from a message system regardless of manual processing or system processing, has single data source, and cannot carry out mutual check on multiple data sources.
(2) Relying on manual treatment
The existing method relies on manual entry or manual data export from a message system and then matches the data with flight basic data, and the method has the problems of long time consumption, high error, untimely data and the like.
(3) Slow data processing speed
When the existing method uses manual entry, the data processing speed is extremely low, a large amount of concurrent real-time data cannot be processed when the system is relied on for processing, and the problems of data accumulation and high server load often exist.
(4) The running condition of the background service cannot be monitored
The existing method does not monitor the running condition of the data processing service, and when the server is down or the data service reports errors under special conditions, managers cannot master the real-time condition.
Disclosure of Invention
The invention aims to provide an airport message data intelligent processing method, device and medium based on big data, which can improve the working efficiency.
The invention relates to an airport message data intelligent processing method based on big data, which comprises the following steps:
s1, collecting multiple data source message data, performing priority ordering on different data source message data, and storing the different data source message data into a big data cluster;
s2, converting the formats of the message data of various data sources into messages in text format or XML format respectively in the big data cluster, and cleaning the converted messages;
s3, analyzing the cleaned message in text format or XML format into structured data;
and S4, carrying out priority matching on the structured data according to the priorities of different data sources, covering the structured data with high priority with the structured data with low priority, carrying out fusion processing in real time, and carrying out automatic auditing, modifying and confirming on the fused data.
The invention also provides an airport message data intelligent processing device based on big data, which comprises:
the multi-data-source message data acquisition module is used for acquiring various data-source message data, performing priority ordering on different data-source message data and storing the data into a big data cluster;
the message data preprocessing module is used for respectively converting the formats of the message data of various data sources into messages in a text format or an XML format in the big data cluster and cleaning the converted messages;
the automatic message analyzing module is used for analyzing the cleaned message in the text format or XML format into structured data;
and the multi-data-source automatic checking module is used for carrying out priority matching on the structured data according to the priorities of different data sources, covering the structured data with high priority with the structured data with low priority and carrying out fusion processing in real time, and carrying out automatic auditing, modifying and confirming on the fused data.
The invention also provides an airport message data intelligent computer processing medium based on big data, which stores computer executable instructions, and when the computer executable instructions are executed by a server, the server executes the method.
The invention adopts multi-data source collection, data cleaning, data priority definition and data automatic matching, realizes real-time automatic intelligent processing of message data, ensures timeliness, accuracy and authority of financial settlement data, and solves the problems of delayed submission of civil aviation funds and delayed provision of financial statements. On the other hand, in the big data cluster, the formats of the message data of various data sources are respectively converted into the messages in the text format or the XML format, and the converted messages are subjected to data cleaning, so that a solid foundation is laid for message analysis, the quality of message analysis is improved, and the time required for message verification is reduced. The method comprises the steps of carrying out priority matching on structured data through priorities of different data sources, enabling the structured data with high priority to cover the structured data with low priority and carry out fusion processing in real time, and automatically auditing, modifying and confirming the fused data, so that the airport message data can be automatically and intelligently processed to the extent that the processing rate is improved to more than 95% from the original blank, the time for manually collecting message data and manually processing the message data is reduced, meanwhile, the printing cost is also reduced, and the effects of improving the working efficiency and improving the real-time application and intelligent decision level are achieved.
Drawings
Fig. 1 is a schematic flow chart of an airport message data intelligent processing method based on big data according to the present invention.
Detailed Description
As shown in fig. 1, the intelligent airport message data processing method based on big data includes the following steps:
s1, collecting multiple data source message data, performing priority ordering on different data source message data, and storing the different data source message data into a big data cluster;
s2, converting the formats of the message data of various data sources into messages in text format or XML format respectively in the big data cluster, and cleaning the converted messages;
s3, analyzing the cleaned message in text format or XML format into structured data;
and S4, carrying out priority matching on the structured data according to the priorities of different data sources, covering the structured data with high priority with the structured data with low priority, carrying out fusion processing, and carrying out automatic auditing, modifying and confirming on the fused data.
Further comprising: and collecting message data of various data sources and monitoring a big data cluster in real time. Due to the fact that multiple data source message data and large data clusters are collected, the data sources have the problems of disconnection, faults and the like, and the server has the unexpected situations of downtime, power failure, hacking and the like.
The collected data source message data comprises flight basic information, SITA messages, medium aviation messages, mail messages, postal aviation and national freight aviation messages, south aviation messages, corridor bridge messages, luggage messages, commercial messages, macroscopic messages, flight quarterly planning messages, civil aviation bureau AFTN messages and air traffic control messages. The format of the message is XML text, TXT text, FTP file, EXCEL file, WEB text, mail text, etc. The priority ordering of the data source message data is as follows:
A. setting the priority of message data based on flight basic information as first;
B. setting the priority of the SITA message data as second;
C. setting the priority of the medium navigation message data as third;
D. setting the data priority of the mail message as the fourth priority;
E. setting the priority of the postal aviation and national freight aviation message data as fifth;
F. setting the priority of the south navigation message data as sixth;
G. setting the priority of the corridor bridge message data as seventh;
H. setting the priority of the luggage message data as eighth;
I. setting the business message data priority as ninth;
J. setting the data priority of the macro message as ten;
K. eleventh the flight quarterly planning message data priority;
l, setting the priority of the AFTN message data of the civil aviation bureau to be twelve;
and M, setting the data priority of the empty pipe message as thirteen.
And subsequently, the priority matching of the structured data can be carried out according to the priorities of the different data sources, the structured data with high priority covers the structured data with low priority and is subjected to fusion processing, and the fused data is automatically audited, modified and confirmed.
The step S2 also includes S2-1, converting the format of the message data of various data sources into the message with text format or XML format in the big data cluster, S2-2, cleaning the converted incomplete message, obviously wrong message and unopenable message, so as to lay a solid foundation for the message analysis, improve the quality of the message analysis and reduce the time needed by the message check. The format conversion mainly comprises the step of converting WEB texts, mail texts, FTP files and EXCEL files into TXT texts, so that the subsequent messages can be conveniently and uniformly analyzed.
Step S3 includes the following steps: s3-1, dividing the cleaned message in text format or XML format into special message and common message; s3-2, for different airlines, different airlines and different flight numbers corresponding to the special message, different lines and different positions of the stowage message represent different data fields respectively and are analyzed into corresponding structured data according to the civil aviation message specification, for example, the stowage message usually places the number of the flight in the first line, and the message data with abnormal format of the A airline company places the 'number' in the second line. And S3-3, uniformly setting rules for the common messages, automatically and intelligently circulating and processing the common messages, and calling a big data cluster to calculate, distribute, process and analyze the big data cluster into corresponding structured data.
Step S3-3 includes the steps of: s3-3-1, classifying the analysis tasks of the common messages, and classifying the tasks which can be parallel into one class and the tasks which can be serial into one class; s3-3-2, grading the data size of the task which can be combined; s3-3-3, designing a parallel big data cluster task distributed processing program; and S3-3-4, performing cluster distributed task processing on large data using the large parallel task data volume, and analyzing the processing on small parallel task data volume into corresponding structured data using a multithread program. Serial tasks are classified into a class that is handled according to a common program.
The airport message data intelligent processing device based on big data provided by the embodiment of the invention comprises: the multi-data-source message data acquisition module is used for acquiring various data-source message data, performing priority ordering on different data-source message data and storing the data into a big data cluster; the message data preprocessing module is used for respectively converting the formats of the message data of various data sources into messages in a text format or an XML format in the big data cluster and cleaning the converted messages; the automatic message analyzing module is used for analyzing the cleaned message in the text format or XML format into structured data; and the multi-data source automatic checking module is used for performing priority matching on the structured data according to the priorities of different data sources, covering the structured data with low priority by the structured data with high priority and performing fusion processing in real time, and automatically auditing, modifying and confirming the fused data.
The embodiment of the invention provides an airport message data intelligent computer processing medium based on big data, wherein the computer processing medium stores computer executable instructions, and when the computer executable instructions are executed by a server, the server executes the method.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.

Claims (7)

1. The airport message data intelligent processing method based on big data is characterized by comprising the following steps:
s1, collecting multiple data source message data, performing priority ordering on different data source message data, and storing the different data source message data into a big data cluster;
s2, converting the formats of the message data of various data sources into messages in text format or XML format respectively in the big data cluster, and cleaning the converted messages;
s3, analyzing the cleaned message in text format or XML format into structured data;
and S4, carrying out priority matching on the structured data according to the priorities of different data sources, covering the structured data with high priority with the structured data with low priority, carrying out fusion processing in real time, and carrying out automatic auditing, modifying and confirming on the fused data.
2. The intelligent big-data-based airport message data processing method of claim 1, wherein said step S2 comprises the steps of:
s2-1, converting the format of the message data of various data sources into the message in text format or XML format in the big data cluster;
and S2-2, cleaning the converted incomplete messages, obviously wrong messages and messages which cannot be opened.
3. The intelligent big-data-based airport message data processing method of claim 1, wherein said step S3 comprises the steps of:
s3-1, dividing the cleaned message in text format or XML format into special message and common message;
s3-2, analyzing different lines and different positions of the stowage message into corresponding structured data respectively representing different data fields according to the civil aviation message specification for different airlines, different airlines and different flight numbers corresponding to the special message;
and S3-3, uniformly setting rules for the common messages, automatically and intelligently circulating and processing the common messages, and calling a big data cluster to calculate, distribute, process and analyze the big data cluster into corresponding structured data.
4. The big data based airport message data intelligent processing method of claim 3, wherein said step S3-3 comprises the following steps:
s3-3-1, classifying the analysis tasks of the common messages, and classifying the tasks which can be parallel into one class and the tasks which can be serial into one class;
s3-3-2, grading the data size of the task which can be combined;
s3-3-3, designing a parallel big data cluster task distributed processing program;
and S3-3-4, performing cluster distributed task processing on large data using the large parallel task data volume, and analyzing the processing on small parallel task data volume into corresponding structured data using a multithread program.
5. The big-data-based airport message data intelligent processing method according to any one of claims 1-4, further comprising: and collecting message data of various data sources and monitoring a big data cluster in real time.
6. The utility model provides an airport message data intelligence processing apparatus based on big data which characterized in that includes:
the multi-data-source message data acquisition module is used for acquiring various data-source message data, performing priority ordering on different data-source message data and storing the data into a big data cluster;
the message data preprocessing module is used for respectively converting the formats of the message data of various data sources into messages in a text format or an XML format in the big data cluster and cleaning the converted messages;
the automatic message analyzing module is used for analyzing the cleaned message in the text format or XML format into structured data;
and the multi-data-source automatic checking module is used for carrying out priority matching on the structured data according to the priorities of different data sources, covering the structured data with high priority with the structured data with low priority and carrying out fusion processing in real time, and carrying out automatic auditing, modifying and confirming on the fused data.
7. An intelligent big-data-based computer processing medium for airport message data, the computer processing medium having stored thereon computer-executable instructions that, when executed by a server, cause the server to perform the method of any one of claims 1-5.
CN202210150040.9A 2022-02-18 2022-02-18 Airport message data intelligent processing method, device and medium based on big data Pending CN114547376A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210150040.9A CN114547376A (en) 2022-02-18 2022-02-18 Airport message data intelligent processing method, device and medium based on big data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210150040.9A CN114547376A (en) 2022-02-18 2022-02-18 Airport message data intelligent processing method, device and medium based on big data

Publications (1)

Publication Number Publication Date
CN114547376A true CN114547376A (en) 2022-05-27

Family

ID=81675144

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210150040.9A Pending CN114547376A (en) 2022-02-18 2022-02-18 Airport message data intelligent processing method, device and medium based on big data

Country Status (1)

Country Link
CN (1) CN114547376A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116089907A (en) * 2023-04-13 2023-05-09 民航成都信息技术有限公司 Fusion method and device of aviation multi-source data, electronic equipment and storage medium
CN117610561A (en) * 2024-01-23 2024-02-27 国网山东省电力公司东营供电公司 Remote supervision learning electric power text audit anomaly identification method and system

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116089907A (en) * 2023-04-13 2023-05-09 民航成都信息技术有限公司 Fusion method and device of aviation multi-source data, electronic equipment and storage medium
CN116089907B (en) * 2023-04-13 2023-06-23 民航成都信息技术有限公司 Fusion method and device of aviation multi-source data, electronic equipment and storage medium
CN117610561A (en) * 2024-01-23 2024-02-27 国网山东省电力公司东营供电公司 Remote supervision learning electric power text audit anomaly identification method and system
CN117610561B (en) * 2024-01-23 2024-04-16 国网山东省电力公司东营供电公司 Remote supervision learning electric power text audit anomaly identification method and system

Similar Documents

Publication Publication Date Title
CN114547376A (en) Airport message data intelligent processing method, device and medium based on big data
CN101689052A (en) Health management of single or multi-platform systems
EP2430399B1 (en) Aircraft dispatch information
CN107563519A (en) Aircraft aperiodicity safeguards schedule system
CN114389359A (en) Intelligent operation and maintenance method of centralized control type relay protection equipment based on cloud edge cooperation
CN110363361A (en) A kind of method and system for predicting variable sliding time based on big data
Rajee Olaganathan et al. Managing safety risks in airline maintenance outsourcing
Wu et al. Improving the IATA delay data coding system for enhanced data analytics
CN111915123A (en) Automatic pre-allocation method for flight full process
CN106022582A (en) Work order control system service automatic analyzing and processing method and system
CN110689147A (en) Airplane fault rapid diagnosis system and method
Efimova et al. Elimination of losses in information services based on approaches of IT-service management
Zhang et al. Optimising the flight turnaround schedules: An improved sliding time windows approach based on MILP and CP models
CN110428167B (en) Method and system for issuing parameter commands of rail transit server
CN113887790A (en) Scheduling optimization method for airport special vehicles
Wang Research on civil aircraft direct maintenance cost analysis and optimization based on data mining
CN112884447A (en) Civil aviation airport data cooperation method and system
CN114139747A (en) AIOps intelligent operation and maintenance system based on artificial intelligence technology
CN111967814A (en) Whole vehicle transportation management system
Li et al. Architecture and Key Technologies for Airline Operational Control System Based on Risk Preprocessing
Kim et al. Requirements for operation procedure and plan for the Korean aviation safety big-data platform based on the case of FAA ASIAS
CN117150420B (en) System and method for removing faults of route operation airplane
Xueqin et al. Research on Chengdu air cargo forecast based on improved ARIMA-GARCH
CN117874700B (en) Flight operation data processing and evaluating system and method and electronic equipment
Qi Research on Large Aircraft Operation Support System in Plateau and Complex Terrain Based on System Engineering

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination