KR102631326B1 - System and Method for controlling airport using recognition technology - Google Patents

Abstract

A control system is being launched that can contribute to improving situational awareness of air traffic controllers by analyzing voice communication between pilots and controllers in real time to verify the implementation of aircraft control instructions and establish a warning system in apron control situations. The control system includes a sensor system that generates sensing information by sensing moving objects including aircraft in the apron, a management server that obtains flight information about the aircraft from a plurality of monitoring equipment, and the sensing information and the flight information. It is characterized by including a control tower server that recognizes verification elements regarding the aircraft situation at the apron using image recognition and analyzes voice recognition of control instructions communicated between the controller and the user of the moving object.

Description

Airport control system and method using recognition technology {System and Method for controlling airport using recognition technology}

The present invention relates to smart airport control support technology, and more specifically, to an airport control system and method that can maximize a controller's forward focus and/or situational awareness using recognition technology.

Separation of aircraft is performed by the control tower (airport control service), approach control center (approach control service), and regional control office (regional control service) to prevent collisions between aircraft, prevent collisions between aircraft and obstacles, and promote air traffic flow. , air traffic control tasks such as radar guidance and air traffic control permit issuance are performed.

In particular, among the airfield control services, there is air traffic control service provided to aircraft moving within the apron area. An apron control tower is formed to manage the apron, and within the apron, tasks such as engine start and rear towing of aircraft, permission to move aircraft, transfer of control authority to the main control tower, de-icing and anti-icing support, and towing control and support are performed.

However, due to the continuous expansion of the airport, the average surveillance area per air traffic controller is the size of about 180 soccer fields, making it physically difficult to monitor the movements of all aircraft in the entire area in real time.

In addition, there is no warning system in place to automatically notify in the event of abnormal situations such as collisions between aircraft, collisions between aircraft and vehicles, incorrect entry of aircraft into taxiways, and violation of air traffic control instructions, so only the visual observation of the air traffic controller and the pilot's report are required. There is a problem that the notification system for abnormal situations is insufficient due to dependence.

In addition, in accordance with ICAO (International Civil Aviation Organization) regulations, visual observation must be prioritized for aircraft, vehicles, personnel, and aircraft flying around the airport. However, shielded areas by buildings, etc. occur throughout the surveillance area, and physical distance is required. The disadvantage is that the limitations of visual observation are clear.

In addition, there is a disadvantage that visual observation of the controller is difficult due to various meteorological phenomena such as fog, rain, snowfall, and fine dust when performing the controller's duties.

In addition, a large number of data required to perform control tasks have text or various shapes and colors, and these data are displayed in a top-down view on a secondary plane on multiple displays and are based on the controller's gaze direction. It is expressed in a direction that is different from .

In addition, the controller's external view of the control tower collects visual information in a three-dimensional Bird's-Eye View, and due to the difference between these two views, the controller must perform control tasks under constant orientation and dimensional cognitive dissonance (Orientational Difference and Dimensional Difference). do. This has a negative impact on the controller's situational awareness and at the same time leads to an increase in the controller's workload, which ultimately has a significant impact on the efficiency of air traffic control work.

In addition, when performing the controller's control duties, various data required for obtaining flight data and monitoring aircraft movements are provided through an average of more than 7 displays, and a similar number of input devices (mouse and keyboard) must be used to operate them. Therefore, during the data acquisition and manipulation process, head-down time is inevitable due to the installation location of the equipment, which affects the controller's visual surveillance ability and causes distraction.

In addition, in the case of the existing control system, control and input are possible with a keyboard and mouse, but this has the disadvantage of causing the controller's head-down, impeding situational awareness, and causing human error due to multiple control devices.

1. Korean Patent No. 10-2063158 (registration date: December 31, 2019)

In order to achieve the tasks presented above, the present invention is an airport control system that analyzes voice communication between pilots and controllers in real time and establishes a verification and warning system for the implementation of aircraft control instructions in an apron control situation, thereby contributing to improving the controller's situational awareness. The purpose is to provide methods and methods.

In addition, the present invention analyzes panoramic view, gate view, and around view images based on computer vision image recognition technology and then applies various algorithms to build a warning system to detect abnormalities in the area under the control of the controller. Another purpose is to provide airport control systems and methods that can contribute to improving response capabilities to situations.

In addition, the present invention utilizes computer vision image recognition technology based on data obtained through ultra-high-resolution video shooting to recognize and track a number of moving objects, including aircraft, and provide the image-based analyzed data to the controller in various display methods. Another purpose is to provide a capable airport control system and method.

In addition, the present invention uses synthetically processed Gate View images to display various information within the gate (aircraft, tow truck, tug car, Dali, ULD (Unit Load Device), GPU (Ground Power Unit) based on computer vision image recognition technology. , cargo, water trucks, sewage trucks, catering trucks, garbage trucks, cargo loaders, ladders, step cars, etc.) are combined with the relevant flight operation data to analyze and learn to predict the operation situation and use this as A-CDM. Another purpose is to provide an airport control system and method that can contribute to achieving punctuality in connection with (Airport Collaborative Decision Making).

In order to achieve the tasks presented above, the present invention provides a control system that can contribute to improving the controller's situational awareness by analyzing voice communication between pilots and controllers in real time and establishing a verification and warning system for the implementation of aircraft control instructions in an apron control situation. do.

The control system is,

A sensor system that generates sensing information by sensing moving objects, including aircraft, within the apron;

a management server that obtains flight information about the aircraft from a plurality of monitoring equipment; and

A control tower server that uses the sensing information and the flight information to recognize verification elements regarding the aircraft situation at the apron by image recognition and to analyze voice recognition for control instructions communicated between the controller and the user of the moving object. It is characterized by

At this time, the control tower server is characterized in that it provides a warning message to the control tower according to the analyzed image recognition and voice recognition results.

In addition, the control tower server determines whether the sensing information collected according to the movement of the moving object matches the execution of the control instruction, and outputs the warning message according to the determination result.

In addition, the analysis of the control instructions is characterized by converting the control instructions recorded as voice recordings into text and extracting the commands of the control instructions from the text.

Additionally, the control tower server is characterized in that it outputs the aircraft location according to the content of voice communication between the controller and the user of the moving object.

Additionally, the control tower server simultaneously outputs the communication content and the aircraft location to the view information.

In addition, the control tower server is characterized by simultaneously outputting the aircraft location, aircraft status, operation stage, special information, and reason for go-around according to analysis of the communication content.

In addition, the control tower server is characterized in that when the aircraft enters the gate zone above a preset value, it is determined to be a gate entry situation and outputs the result.

In addition, the control tower server determines whether there is a discrepancy by comparing the flight information including the flight number, registration number, and arrival gate of the aircraft with the image recognition information of the aircraft entering the apron area, and reports the determination result to the above. It is characterized by displaying it in view information.

In addition, the control tower server is characterized in that it predicts the operation stage of the aircraft through video analysis at the end of movement of the aircraft after the apron entry situation is completed.

On the other hand, another embodiment of the present invention includes (a) a sensor system generating sensing information by sensing a moving object including an aircraft in an apron; (b) a management server acquiring flight information about the aircraft from a plurality of monitoring devices; and (c) the control tower server uses the sensing information and the flight information to recognize verification elements regarding the aircraft situation at the apron by image recognition and analyze voice recognition for control instructions communicated between the controller and the user of the moving object. Provides an airport control method using recognition technology, comprising the step of:

According to the present invention, control of the control area in charge can be strengthened by improving the situational awareness of the apron controller using computer vision image recognition technology, and various warning and notification functions can be used to prevent human errors and improve the ability to respond to abnormal situations. Additionally, it is possible to prevent aircraft from entering taxiways incorrectly, which frequently occurs due to the expansion of airports.

In addition, another effect of the present invention is that voice recognition technology can be used to confirm compliance with control instructions and improve abnormal situation recognition and response capabilities through warning functions.

In addition, another effect of the present invention is that the apron controller's situational awareness and ability to respond to abnormal situations increased upon introduction of the system leads to an increase in control throughput per person, which can be expected to reduce overall delays and improve airport operation capabilities. can be mentioned.

In addition, another effect of the present invention is that it is possible to monitor the digital situation of the area in charge by analyzing data in the area in charge through analysis of acquired images and analyzing voice communication between pilots and controllers, and based on this, it is possible to predict the situation of future operations. It can be said that it is possible.

In addition, another effect of the present invention is that it is possible to secure a standard that can be developed into a standard in the future by securing computer vision reinforcement learning and algorithms dedicated to airfields.

In addition, another effect of the present invention is that it is possible to operate a safe and efficient state-of-the-art control system by applying the system when promoting the construction of a new domestic or overseas airport.

In addition, another effect of the present invention is that the system can be modified and applied to ice houses or similar control tasks that comprehensively control aircraft de-icing and anti-icing operations, and it is possible to construct an inexpensive, safe, and efficient system. .

In addition, another effect of the present invention is to apply motion control and voice control using image recognition technology in addition to existing control devices, so that the main control system functions with specific motion and voice while the controller maintains head-up. It can be improved by inputting, modifying, and controlling.

Figure 1 is a block diagram of the configuration of a smart airport control system according to an embodiment of the present invention.
FIG. 2 is a detailed configuration diagram of the management server shown in FIG. 1.
Figure 3 is a detailed configuration diagram of the control tower server shown in Figure 1.
Figure 4 is a detailed configuration diagram of the processing unit shown in Figure 3.
Figure 5 shows control instruction implementation verification and warning through analysis of voice communication between pilot and controller according to an embodiment of the present invention. This is a flowchart showing the process.
Figure 6 is a concept of aircraft location and operation stage analysis through voice communication analysis between pilots and controllers according to an embodiment of the present invention.
Figure 7 is a diagram showing a situation of entering the parking lot according to an embodiment of the present invention.
Figure 8 is a diagram showing a pushback situation after entry according to Figure 7.
Figure 9 is a diagram showing the pushback preparation state according to Figure 8.
FIG. 10 is a diagram showing a pushback impossible situation according to FIG. 8.
Figure 11 is a conceptual diagram of aircraft rear towing restrictions identification and warning according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. These embodiments only serve to ensure that the disclosure of the present invention is complete, and those skilled in the art It is provided to fully inform the person of the scope of the invention, and the present invention is only defined by the scope of the claims. The sizes and relative sizes of components shown in the drawings may be exaggerated for clarity of explanation.

Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and all combinations of one or more of the referenced items.

The terms used in this specification are for describing embodiments and are not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “consisting of” stated components, steps, operations and/or elements do not exclude the presence or addition of one or more other components, steps, operations and/or elements. .

Although it is used to describe various components such as first and second, it goes without saying that these components are not limited to these terms. These terms are used to distinguish between only one component. Therefore, it goes without saying that the first component mentioned below may also be the second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, an airport control system and method using recognition technology according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a block diagram of a smart airport control system 100 according to an embodiment of the present invention. Referring to FIG. 1, the smart airport control system 100 includes a sensor system 120. It may be configured to include a communication network 130, a management server 140, a control tower server 150, etc.

The sensor system 120 is composed of first to nth sensors (120-1 to 120-n). The first to nth sensors (120-1 to 120-n) are used to detect not only the aircraft 110 but also various objects within the gate (aircraft, tow truck, tug car, Dali, Unit Load Device (ULD), Ground Power Unit (GPU) ), cargo, water truck, sewage truck, catering truck, garbage truck, cargo loader, ladder, step car, etc.), and performs the function of detecting the movement and/or presence of the pilot's review window, etc.

To this end, the first to nth sensors 120-1 to 120-n may be digital cameras, closed circuit televisions (CCTVs), voice sensors, infrared cameras, thermal image sensors, position sensors, etc. CCTV (Closed Circuit Television) can be fixed CCTV, thermal imaging CCTV, hybrid PTZ (Pan-Tilt-Zoom) camera, etc. Fixed CCTV can be used to configure the same view as the controller's field of view, configure 360° panoramic images, provide control information through AR, and provide surveillance video of the apron. Thermal imaging CCTV can be used to reproduce the field of view in low visibility situations, compose 360° panoramic images, and provide control information through AR. Hybrid PTZ (Pan-Tilt-Zoom) cameras can be used to provide target tracking images and to switch between visible and thermal images.

Of course, the first to nth sensors 120-1 to 120-n may be composed of some blocks.

The communication network 130 refers to a connection structure that allows information exchange between nodes such as a plurality of terminals and servers, such as public switched telephone network (PSTN), public switched data network (PSDN), and integrated information and communication network (ISDN). It can be Integrated Services Digital Networks (BISDN: Broadband ISDN), Local Area Network (LAN), Metropolitan Area Network (MAN), Wide LAN (WLAN), etc. there is,

However, the present invention is not limited to this, and is applicable to wireless communication networks such as CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), Wibro (Wireless Broadband), WiFi (Wireless Fidelity), and HSDPA (High Speed Downlink Packet Access). ) network, Bluetooth, NFC (Near Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc. Alternatively, it may be a combination of these wired communication networks and wireless communication networks.

The management server 140 collects voice information, video information, weather information, flight information, aircraft information, etc. from the sensor system 120, terminals, and servers connected to the communication network 130 and stores them in the database 141, It performs the function of recognizing, analyzing, and tracking multiple moving objects, analyzing data based on this, and generating analysis-provided information.

Here, a moving object is a concept that can include not only objects but also people. Additionally, flight information, aircraft information, etc. may be included in flight information indicating information about the aircraft 110 itself, operation of the aircraft 110, and movement path.

In particular, the management server 140 can acquire flight information from existing surveillance equipment (Airport Surveillance Radar (ASR), Airport Surface Detection Equipment (ASDE), multilateration (MLAT), and Automatic Dependent Surveillance-Broadcast (ADS-B). It may be possible. Flight information can be obtained and displayed through computer vision technology. Flight information may include aircraft flight number, registration number, and arrival gate.

Of course, in Figure 1, the management server 140 is configured separately from the control tower server 150, but it is also possible to configure the management server 140 by merging it with the control tower server 150.

The control tower server 150 provides a view to perform tasks such as engine start of the aircraft 110, rear towing permission, aircraft movement permission, transfer of control authority to the main control tower, deicing and anti-icing support, and towing control and support of the aircraft 110 within the apron. Provides information. Accordingly, the control tower server 150 may be installed at each apron, or multiple aprons may be managed with one control tower server 150. The control tower server 150 may have a server duplication structure. Server redundancy configures physical or logical servers (or LAPRs) to ensure that services can be continued through other servers in the event of a failure in one service.

The apron is used by aircraft for purposes such as boarding passengers, loading and unloading of cargo and mail, refueling, parking, de-icing (removing snow, ice and frost on the surface of aircraft and preventing their formation), or maintenance. This refers to an area set up to allow Rear towing refers to the operation of pushing a departing aircraft backwards with a tow vehicle and placing it on the taxiway.

The control tower server 150 provides the controller with view information based on flight situation information generated using analysis provided information and flight information transmitted from the management server 140. The flight status information combines analysis provided information and flight information to show the location, movement path, status, etc. of the aircraft 110.

This navigation status information is processed into panoramic view, gate view, and around view using synthesis technology and output on an ultra wide display consisting of multiple panels. Therefore, when performing control tasks, the controller maximizes situational awareness by strengthening the monitoring ability within the control area in charge.

FIG. 2 is a detailed configuration diagram of the management server 140 shown in FIG. 1. Referring to FIG. 2, the management server 140 may be configured to include a communication unit 210, a collection unit 220, an analysis unit 230, an analysis provision information generation unit 240, an output unit 250, etc. there is. The communication unit 210 performs a function of performing a communication connection with the communication network 130. For this purpose, the communication unit 210 may be configured to include a LAN card, a modem, etc.

The collection unit 220 performs a function of collecting data (i.e., information) from terminals and servers connected to the communication network 130 through the communication unit 210. Collected data may include audio information, video information, weather information, flight information, aircraft information, flight information, etc., and the collection unit 220 stores such collected data in the database 141.

The database 141 may be configured within the management server 140 itself, or may be configured as a separate database server. The collected data is interconnected with aviation information generated and collected at the airport. These data may include video data, track data, flight and airport resource data, AFL (AirField Lighting) data, weather data, voice communication data, etc.

- Video materials: Multiple ultra-high resolution 360-degree optical image synthesis, use of a multi-functional high-magnification camera

- Track data: Apron control platform, ASDE (Airport Surface Detection Equipment), ADS-B (Automatic Dependent Surveillance-Broadcast), MLAT (multilateration), Laser RANGE FINDER, ASDE-EFS (Electronic Flight Strip) , ASDE-EFS includes aircraft operation control information, aircraft departure and arrival information, and aircraft time information at specific points.

- Flight and airport resource data: Apron control platform, IIS (Integrated Information System), FDT (Flight Data Terminal), NOTAM (Notice to Airman), ATFM (Air Traffic Flow Management and Airspace Management), IIS is daily Flight operations, departure/arrival ramp access times, departure and arrival flight operations, A-CDM (Airport Collaborative Decision Making) information, runway friction coefficient, boarding gate ON/OFF, movement area vehicles, movement area work plan, operation schedule, etc. there is.

- Airfield Lighting (AFL) data: A-SMGCS (Advanced-Surface Movement Guidance and Control System)

- Meteorological data: Airport weather observation equipment (Aerodrome Meteorolgical Observation System, AMOS), Low Level Windshear Alert System (LLWAS), Terminal Doppler Weather Radar (TDWR)

- Voice communication data: VCCS (Voice Communication Control System)

The analysis unit 230 analyzes the collected data and extracts information such as the type, movement, and location of the moving object. The analysis unit 230 can receive collected data from the collection unit 220 in real time, and can also search and obtain the data from the database 141.

The analysis provision information generation unit 240 generates analysis provision information to be provided to the control tower server 150 based on the analysis information generated in the analysis unit 230. In other words, it generates analysis-provided information including the movement and location of the moving object.

The output unit 250 performs the function of displaying information being processed or a screen showing a setting menu, input menu, etc. To this end, the output unit 250 may be configured to include a display, a sound system, etc. Displays include LCD (Liquid Crystal Display), LED (Light Emitting Diode) display, PDP (Plasma Display Panel), OLED (Organic LED) display, touch screen, CRT (Cathode Ray Tube), flexible display, micro LED, mini LED, etc. This can be. At this time, in the case of a touch screen, it can also be used as an input means.

The collection unit, analysis unit, and analysis providing information generation unit shown in FIG. 2 refer to units that process at least one function or operation, and may be implemented in software and/or hardware. In hardware implementation, an application specific integrated circuit (ASIC), digital signal processing (DSP), programmable logic device (PLD), field programmable gate array (FPGA), processor, microprocessor, and other devices designed to perform the above-described functions. It may be implemented as an electronic unit or a combination thereof. In software implementation, software composition components (elements), object-oriented software composition components, class composition components and task composition components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, data. , databases, data structures, tables, arrays, and variables. Software, data, etc. can be stored in memory and executed by a processor. The memory or processor may employ various means well known to those skilled in the art.

FIG. 3 is a detailed configuration diagram of the control tower server 150 shown in FIG. 1. Referring to FIG. 3, the control tower server 150 may be configured to include an input unit 310, a processing unit 320, a view creation unit 330, a storage unit 340, a communication unit 360, etc.

The input unit 310 performs the function of inputting the controller's command. Therefore, the input unit 310 may be a mouse, keyboard, microphone, motion sensor, etc. Therefore, the controller can input commands through voice, motion, etc., without using a mouse or keyboard. In general, the controller would check the monitor by looking down or up, and then check the results by clicking the mouse and/or operating the keyboard. In this case, as the direction is displayed different from the controller's fixed direction, the controller must perform control tasks under conditions of continuous orientation and dimensional cognitive dissonance (Orientational Difference and Dimensional Difference). This has a negative impact on the controller's situational awareness and at the same time leads to an increase in the controller's workload, which ultimately has a significant impact on the efficiency of air traffic control work.

In one embodiment of the present invention, commands can be input through the controller's motion (including gestures), voice, etc., without using a mouse, keyboard, etc. Therefore, the controller can input, modify, and control the functions of the main control system using specific motions and voices while maintaining head-up.

The processing unit 320 performs a function of predicting the flight situation using analysis provided information and flight information transmitted through the communication unit 360.

The view generator 330 performs a function of generating view information that shows the navigation situation graphically (which may include images). The view information may include a panoramic view 351, an apron view 352, an around view 353, and a tracking CCTV view 354. View information can be expressed as virtual reality (VR) using actually captured images, virtual images, etc., augmented reality (AR), and mixed reality (MR: Mixed Reality) that combines virtual reality and augmented reality. You can. The actual captured video can be ultra-high resolution omnidirectional video-based surveillance video acquired using multiple visible and thermal cameras with resolutions of FHD (Full High Definition) or higher.

Using this, it can be expressed as follows.

- Displayed in various data overlay formats on omnidirectional composite images and various images

- Detection and tracking of moving objects, including aircraft

- Real-time display in conjunction with actual detection and tracking data and various data

- Display of weather data (wind direction and speed, runway visual range (RVR), wind shear, microburst warning, etc.)

- Utilization of thermal imaging and display of virtual wake when visibility is limited

- Display of gate information and A-CDM ((Airport Collaborative Decision Making) (TTOT (Target Take Off Time), TSAT (Target Start Up Approval Time)) related information

- Display of control frequency setting value and transmission status

- Various equalization displays including individual routing information (INDIVIDUAL ROUTING)

- CPDLC (Controller Pilot Data Link Communications) data display

- CCTV video display for each gate

In addition, view information integrates and displays images obtained from high-resolution CCTV and various aviation information through existing control systems by applying AI (computer vision), AR, and image synthesis technology. The panoramic view (351), which is the controller's view, is a single view of the entire control area that is a composite of individual camera (i.e. CCTV) images, and readability can be improved using augmented reality. The apron view (352) is a view that enables monitoring of the view-blocked apron, can provide necessary information for each aircraft situation, and enables monitoring and notification of pushback situations using AI (Artificial Intelligence) technology.

The apron view 353, which is an around view, is a view that shows the display status of the entire aircraft at the gate using 3D virtual space, and can display the movement path of the aircraft and show schedule and status information for each gate.

The tracking CCTV view 354 is a view that automatically tracks and shows an aircraft entering a port, shows automatic image tracking of a situation-aware target, and can show apron images as needed by the controller.

The storage unit 340 functions to store programs, software, data, etc. having an algorithm for predicting flight conditions using analysis provided information and flight information.

The storage unit 340 is a flash memory type, hard disk type, multimedia card micro type, or card type memory (for example, SD (Secure Digital) or (eXtreme Digital memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), PROM (Programmable Read Only Memory) ), and may include at least one type of storage medium among magnetic memory, magnetic disk, and optical disk. Additionally, it may operate in connection with web storage and cloud servers that perform storage functions on the Internet.

Continuing to refer to FIG. 3 , the communication unit 360 performs a function of performing a communication connection with the communication network 130. For this purpose, the communication unit 360 may be configured to include a LAN card, a modem, etc.

FIG. 4 is a detailed configuration diagram of the processing unit 320 shown in FIG. 3. Referring to FIG. 4, the processing unit 320 may be configured to include a data recognition module 410, a data analysis module 420, a flight situation information generation module 430, etc.

The data recognition module 410 performs voice recognition, image recognition, situation recognition, etc. on data transmitted from the management server 140 (including analysis provision information, flight information, etc.). Voice recognition can be done by converting VCCS and TRS voice information into TEXT information, identifying communication targets through callsign identification, and reading whether the control instructions and the name recovery window match. The call sign is used to clearly indicate the airline and flight number of the aircraft when communicating between the aircraft and the air traffic control center. To elaborate, real-time communication content is collected from the control communication server (VCCS), noise is removed, features are extracted, and the patterns of the extracted features are compared and converted into text. Additionally, the syntax of the control instructions is analyzed.

Whether or not the names are consistent is determined through readback between the controller and the pilot, and is as follows.

For voice recognition, the following procedure is performed.

- Automatic login/out of HMI (Human Machine Interface) after voice recognition and identification of individual controllers

- Provides various aircraft information and other flight information after voice recognition of air traffic control terms

- In the case of voice recognition of air traffic control terms, air traffic control terms and procedures are analyzed and understood, converted into text, and the voice content of communication between air traffic controllers and pilots is analyzed and checked for consistency.

- The aircraft information currently being communicated with can be displayed on the AR monitor based on the aircraft information recognized through the readback process during air traffic control communication.

- Aircraft information displayed at the bottom of the 3D modeled aircraft within AR

Image recognition can include identifying information about aircraft and vehicles that can be seen in received images, and recognizing abnormal situations within the apron. Situational awareness can include collision prediction recognition and notification, abnormal situation recognition and notification, and non-compliance with air traffic control instructions recognition and notification.

Motion recognition can be achieved through hand tracking. That is, it can be a fist, number of fingers, or movement. Therefore, commands are delivered using predetermined hand gestures. For example, clenching a fist stops the screen movement and moves the AR position according to the direction of the finger (index finger). Open your palm to enlarge the screen, and pinch it together to zoom out.

The data analysis module 420 performs analysis using the data identified by the data recognition module 410. To elaborate, through analysis, flight situation information is generated. In this case, map information and navigation status information are synthesized.

In addition, the data analysis module 420 analyzes the voice communication between the pilot and the controller in real time to detect abnormal situations, determine whether the pilot's name recovery matches, verify the control instructions and lights, and verify the aircraft movement status.

In addition, the data analysis module 420 utilizes AI (Computer Vision) technology to block the Line of Sight (LoS) view due to physical obstacles such as airport buildings and block the view due to meteorological phenomena (fog, rainfall, snowfall, etc.). Creates virtual objects (contours or object 3D modeling, etc.) using object recognition and analysis, AR (Augmented Reality) technology, and digital twin technology. Therefore, it is possible to overcome the controller's field of view occlusion.

Digital twin technology is a technology that predicts results in advance by creating twins of objects in reality on a computer and simulating situations that may occur in reality on a computer.

In addition, the data analysis module 420 uses synthetically processed Gate View images based on computer vision image recognition technology to view various objects within the gate (aircraft, tow truck, tug car, Dali, ULD, GPU, cargo, water truck, sewage truck, The movement and presence of catering trucks, garbage trucks, cargo loaders, ladders, step cars, etc.) are combined with the corresponding flight operation data to analyze and learn to predict the operation situation and link this to A-CDM. Learning generally uses deep learning technology, but is not limited to this and machine learning may also be used.

The flight situation information generation module 430 performs a function of generating flight situation information using data generated by the data analysis module 420. To elaborate, cognitive dissonance arising from the dimensional difference between the apron controller's field of view and various system data, human error due to head-down due to the operation of multiple independent equipment and individual input equipment, and physical damage such as terminals within the airport. To overcome visibility obstruction due to structures and limited visibility due to meteorological phenomena (fog, rain, snowfall), information from the main control system and flight information system within the airport is integrated. The integrated information is displayed on a multi-screen telescopic display in the apron controller's front view by combining AI and AR-based computer vision image recognition and voice recognition technologies.

Therefore, provision of real-time integrated information screens, introduction of various notification and warning systems, application of touch-less control methods, and simple and intuitive interfaces can be applied.

The data recognition module 410, data analysis module 420, and flight situation information generation module 430 shown in FIG. 4 refer to units that process at least one function or operation, and are implemented in software and/or hardware. It can be. In hardware implementation, an application specific integrated circuit (ASIC), digital signal processing (DSP), programmable logic device (PLD), field programmable gate array (FPGA), processor, microprocessor, and other devices designed to perform the above-described functions. It may be implemented as an electronic unit or a combination thereof. In software implementation, software composition components (elements), object-oriented software composition components, class composition components and task composition components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, data. , databases, data structures, tables, arrays, and variables. Software, data, etc. can be stored in memory and executed by a processor. The memory or processor may employ various means well known to those skilled in the art.

Figure 5 shows control instruction implementation verification and warning through analysis of voice communication between pilot and controller according to an embodiment of the present invention. This is a flowchart showing the process. When there is voice communication between the pilot of the aircraft 110 and the controller, the controller in the control tower gives control instructions using the control tower server 150 (step S510). Control instructions are delivered to the pilot of the aircraft 110 or the driver of the moving object through voice communication. At this time, voice analysis is performed on the standardized controller's control instructions (step S520).

In the voice analysis process (S520), the processing unit 320 records the standardized voice recording of the controller in the storage unit 340 (step S521). Afterwards, the processing unit 320 analyzes the sentence of the control instruction and extracts the command (steps S522 and S523). The standardized controller's control instructions are recorded through voice recording, and the meaning of the relevant control instructions can be extracted by analyzing the sentences of the communication. Sentence extraction removes noise, extracts only audio data, and converts this audio data into speech-to-text. Afterwards, commands are extracted from the text.

Here, the movement of the characteristic target aircraft 110 (or a specific target moving object) that matches the meaning of the extracted control instruction can be set and related captured images can be collected using the sensor system 120 (step S530). For example, it is possible to check whether the ground movement of the moving object matches and whether the pushback direction matches.

Afterwards, it is possible to verify whether the control instructions and the movement of the aircraft match through analysis of the captured aircraft (step S540).

For example, assume that the controller's control instruction is "Push back approved to face south." This control instruction is to perform rear towing with the nose of the aircraft (110) facing south, but if this is violated and the nose of the aircraft is facing north, this is detected through video analysis and track data from existing airport surveillance equipment is used. By mixing, the ability to verify the control performance can be improved. When the above case occurs, the processing unit 320 confirms that there is a discrepancy between the control instruction and the actual performance of the aircraft and outputs the analysis result as a warning message to the controller view 351. Through the warning function, controllers can expect to be able to recognize non-compliance with control instructions and abnormal situations early and secure sufficient time for appropriate response. Warning messages may consist of a combination of graphics, audio, and text.

Figure 6 is a concept of aircraft location and operation stage analysis through voice communication analysis between pilots and controllers according to an embodiment of the present invention. Based on the communication content (610), the aircraft's location (620) is displayed. To elaborate, the location of the currently communicating aircraft is displayed on the panoramic view, etc. Of course, the location of the aircraft in communication (620) and the content of the communication (610) can also be displayed. The communication content (610) is the communication status with aircraft ESR1234.

Existing surveillance equipment (ASR, ASDE, MLAT, etc.) in operation at the airport is effective in identifying accurate location information of aircraft, but it is difficult to obtain information on the status, operation phase, and control instructions of the aircraft. The EFS (Electronic Flight Strip) system, which is directly operated by the apron controller (airport controller), records the aircraft's operation stages and control instructions, but not all details are recorded.

Therefore, since the information is recorded by the controller's additional input actions after the control instruction is issued or at a specific time, a difference between the actual instruction and the input time occurs. Therefore, when analyzing the location and operation stage of an aircraft through analysis of pilot-controller voice communication, immediate location estimation, aircraft status, and operation stage can be displayed in the view at the moment of communication between the controller and pilot. In this case, it is possible to quickly determine the status of the aircraft, and the information can be used to identify airport operations and abnormal situations. An example is as follows:

Example 1)

"KAL123: Apron, KAL123 Request Push-Back Gate 253"

Aircraft location : Within Gate 253 of T2 Terminal

Aircraft status : All doors closed, Engines-off, Tow Truck Connected, Ready for Push-back

Operation stage : Ready for rear towing

Example 2)

"ICN TWR: KAL123, Incheon Tower, Line-up and Wait RWY33L"

Aircraft location : Estimated before entering runway 33L

Aircraft Status : Ready for Departure(Take-off Configuration), Engines Running, Taxing, On the Ground

Operation stage : Parallel taxiway taxiing in maneuvering area

Example 3)

"ICN TWR: KAL124, Go-Around, Traffic on the RWY, Maintain RWY Heading, Climb to 3000ft.

KAL124: Going-Around, Maintain RWY Heading, Climb to 3000ft, KAL124

Aircraft location : Landing runway final approach route

Aircraft Status : Airborne, Going-Around, Engines Running

Operation stage : Go around while performing final approach stage

Special note : Change in scheduled landing time due to carrying out go-around procedures, expected delay for subsequent approach aircraft

Reason for go-around : Traffic on runway

FIG. 7 is a diagram showing a situation of entering the parking lot according to an embodiment of the present invention, and FIG. 8 is a diagram showing a pushback situation after entering according to FIG. 7. Referring to Figures 7 and 8, it is possible to identify aircraft entering/exiting the apron and verify the safety of the apron. For example, when the aircraft 110 enters more than 50% of the gate zone, it is judged to be entering the gate zone. To elaborate, it includes the function of identifying aircraft entering/exiting the apron through image recognition, verifying the apron situation, and issuing a warning. Verification elements prior to entering the apron include opening of the apron, opening of workers, opening of the rear GSE road (road installed to access the apron and aircraft to perform activities supporting airport and airline operations), wing guard placement, and PBB (Passenger Boarding). Bridges) can be in fixed positions, etc.

Verification elements after entering the apron can include analysis of registration number and flight number, apron grade-main type comparison, confirmation of engine stop based on engine temperature analysis, analysis of nose gear stop position, and recognition of toy car preparation.

Pushback refers to a state in which a towing car is coupled to the front of an aircraft. If the toy car and the aircraft are moving in the same direction and at the same speed, the two objects are recognized as pushback and tracked. In contrast, if it falls over a certain distance in the pushback state, it is recognized as an individual object again. Pushback readiness verification is done through confirmation of equipment/personnel opening, recognition of the toy car, confirmation of rear opening, and confirmation of PBB correct position. Meanwhile, verification of the pushback impossible status is done through confirmation of equipment and personnel detection, toy car not recognized, and PBB contact. To elaborate, examples include the presence of an aircraft in the rear parking lot, engine on/off, door opening, toy car not connected even if toy car is recognized, toy car not recognized, and PBB approach.

After acquiring images from digital cameras and thermal imaging cameras installed in front of the apron, they are analyzed, and the corresponding types of aircraft entering and exiting the apron are recognized through computer vision analysis using a custom data set and compared with flight information to check for consistency. .

It constitutes a preliminary safety verification function of the apron before aircraft entry and exit, and this function operates from the scheduled landing time of the incoming aircraft to verify the preparation status of the apron. Verification items for this function include checking whether an aircraft exists, checking the exact location of the boarding bridge, checking people and objects within the aircraft parking protection line, checking whether vehicles are passing on the rear GSE road, and checking whether wing-guard personnel are deployed, etc. These items are reviewed. If safety is not secured, a warning message can be displayed to the controller by changing the color, creating a symbol, and lighting it on the control screen.

The entering aircraft utilizes image recognition technology to identify the type of aircraft entering the port, and compares it to the maximum capacity for each type of aircraft at the apron. When an attempt is made to enter an aircraft that exceeds this, this is displayed on the control screen to prevent mis-entry accidents involving excess aircraft. prevent it

In addition, the operation information such as aircraft flight number, registration number, and arrival gate is compared with the image recognition information of the entering aircraft to determine the authenticity of the aircraft's entry into the apron. If a discrepancy is determined, this is displayed on the control screen to prevent erroneous entry into the apron.

After the aircraft enters the parking lot, the entry speed of the aircraft is determined to determine whether or not to stop. By recognizing the position of the nose gear, it is possible to determine whether the aircraft is underparked by identifying the appropriate parking position of the aircraft. This is displayed as a warning on the control screen to indicate underparking. Collisions between aircraft and aircraft moving on the rear guide line can be prevented.

Whether or not to terminate the operation of an incoming aircraft can be determined by analyzing the temperature data of the aircraft engine collected from thermal imaging images, and whether the engine is turned on or off can be obtained as reference data. Safety can be secured when entering and exiting an aircraft by analyzing cases where vehicles or personnel enter the aircraft parking protection line during the aircraft entry phase or when the boarding bridge moves before the aircraft comes to a complete stop.

When a towed aircraft enters the apron, the presence/absence of the towed aircraft and aircraft type can be determined through video analysis, and this can be reviewed with flight data to provide the same verification function as that of self-arriving aircraft.

Figure 9 is a diagram showing the pushback preparation state according to Figure 8. In other words, the apron operation situation is analyzed and the operation stage is predicted according to the analysis of the pushback readiness state. At the end of the movement of an aircraft that has completed self-introduction or has been moved to the apron by towing, analysis of the operation status of the aircraft and predictive analysis of the operation stage begin.

The analysis utilizes video analysis from digital cameras and thermal imaging cameras installed at the apron, and the analysis includes access to the boarding bridge within the apron, passenger boarding/disembarkation, front/rear cargo door opening/closing, cargo loader movement and work status, tug car movement and Work status, Dali and Dali-loaded cargo and ULD movement and loading status, belt loader operation status, add-back vehicle operation status, sewage vehicle operation status, fresh water vehicle operation status, catering truck operation status, garbage truck operation status, cabin cleaning and disinfection , the degree of snowfall outside the aircraft, whether the aircraft engine is started, the operation status of the maintenance vehicle, the operation status of the refueling pump car, confirmation of the presence of a toy car, confirmation of whether the tow bar of the toy car is connected, and confirmation of the deployment of wing-guard personnel, etc. It is used as data for predictions for each stage of aircraft operation.

Referring to Figure 9, the pushback readiness state is verified in three stages.

1) Stage: Check the location of the toy car and aircraft (i.e. airplane)

2) Stage: Check the separation of the passenger boarding bridge (PBB)

3) Stage: Check the presence of vehicles on the left, right, and rear of the aircraft

FIG. 10 is a diagram showing a pushback impossible situation according to FIG. 8. In other words, in the following three situations, pushback is judged impossible.

1) Presence of vehicles around

2) Boarding bridge installation status

3) Toy Car not recognized

Figure 11 is a conceptual diagram of aircraft rear towing restrictions identification and warning according to an embodiment of the present invention. Referring to Figure 11, 1) the voice communication content is analyzed, 2) the aircraft's rear towing readiness status is verified, and 3) the traffic situation near the aircraft is verified.

1) Analysis of voice communication contents can include pilot departure request for rear towing, voice content analysis, and target aircraft and apron identification.

2) Verification of the aircraft's rear towing readiness status can be confirmed by video confirmation of aircraft departure preparation, video confirmation of stand preparation (personnel, vehicle), confirmation of acquisition of flight permit (ATC (Air Traffic Control) Clearance), and confirmation of air traffic flow management time. there is.

3) Verification of traffic conditions near the aircraft can include verification of self-moving aircraft, verification of aircraft towing the rear of a nearby apron, and verification of workers and vehicles.

Through video analysis of the apron, if a rear-traction aircraft for departure from an adjacent apron is close or a rear-traction aircraft for dissipation movement is close, enhanced apron video analysis and restriction identification and warning are provided to ensure the safety of the rear-traction aircraft at the relevant apron. The function works.

Whether or not a rear tow permit is issued is obtained through IIS, EFS, or air traffic control voice analysis, and identification and warning functions are activated based on whether a rear tow permit is issued to the aircraft at the relevant apron. Preparation for rear towing of a parking aircraft depends on the presence of an aircraft, whether the aircraft exterior is suitable for departure rear towing (front/rear cargo doors closed, boarding gate closed, etc.), confirmation of the correct position of the boarding bridge, presence of a tow truck, and tow truck. Check whether the tow bar and nose gear of the aircraft are connected, whether wing-guard personnel are deployed, whether the engine is turned on/off, and whether flight permission for the aircraft is obtained. When a rear towing permit is issued without the relevant conditions being met, it is displayed on the control screen and the controller is made aware of the unmet items, thereby ensuring rear towing safety.

Additionally, the steps of the method or algorithm described in relation to the embodiments disclosed herein are implemented in the form of program instructions that can be executed through various computer means such as a microprocessor, processor, CPU (Central Processing Unit), etc., and are computer readable. Can be recorded on any available medium. The computer-readable medium may include program (instruction) codes, data files, data structures, etc., singly or in combination.

The program (instruction) code recorded on the medium may be specially designed and constructed for the present invention, or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROM, DVD, and Blu-ray, and ROM and RAM. Semiconductor memory elements specially configured to store and execute program (instruction) code, such as RAM), flash memory, etc., may be included.

Here, examples of program (instruction) code include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

100: Smart airport control system
120: sensor system 130: communication network
140: management server 141: database
150: Control tower server
210: Communication Department 220: Collection Department
230: analysis unit 240: analysis provision information generation unit
250: output unit 310: input unit
320: Processing unit 330: View creation unit
340: storage unit
351: Panoramic view 352: Apron view
353: Around view 354: CCTV (Closed Circuit Television) view for tracking
410: Data recognition module 420: Data analysis module
430: Flight situation information generation module

Claims (10)

A sensor system 120 that generates sensing information by sensing a moving object including an aircraft 110 in the apron;
A management server 140 that obtains flight information about the aircraft 110 from a plurality of monitoring equipment; and
A control tower server 150 that uses the sensing information and the flight information to recognize verification elements regarding the aircraft situation at the apron by image recognition and to analyze voice recognition for control instructions communicated between a controller and a user of the moving object; Includes,
The control tower server 150 provides a warning message to the control tower according to the analyzed image recognition and voice recognition results,
The control tower server 150 determines whether the sensing information collected according to the movement of the moving object matches the execution of the control instruction, and outputs the warning message according to the determination result,
The control tower server 150 determines that the aircraft 110 is entering the gate zone when it enters the gate zone by more than a preset value, and outputs the judgment,
The control tower server 150 compares the flight information including the flight number, registration number, and arrival gate of the aircraft 110 with the image recognition information of the aircraft 110 entering the apron area to determine whether there is a discrepancy. An airport control system using recognition technology that makes judgments and displays the judgment results. delete According to claim 1,
The analysis of the above control instructions is,
An airport control system using recognition technology, characterized in that the control instructions recorded as voice recordings are converted into text and the commands of the control instructions are extracted from the text.
According to claim 1,
The control tower server (150) is an airport control system using recognition technology, characterized in that the aircraft location (620) is output according to the content of voice communication (610) between the controller and the user of the moving object.
In clause 4
The control tower server (150) is an airport control system using recognition technology, characterized in that the communication content (610) and the aircraft location (620) are simultaneously output.
According to claim 4,
The control tower server (150) is an airport control system using recognition technology, wherein the control tower server (150) simultaneously outputs the aircraft location, aircraft status, operation stage, unusual information, and reason for go-around according to the analysis of the communication content (610).
delete delete A sensor system 120 that generates sensing information by sensing a moving object including an aircraft 110 in the apron;
A management server 140 that obtains flight information about the aircraft 110 from a plurality of monitoring equipment; and
A control tower server 150 that uses the sensing information and the flight information to recognize verification elements regarding the aircraft situation at the apron by image recognition and to analyze voice recognition for control instructions communicated between a controller and a user of the moving object; Includes,
The control tower server 150 provides a warning message to the control tower according to the analyzed image recognition and voice recognition results,
The control tower server 150 determines whether the sensing information collected according to the movement of the moving object matches the execution of the control instruction, and outputs the warning message according to the determination result,
The control tower server 150 determines that the aircraft 110 is entering the gate zone when it enters the gate zone by more than a preset value, and outputs the judgment,
The control tower server 150 predicts the operation stage of the aircraft 110 through video analysis at the end of movement of the aircraft 110 after the apron entry situation is completed. An airport control system using recognition technology.
(a) the sensor system 120 generating sensing information by sensing a moving object including the aircraft 110 in the apron;
(b) the management server 140 acquiring flight information about the aircraft from a plurality of monitoring equipment; and
(c) The control tower server 150 uses the sensing information and the flight information to recognize verification elements regarding the aircraft situation at the apron by image recognition and voice recognition of control instructions communicated between the controller and the user of the moving object. Analyzing step; Includes,
The control tower server 150 provides a warning message to the control tower according to the analyzed image recognition and voice recognition results,
The control tower server 150 determines whether the sensing information collected according to the movement of the moving object matches the execution of the control instruction, and outputs the warning message according to the determination result,
The control tower server 150 determines that the aircraft 110 is entering the gate zone when it enters the gate zone by more than a preset value, and outputs the judgment,
The control tower server 150 compares the flight information including the flight number, registration number, and arrival gate of the aircraft 110 with the image recognition information of the aircraft 110 entering the apron area to determine whether there is a discrepancy. An airport control method using recognition technology characterized by making a judgment and displaying the judgment result.