TW202416148A - Method and system for detecting aircraft behavior on a ramp - Google Patents

Abstract

The present invention provides a method for detecting the behavior of an aircraft on an apron, comprising: identifying at least one apron image to obtain state conditions representing an aircraft area and an air bridge area; determining whether the apron image belongs to one of a plurality of scenario modes according to the state conditions; judging the spatial relationship and the temporal relationship between aircraft objects in the aircraft area and air bridge objects in the air bridge area in the apron image according to a plurality of judgment conditions set in the scenario mode, and generating a judgment result accordingly; and outputting an event image formed by combining the judgment result with the apron image. The present invention can accurately and instantly obtain the time when an aircraft and an air bridge enter and leave the apron.

Description

Method and system for detecting aircraft behavior on a ramp

The present invention relates to a method and a system for detecting the behavior of an aircraft on an apron, and more particularly to a method for detecting the entry, departure, and joining time of an aircraft and an airbridge.

When the aircraft taxis into the apron, the captain relies on manual guidance or guidance systems to land and park in the correct position. Common guidance systems will identify the aircraft model and use signals to guide the captain to correct the taxi route, slow down, and brake to the nose wheel stop line. On the other hand, manual guidance uses a trailer to move the aircraft position, that is, the trailer driver operates the aircraft's parking route and parking position.

Since the time an aircraft enters and leaves an airport affects landing fees and parking fees, the airport needs to accurately record the aforementioned time. In response to the large number of aircraft takeoffs and landings, the airport can solve this problem through a large amount of manpower, but this will increase costs accordingly and there are concerns about the accuracy of the records. In previous technologies, clamps based on the Internet of Things (IoT) technology were used to clamp the tires or other contact points of the aircraft and automatically send time information. However, this method still relies on manual transmission of time information and can only provide basic time information, so it has not yet achieved a fully automated process.

In view of the shortcomings of the prior art, the present invention provides a method for detecting the behavior of aircraft on the apron. Through object detection technology and foreground-background separation technology, the present invention can accurately and quickly obtain the entry, departure and connection time of aircraft and air bridge.

The present invention provides a method for detecting the behavior of an aircraft on an apron, which identifies at least one apron image to obtain a state condition represented as an aircraft area and an empty bridge area in the at least one apron image, wherein the apron image is generated by at least one camera; based on the state condition, it is determined that the at least one apron image belongs to one of a plurality of scenario modes; based on a plurality of determination conditions set in the scenario mode, the spatial relationship and the temporal relationship between an aircraft object existing in the aircraft area and an empty bridge object existing in the empty bridge area in the at least one apron image are determined, and a determination result is generated; and an event image formed by combining the determination result with the at least one apron image is output.

In some embodiments, the present invention utilizes at least one of deep learning technology and foreground-background separation technology to identify the spatial relationship and temporal relationship of aircraft objects in the aircraft area on the apron, and to identify the spatial relationship and temporal relationship between skybridge objects in the skybridge area and the apron in at least one apron image, so as to obtain state conditions.

In some embodiments, the multiple scenario modes include an aircraft entry mode, an aircraft departure mode, an air bridge entry mode, an air bridge departure mode, and an export mode.

In some embodiments, the state relationship of the aircraft entry mode, aircraft departure mode, air bridge entry mode, air bridge departure mode, and exit mode is the spatial relationship and time relationship between the aircraft object and the air bridge object, and is defined as follows: the aircraft entry mode is defined as the aircraft object does not enter and does not leave in at least one apron image; the aircraft departure mode is defined as the aircraft object enters and the aircraft object leaves in at least one apron image. The object has not left; the air bridge entry mode is defined as an aircraft object entering at least one apron image, the aircraft object has not left, and the air bridge object has not yet been connected to the aircraft object; the air bridge exit mode is defined as an aircraft object entering at least one apron image, the air bridge object has been connected to the aircraft object, and the air bridge object has not left the aircraft object; and the output mode is defined as an aircraft object entering at least one apron image and the aircraft object has left.

In some embodiments, the judgment conditions include: the moving direction and speed of the aircraft object in the aircraft area or the empty bridge object in the empty bridge area, and whether the aircraft object and the empty bridge area exist. These judgment conditions are also used to determine whether to output the event image.

The present invention also provides a system for implementing the aforementioned method for detecting the behavior of an aircraft on a ramp.

As described above, the present invention identifies the aircraft area and the sky bridge area in the apron image based on object detection technology and foreground-background separation technology, thereby determining the relationship between the aircraft and the sky bridge in time and space. Therefore, the present invention can accurately and quickly determine the time when the aircraft and the sky bridge enter, leave, and connect. As a result, the present invention has multiple competitive advantages such as automated procedures, the ability to quickly trace back time data, and the reduction of operating time and labor costs.

The embodiments of the present invention will be further explained below with reference to the accompanying drawings. As far as possible, the same reference numerals in the drawings and the specification represent the same or similar components.

First, please refer to Figure 1, which is a schematic diagram of the system for detecting the behavior of aircraft on the apron of the present invention. The system 10 for detecting the behavior of aircraft on the apron of the present invention includes at least one camera 100, an image recognition unit 110, a processing unit 120, and an image combination unit 130. The camera 100 can be set on the airside of the airport, and the number of settings is selected according to practical needs. The camera 100 is used to take at least one apron image of the apron. The image recognition unit 110 is signal-connected to the camera 100 to obtain each apron image taken by each camera 100. The image recognition unit 110 is used to recognize the aircraft area and the sky bridge area in the apron image, and generate a state condition accordingly. The aircraft area represents the area where the aircraft object drives into and stops, and the sky bridge area represents the area where the sky bridge object approaches the aircraft object and connects with it. The state condition represents the time relationship and space relationship between the aircraft object and the sky bridge object. The processing unit 120 of the signal connection image recognition unit 110 is used to determine one of multiple situation modes according to the state condition; the processing unit 120 also judges the apron image according to multiple judgment conditions of the determined situation mode to generate a judgment result. The judgment result includes the spatial relationship and time relationship between an aircraft object existing in the aircraft area and an air bridge object existing in the air bridge area, such as the entry time and departure time of the aircraft object marked in the upper left corner of Figures 6 to 11, and the entry time and departure time of the air bridge. The image combining unit 130 is connected to the signal processing unit 120; the image combining unit 130 is used to combine the judgment result with the apron image to output the event image, that is, the apron image shown in Figures 6 to 11 and at least one message of the above judgment result.

In some embodiments, the image recognition unit 110 includes at least one of an image processing unit 111 and a deep learning unit 112. The image processing unit 111 processes the apron image based on the foreground-background separation technology and outputs the state condition. The deep learning unit 112 recognizes the apron image based on the deep learning model and outputs the state condition; or, the deep learning unit 112 recognizes the apron image processed by the image processing unit 111 based on the deep learning model and outputs the state condition.

Please refer to FIG. 2 to explain the method for detecting the behavior of an aircraft on the apron according to the present invention. The steps of the behavior detection method are as follows:

In step (A), the state conditions of the aircraft area and the air bridge area in the plurality of apron images are first identified from the plurality of apron images, and the plurality of apron images are generated by shooting with a camera.

In addition, please refer to Figure 3 to establish a deep learning technology and/or foreground-background separation technology to identify the spatial relationship and temporal relationship of aircraft objects on the apron in the aircraft area of the apron image, and the spatial relationship and temporal relationship between the sky bridge object and the apron in the sky bridge area, so as to obtain the status conditions of the aircraft objects.

In some embodiments, the aforementioned recognition process is implemented based on at least one of a deep learning technique and a foreground-background separation technique.

The method of establishing deep learning technology includes the preparatory work of establishing a deep learning model. In the preparatory work, first collect relevant public datasets such as aircraft of different types and work vehicles recorded in the airport terminal, and collect images of the apron on the actual side of the present invention; then, use annotation tools such as Label Image to annotate these images and data and define categories, such as: aircraft nose and sky bridge, etc.; then, input the annotated data into the deep learning neural network for model training, and output the deep learning initial model. Customized model optimization can be performed in response to different detection locations, for example: to improve the accuracy or speed of judgment. Through the above steps and model optimization, the preparatory work of the deep learning model can be completed.

The deep learning of the present invention is a deep learning model for object detection. After the apron image acquired by the camera is read and imported into the deep learning model for object detection, the deep learning model can use algorithms related to object detection (including but not limited to YOLO series or SSD series algorithms) to infer and detect the type and position of the aircraft.

In some embodiments, the foreground-background separation technology is implemented based on Gaussian Mixture. The foreground-background separation technology can perform binarization and noise processing on the apron image. For example, the present invention can filter the noise in the apron image according to a threshold set to 125, thereby providing accuracy in recognition. Alternatively, the present invention can also obtain the appearance of the foreground object in the apron image through a contour detection algorithm. Specifically, by retaining foreground objects that occupy more than 2% of the image frame for subsequent determination, and ignoring foreground objects that occupy less than 2% of the image frame, the present invention can effectively reduce the noise of the apron image.

Based on the above, deep learning technology can detect and identify the type and location of aircraft in the apron image, and the foreground-background separation technology can produce the foreground object in the apron image. Next, please refer to Figures 4A and 4B, which respectively show the aircraft and apron images detected by deep learning, and the images of foreground objects produced based on the foreground detection algorithm. Since the present invention can compare the two images obtained based on deep learning technology and foreground-background separation technology, it can improve the analysis accuracy of the object location.

By reading and identifying the apron images generated in a time series, the temporal and spatial order relationship between aircraft objects and skybridge objects, i.e., the state condition, can be obtained; the state condition includes but is not limited to the category, movement direction, and position of the aircraft, skybridge, and foreground objects. Next, the present invention will determine the context mode according to the state condition of the apron image in step (B). Please refer to Figures 2 and 5. In some embodiments, multiple context modes may include aircraft entry mode, aircraft departure mode, skybridge entry mode, skybridge departure mode, and output mode. Through the spatial relationship and time relationship between the aircraft object and the airbridge object, it can be determined whether to enter the aircraft entry mode, aircraft departure mode, airbridge entry mode, airbridge departure mode or output mode.

The judgment logic and discriminants (i.e. judgment conditions) of the above-mentioned multiple situational modes are specifically described as follows:

1. When there is no aircraft object entering the apron image (the system displays the flag related to aircraft entry: False), and no aircraft object leaving (the system displays the flag related to aircraft departure: False), enter the aircraft entry discriminant;

2. When an aircraft object enters the apron image (the system displays a flag related to aircraft entry: True) and the aircraft object has not left (the system displays a flag related to aircraft departure: False), enter the aircraft departure discriminant;

3. The aircraft entering the empty bridge entry mode is defined as an aircraft object entering the apron image (the system displays a flag related to aircraft entry: True), the aircraft object has not yet left (the system displays a flag related to aircraft departure: False), and the empty bridge object has not yet been connected to the aircraft object (the system displays a flag related to the empty bridge connecting to the aircraft: False), and enter the empty bridge entry discriminant;

4. When an aircraft object enters the apron image (the system displays a flag indicating that the aircraft has entered: True), the air bridge object has been connected to the aircraft object (the system displays a flag indicating that the air bridge has been connected to the aircraft: True), and the air bridge object has not left the aircraft object (the system displays a flag indicating that the air bridge has left the aircraft: False), enter the air bridge leaving discriminant;

5. When an aircraft object in the apron image has entered (the system flag about aircraft entering is displayed: True) and has left (the system flag about aircraft leaving is displayed: True), the output result initialization process is entered.

Please refer to FIG. 2. In step (C), the multiple judgment conditions set in each scenario mode are used to judge the spatial and temporal relationships between the aircraft objects in the aircraft area and the sky bridge objects in the sky bridge area in the apron image to generate a judgment result.

Step (D) Output the event image formed by combining the judgment result with the apron image. Please refer to Figures 6 to 11, which are schematic diagrams of event images in different embodiments of the present invention. It should be noted that Figures 6 to 11 are all marked with time records, including: Gate, airplane_in, airplane_out, bridge_in, bridge_out. Among them, Gate will display the time of aircraft entry, aircraft departure, bridge entry, and bridge exit in time sequence. The other time records in these diagrams are generated when the judgment conditions of each situation mode are met and displayed in the corresponding time area; in other words, the other time records are the time of the judgment results.

In some embodiments, the aforementioned judgment conditions may also include: the moving direction, moving speed, or whether there are aircraft objects or empty bridge objects in the aircraft area or the empty bridge area. When generating the judgment result, a combination of one or more of the aforementioned conditions may be selected as the limiting standard for different judgment result levels. The aforementioned limiting standards may include the most stringent, normal, and loose, as described later.

If the most stringent limitation standard is adopted, all conditions of the moving direction, moving speed and whether there are foreground objects of the above-mentioned aircraft objects or empty bridge objects must be met, and the above-mentioned conditions must remain unchanged for N seconds before the judgment result will be output and combined into an event image. N is a positive integer. If the normal limitation standard is adopted, as long as the two conditions of the moving direction and moving speed of the aircraft objects in the aircraft area or the empty bridge objects in the empty bridge area are met, the judgment result will be output and combined into an event image. If a loose standard is adopted, as long as the condition of the moving direction of the aircraft objects in the aircraft area or the empty bridge objects in the empty bridge area is met, the judgment result will be output and combined into an event image. Of course, the above-mentioned conditions are only exemplary, and the present invention should not be limited to the above.

In some embodiments, the judgment standard of the moving speed of the so-called aircraft object in the aircraft area or the empty bridge object in the empty bridge area can be determined based on whether the pixel value of the movement exceeds the threshold. Similarly, the judgment standard of the moving direction of the so-called aircraft object in the aircraft area or the empty bridge object in the empty bridge area can be whether the moving direction of the aircraft object or the empty bridge object is the same as the pre-set moving direction of the aircraft object or the empty bridge object. In addition, the so-called judgment standard of whether there is an aircraft object or an empty bridge object can be whether the detection frame set by deep learning has a foreground object occupying more than 80%.

Next, Figures 6 to 11 will explain the time recording process of aircraft entry and departure.

As shown in the upper right corner of Figures 6 and 7, analysis_airplane_in represents that the method or system of the present invention has entered the aircraft entry judgment formula. The brackets on the right side of analysis_airplane_in show the judgment conditions, such as the above-mentioned moving direction and moving speed. As shown in Figure 6, when the judgment condition is [False, True, False], it means that no aircraft has entered the aircraft area (i.e., the apron gate). As shown in Figure 7, when the judgment condition is [True, True, True], it means that an aircraft has entered the apron gate; at this time, the judgment result of airplane_in is generated on the left side and displayed as airplane_in: 2022-09-28 14:56:58, indicating that the time when the aircraft entered the apron gate was fourteen fifty-six minutes and fifty-eight seconds. Among them, box 502 displays the aircraft area position, and box 504 displays the aircraft nose position, which allows the system to determine the type of aircraft and the time and space relationship of the aircraft entering the aircraft area through the image.

As shown in the upper right corner of Figure 8, analysis_bridge_in represents that the method or system of the present invention enters the empty bridge entry judgment formula. When the judgment condition of analysis_bridge_in is [True, True, True], the judgment result of bridge_in is generated and displayed as bridge_in: 2022-09-28 14:59:07, indicating that the time when the empty bridge was connected to the aircraft was fourteen fifty-nine minutes and seven seconds. Among them, the box 506 represents the empty bridge area, that is, the position where the aircraft is connected to the empty bridge, which is used to detect whether the empty bridge is in the box and complete the empty bridge connection with the aircraft. Figure 9 is an image taken by another camera device, in which the judgment condition of analysis_bridge_out is [False, False, False], indicating that the empty bridge has not yet separated from the aircraft.

As shown in the upper right corner of FIG. 10, analysis_bridge_out indicates that the method or system of the present invention enters the air bridge departure judgment formula. Specifically, when the judgment condition is [True, True, True], the judgment result of bridge_out is generated and displayed as bridge_out: 2022-09-28 16:57:40, indicating that the time when the air bridge leaves the aircraft is 16:57:40.

The analysis_airplane_out shown in the upper right corner of FIG. 11 represents that the method or system of the present invention enters the aircraft departure judgment formula. Specifically, when the judgment condition is [True, True, True], the system generates a judgment result of airplane_out and displays it as airplane_out: 2022-09-28 16:59:16, indicating that the time when the aircraft left the apron aircraft area was 16:59:16.

After obtaining the above-mentioned multiple time records, the method or system of the present invention can accurately calculate the time when the aircraft lands on the apron by calculating the time difference between airplane_in and airplane_out, and can calculate the time of the air bridge operation by calculating the time difference between bridge_in and bridge_out. Therefore, the present invention can accurately and quickly determine the time when the aircraft and the air bridge enter and leave through the system, thereby saving manpower and quickly tracing back the time point that needs to be confirmed, greatly reducing working time and costs.

The present invention provides an algorithm for determining the time when aircraft and air bridges enter and leave. Through a multi-level determination design, the problem of insufficient output accuracy caused by detection errors can be reduced, and the accuracy can be improved by more than 30%. In addition, it can provide real-time information on the entry and exit of aircraft air bridges. In addition to providing the user unit with data for the charging system for aircraft stays, it can also allow the user unit to control the status of each terminal in real time.

The camera device of the present invention can be set up in front of the apron. As long as the object to be detected is not largely obscured, a small amount of data set can be added as training data to expand the present invention to perform detection of multiple operating procedures. In addition, the combination of deep learning object detection and image processing foreground separation technology allows the present invention to allow object detection to have better result output accuracy even when the accuracy is insufficient due to a small data set. Furthermore, the state of object movement can be obtained through deep learning object detection and tracking algorithms. In addition, data such as airport environment, weather changes, light and shadow changes, etc. can be added according to customer needs to provide accuracy.

The above is only an example to illustrate the preferred implementation of the present invention, and is not intended to limit the scope of implementation. All simple substitutions and equivalent changes made according to the scope of the patent application of the present invention and the content of the patent specification are within the scope of the patent application of the present invention.

10: System 100: Camera 110: Image recognition unit 111: Processing unit 112: Deep learning unit 120: Processing unit 130: Image combination unit 502, 504, 506: Box A, B, C, D: Steps

Figure 1 is a schematic diagram of a system for detecting the behavior of aircraft on the apron of the present invention; Figure 2 is an operation flow chart of an embodiment of the present invention; Figure 3 is an operation flow chart of deep learning technology and foreground-background separation technology in an embodiment of the present invention. Figures 4A and 4B are schematic diagrams of images obtained by deep learning technology and foreground-background separation technology in an embodiment of the present invention. Figure 5 is a schematic diagram of a situational model and judgment conditions in an embodiment of the present invention. Figures 6 to 11 are schematic diagrams of the present invention when judging the entry and departure of an aircraft. .

A, B, C, D: Steps

Claims (8)

A method for detecting the behavior of an aircraft on an apron, the method comprising: (A)     identifying at least one apron image to obtain a state condition representing an aircraft area and a sky bridge area in the at least one apron image, wherein the at least one apron image is generated by at least one camera device; (B)      judging that the at least one apron image belongs to one of a plurality of scenario modes according to the state condition; (C)      judging the spatial relationship and temporal relationship between an aircraft object existing in the aircraft area and a sky bridge object existing in the sky bridge area in the at least one apron image according to a plurality of judgment conditions set in the scenario mode, and generating a judgment result; and (D)    Output an event image formed by combining the judgment result with the at least one apron image. The method for detecting the behavior of an aircraft on an apron as described in claim 1, the step (A) comprises: Using at least one of a deep learning technology and a foreground-background separation technology, identifying the spatial relationship and temporal relationship between the aircraft object in the aircraft area in the at least one apron image and the apron, and identifying the spatial relationship and temporal relationship between the air bridge object in the air bridge area and the apron, to obtain the state condition. A method for detecting the behavior of an aircraft on a ramp as described in claim 2, wherein the foreground-background separation technique is implemented based on Gaussian Mixture. In the method for detecting the behavior of an aircraft on a ramp as described in claim 2, the foreground and background separation technique includes binarizing the at least one apron image and performing noise filtering based on a threshold. A method for detecting the behavior of an aircraft on a ramp as described in claim 1, wherein the scenario modes include an aircraft entry mode, an aircraft departure mode, an air bridge entry mode, an air bridge departure mode, and an output mode. A method for detecting the behavior of an aircraft on an apron as described in claim 5, wherein the state relationship of the aircraft entry mode, the aircraft departure mode, the air bridge entry mode, the air bridge departure mode, and the output mode corresponds to the spatial relationship and the temporal relationship between the aircraft object and the air bridge object, and is defined as follows: The aircraft entry mode is defined as the aircraft object not entering and not leaving the at least one apron image; The aircraft departure mode is defined as the aircraft object entering and not leaving the at least one apron image; The air bridge entry mode is defined as the aircraft object entering the at least one apron image, the aircraft object has not left, and the air bridge object has not yet docked with the aircraft object; The air bridge exit mode is defined as the aircraft object entering the at least one apron image, the air bridge object has docked with the aircraft object, and the air bridge object has not left the aircraft object; and The output mode is defined as the aircraft object entering the at least one apron image and the aircraft object has left. As described in claim 6, the method for detecting the behavior of an aircraft on the apron, the judgment conditions include: the moving direction and speed of the aircraft object in the aircraft area or the sky bridge object in the sky bridge area, and whether the aircraft object and the sky bridge area exist. The judgment conditions are also used to determine whether to output the event image. A system for detecting the behavior of an aircraft on an apron, for implementing the method for detecting the behavior of an aircraft on an apron as described in any one of claims 1 to 7.