TWI828368B - Method and system for detecting aircraft behavior on the tarmac - Google Patents

Abstract

The present invention provides a method for detecting the behavior of an aircraft on an apron, which includes: identifying at least one apron image to obtain a state condition represented as an aircraft area and an air bridge area; and determining whether the apron image belongs to multiple apron images based on the state conditions. One of the situational modes; based on the multiple judgment conditions set in the situational mode, the spatial and temporal relationships between the aircraft objects in the aircraft area in the apron image and the airbridge objects in the airbridge area are judged, and judgments are made accordingly. The result; and, output the event image formed by combining the judgment result with the tarmac image. The invention can accurately and immediately obtain the time when aircraft and air bridges enter and leave the apron.

Description

Method and system for detecting aircraft behavior on the tarmac

The present invention relates to a method and system for detecting the behavior of aircraft on an apron, and in particular to a method for detecting the entry, departure, and engagement times of aircraft and air bridges.

When the aircraft taxis onto the tarmac, the captain relies on manual guidance or a guidance system to land and stop at the correct location. A common guidance system will identify the aircraft type and use signals to guide the captain to correct the taxiing route, slow down, and stop at the nose gear stop line. On the other hand, manual guidance uses a trailer to move the aircraft position, that is, the trailer driver operates the aircraft's stopping route and parking position.

Since the time an aircraft enters and leaves the airport will affect landing fees and parking fees, the airport needs to accurately record the aforementioned times. In response to the large number of aircraft taking off and landing, the airport can use a large amount of manpower to solve the problem, but this will increase the cost accordingly and raise doubts about the accuracy of the records. In previous technologies, calipers operated based on Internet of Things (IoT) technology were used to clamp the tires or other contact points of the aircraft and automatically send time information. However, this method still relies on manual transmission of time information and can only provide basic time information, so it has not yet reached a fully automated process.

In view of the shortcomings of the prior art, the present invention provides a method for detecting aircraft behavior on the tarmac. Through object detection technology and foreground and background separation technology, the present invention can accurately and quickly obtain the entry, departure and connection times of aircraft and air bridges.

The method provided by the present invention for detecting the behavior of aircraft on an apron identifies at least one apron image to obtain a state condition represented as an aircraft area and an air bridge area in at least one apron image, wherein the apron The image is generated by at least one camera device; according to the status condition, it is judged that at least one apron image belongs to one of a plurality of situation modes; according to a plurality of judgment conditions set in the situation mode, it is judged that at least one apron image exists in the aircraft area The spatial relationship and the time relationship between an aircraft object and an air bridge object existing in the air bridge area, and a judgment result is generated; and an event image formed by combining the judgment result with at least one apron image is output.

In some embodiments, the present invention utilizes at least one of deep learning technology and foreground and background separation technology to identify the spatial and temporal relationships between aircraft objects in the aircraft area and the apron in at least one apron image, and identify The spatial and temporal relationships between the air bridge objects and the apron in the air bridge area are used to obtain status conditions.

In some embodiments, the plurality of situational modes include aircraft entry mode, aircraft departure mode, air bridge entry mode, air bridge departure mode, and exit mode.

In some embodiments, the status relationship between aircraft entry mode, aircraft departure mode, air bridge entry mode, air bridge departure mode, and output mode is the spatial relationship and time relationship between the aircraft object and the air bridge object, and is defined as: The aircraft entry mode is defined as, at least one aircraft object in the apron image has not entered, and the aircraft object has not left; the aircraft departure mode is defined as, at least one aircraft object in the apron image has entered, and the aircraft object has not left; Air Bridge The entry mode is defined as at least one aircraft object entering the apron image, the aircraft object has not yet left, and the air bridge object has not yet connected to the aircraft object; the air bridge departure mode is defined as at least one aircraft object entering the apron image, The air bridge object is connected to the aircraft object, and the air bridge object has not left the aircraft object; and the output mode is defined as the aircraft object entering and the aircraft object has left in at least one apron image.

In some embodiments, the judgment conditions include: the moving direction and speed of aircraft objects in the aircraft area or air bridge objects in the air bridge area, and whether the aircraft objects and air bridge areas exist. These judgment conditions are also used to determine whether to output an event. image.

The present invention also provides a system for implementing the aforementioned method for detecting the behavior of an aircraft on an apron.

According to the above, the present invention uses object detection technology and foreground and background separation technology to identify the aircraft area and air bridge area in the tarmac image, thereby determining the relationship between the aircraft and the air bridge in time and space. Therefore, the present invention can accurately and quickly determine the time of entry, departure and connection of aircraft and air bridges. In this way, the present invention has competitive advantages in multiple aspects such as automated procedures, audit capabilities that can quickly trace back time data, and reduced operating time and labor costs.

The embodiments of the present invention will be further explained below with reference to relevant drawings. Wherever possible, the same reference numbers are used in the drawings and description to refer to the same or similar components.

First, please refer to Figure 1, which is a schematic diagram of the system used to detect the behavior of aircraft on the tarmac. The system 10 of the present invention for detecting the behavior of aircraft on the tarmac includes at least one camera device 100, an image recognition unit 110, a processing unit 120, and an image combining unit 130. The camera device 100 may be installed at the airside of the airport, and the number of installations may be selected according to practical requirements. The camera device 100 is used to capture at least one apron image of the apron. The image recognition unit 110 is connected to the camera devices 100 with signals to obtain each tarmac image captured by each camera device 100 . The image recognition unit 110 is used to identify the aircraft area and the air bridge area in the apron image, and generate a status condition accordingly. The aircraft area represents the area where the aircraft object enters and stops, while the air bridge area represents the area where the air bridge object approaches and engages the aircraft object. The status condition represents the time and space relationship between the aircraft object and the air bridge object. The processing unit 120 of the signal connection image recognition unit 110 is used to determine whether it belongs to one of the plurality of situation modes according to the status condition; the processing unit 120 also judges the tarmac image according to the plurality of judgment conditions of the determined situation mode to generate a judgment result. . The judgment results include the spatial relationship and time relationship between an aircraft object existing in the aircraft area and an air bridge object existing in the air bridge area, such as the entry time of the aircraft object marked in the upper left corner of Figures 6 to 11 and departure time, as well as the entry time and departure time of the air bridge. Image combining unit 130, signal connection processing unit 120; the image combining unit 130 is used to combine the judgment result and the apron image to output the event image, that is, the apron image shown in Figures 6 to 11 and at least one of the aforementioned judgment results. message.

In some embodiments, the image recognition unit 110 includes at least one of an image processing unit 111 and a deep learning unit 112 . The image processing unit 111 performs image processing on the tarmac image based on foreground and background separation technology, and outputs status conditions. The deep learning unit 112 identifies the tarmac image based on the deep learning model and outputs the status condition; or the deep learning unit 112 identifies the tarmac image processed by the image processing unit 111 based on the deep learning model and outputs the status condition.

Please refer to Figure 2. Next, the method for detecting the behavior of an aircraft on the apron of the present invention will be described. The steps of the behavior detection method are as follows:

In step (A), multiple apron images are first identified to obtain the status conditions of the aircraft area and the air bridge area in the multiple apron images. The multiple apron images are generated by camera devices.

In addition, please refer to Figure 3 to establish deep learning technology and/or foreground and background separation technology to identify the spatial and temporal relationships between aircraft objects in the aircraft area and the apron in the apron image, and the relationship between the air bridge objects in the air bridge area and The spatial relationship and time relationship of the apron are used to obtain the status conditions of aircraft objects.

In some embodiments, the aforementioned recognition process is implemented based on at least one of deep learning technology and foreground and background separation technology.

The method of establishing deep learning technology includes preparatory work to establish a deep learning model. In the preparatory work, first collect relevant public data sets of different types of aircraft and operating vehicles recorded in the airport terminal, and collect images of the apron actually detected by the present invention; then, use labeling tools such as Label Image to classify these images Label the data and define categories, such as aircraft nose and air bridge, etc.; then input the labeled data into the deep learning neural network for model training, and output the initial deep learning model. In response to different detection locations, customized model optimization can be carried out, for example, to improve the accuracy or speed of judgment. Through the above steps and model optimization, the preparatory work of the deep learning model can be completed.

The deep learning of the present invention is a deep learning model for object detection. After the tarmac image obtained by the camera device is read and imported into the deep learning model of object detection, the aforementioned deep learning model can make inferences using algorithms related to object detection (including but not limited to YOLO series or SSD series algorithms) and detect the type and location of aircraft.

In some embodiments, the foreground and background separation technology is implemented based on Gaussian Mixture. The foreground and background separation technology can perform binarization and noise processing on the tarmac image. For example, the present invention can filter the noise in the tarmac image according to a threshold set to 125, thereby improving the accuracy of identification. Alternatively, the present invention can also obtain the appearance of the foreground object in the tarmac image through a contour detection algorithm. Specifically, by retaining foreground objects occupying more than 2% of the image frame for subsequent determination, and ignoring foreground objects occupying less than 2% of the image frame, the present invention can effectively reduce the noise of the tarmac image.

Based on the above, deep learning technology can detect and identify aircraft categories and aircraft positions in tarmac images, while foreground and background separation technology can generate foreground objects in tarmac images. Next, please refer to Figures 4A and 4B, which respectively illustrate images of aircraft and apron detected by deep learning, and images of foreground objects produced based on the foreground detection algorithm. Since the present invention can compare two images obtained based on deep learning technology and foreground and background separation technology, it can improve the accuracy of analysis of object positions.

By reading and identifying tarmac images generated in time series, the temporal and spatial relationship between aircraft objects and air bridge objects can be obtained, that is, state conditions; state conditions include but are not limited to aircraft, air bridges, and foreground objects. category, movement direction and location. Next, the present invention will determine the situation mode according to the status condition of the tarmac image in step (B). Referring to Figures 2 and 5, in some embodiments, the plurality of situational modes may include an aircraft entry mode, an aircraft departure mode, an air bridge entry mode, an air bridge departure mode, and an output mode. Through the spatial relationship and time relationship between the aircraft object and the air bridge object, it can be determined which of the aircraft entry mode, aircraft departure mode, air bridge entry mode, air bridge departure mode and output mode should be entered.

The judgment logic and discriminant (i.e. judgment condition) of the above multiple situational modes are as follows:

1. When there is no aircraft object entering the tarmac image (the system has a flag about the aircraft entering: False), and there is no aircraft object leaving (the system has a flag about the aircraft leaving: False), the entering aircraft enters the discriminant mode. ;

2. When an aircraft object enters the apron image (the system has a flag about the aircraft entering: True) and the aircraft object has not left (the system has a flag about the aircraft leaving: False), enter the aircraft departure criterion;

3. The aircraft entering the air bridge entry mode is defined as an aircraft object entering the apron image (the system has a flag about the aircraft entering: True), and the aircraft object has not yet left (the system has a flag about the aircraft leaving: False). , and the air bridge object has not yet connected to the aircraft object (the system has a flag about the air bridge connecting aircraft: False), enter the air bridge and enter the discriminant;

4. When an aircraft object enters the apron image (the system's flag about the aircraft entering is displayed: True), the air bridge object is connected to the aircraft object (the system's flag about the air bridge connecting aircraft is displayed: True), and the air bridge object is empty. The bridge object has not left the aircraft object (the system has a flag about the air bridge leaving the aircraft: False) and enters the air bridge departure judgment;

5. When an existing aircraft object in the apron image has entered (the system has a flag about the aircraft entering: True) and has left (the system has a flag about the aircraft leaving: True), enter the initialization process of the output result.

Please refer to Figure 2. In step (C), the multiple judgment conditions set in the situation mode are used to determine the spatial relationship and time relationship between the aircraft objects in the aircraft area and the air bridge objects in the air bridge area in the apron image. Judgment results.

Step (D) Outputs the event image that is combined with the judgment result and the apron image. Please refer to Figures 6 to 11, which are schematic diagrams of event images in different embodiments of the present invention. It should be noted that Figures 6 to 11 are all marked with time records, including: Gate (gate), aircraft_in (aircraft entering), aircraft_out (aircraft leaving), bridge_in (air bridge entering), bridge_out (air bridge) Leave). Among them, Gate will display the time of aircraft entry, aircraft departure, air bridge entry, and air bridge departure in a time sequence. Other time records in these schemas will be generated when the judgment conditions of each situation mode are met, and displayed in the corresponding time area; in other words, other time records are the times of the judgment results.

In some embodiments, the aforementioned judgment conditions may also include: the moving direction and speed of aircraft objects in the aircraft area or air bridge objects in the air bridge area, or whether there are aircraft objects or air bridge objects, etc. When generating a judgment result, one or a combination of the above can be selected as a limiting standard for the degree of different judgment results. The aforementioned limiting standards can include the most stringent, normal, and loose, as detailed below.

If the most stringent restriction standard is adopted, all the conditions of the moving direction, moving speed of the above-mentioned aircraft object or air bridge object, and whether there is a foreground object must be established, and the above conditions have not changed for N seconds, then the judgment result will be output and based on it. Combined into event images. N is a positive integer. If normal limiting standards are used, as long as the two conditions of movement direction and movement speed of aircraft objects in the aircraft area or air bridge objects in the air bridge area are established, the judgment results will be output and combined into event images. If loose standards are adopted, as long as the condition of the moving direction of aircraft objects in the aircraft area or air bridge objects in the air bridge area is met, the judgment results will be output and combined into event images. Of course, the above conditions are only exemplary, and the present invention should not be limited by the above conditions.

In some embodiments, the criterion for determining the moving speed of the so-called aircraft object in the aircraft area or the air bridge object in the air bridge area may be determined based on whether the moving pixel value exceeds a threshold. Similarly, the so-called criterion for judging the moving direction of an aircraft object in an aircraft area or an air bridge object in an air bridge area can be whether the moving direction of an aircraft object or air bridge object is consistent with the movement of a preset aircraft object or air bridge object. in the same direction. In addition, the so-called criterion for judging whether there is an aircraft object or an air bridge object can be whether the detection frame set by deep learning has more than 80% of the foreground objects occupying it.

Next, the time recording process of the entry and departure of the aircraft will be explained through Figures 6 to 11.

As shown in the upper right corner of Figures 6 and 7, analysis_airplane_in represents that the method or system of the present invention has entered the aircraft entry judgment formula. The brackets on the right side of analysis_airplane_in show the judgment conditions, such as the above-mentioned moving direction and moving speed. As shown in Figure 6, when the judgment condition is [False, True, False], it means that no aircraft has yet entered the aircraft area (i.e., the apron gate). As shown in Figure 7, when the judgment condition is [True, True, True], it means that an aircraft enters the apron gate; at this time, the judgment result of aircraft_in is generated on the left side and displayed as aircraft_in: 2022-09 -28 14:56:58, indicating that the time when the aircraft entered the apron gate was 14:56:58. Among them, box 502 displays the position of the aircraft area, and box 504 displays the position of the aircraft nose, which allows the system to determine the type of aircraft and the time and space relationship of the aircraft entering the aircraft area through the image.

As shown in the upper right corner of Figure 8, analysis_bridge_in represents that the method or system of the present invention enters the air bridge and enters the judgment formula. When the judgment condition of analysis_bridge_in is [True, True,True], the judgment result of bridge_in is generated and displayed as bridge_in: 2022-09-28 14:59:07, indicating that the time when the air bridge connected to the aircraft is 14:5 Nineteen minutes and seven seconds. Among them, box 506 represents the air bridge area, that is, the position where the aircraft connects to the air bridge, and is used to detect whether the air bridge is within the box and complete the air bridge connection to the aircraft. Figure 9 is an image captured by another camera device. The judgment condition of analysis_bridge_out is [False, False, False], indicating that the air bridge has not been separated from the aircraft.

As shown in the upper right corner of Figure 10, analysis_bridge_out represents the method or system of the present invention entering the air bridge departure judgment formula. Specifically, when the judgment condition is [True, True, True], the judgment result of bridge_out is generated and displayed as bridge_out: 2022-09-28 16:57:40, which means that the time when the air bridge left the aircraft is sixteen o'clock Fifty-seven minutes and forty seconds.

Analysis_airplane_out shown in the upper right corner of Figure 11 represents that the method or system of the present invention enters the aircraft departure judgment formula. Specifically, when the judgment condition is [True, True, True], the system generates the judgment result of aircraft_out and displays it as aircraft_out: 2022-09-28 16:59:16, which means that the time when the aircraft left the apron aircraft area is Sixteen fifty-nine minutes and sixteen seconds.

After obtaining the above multiple time records, the method or system of the present invention can accurately calculate the time of the aircraft landing on the apron by calculating the time difference between aircraft_in and aircraft_out, and calculate the time difference between bridge_in and bridge_out to obtain the time of air bridge operation, so This invention can accurately and quickly determine the entry and departure times of aircraft and air bridges through the system, thereby saving manpower and quickly tracing back to the time points that need to be confirmed, greatly reducing working time and costs.

The present invention provides an algorithm for determining the entry and departure time of aircraft and air bridges. Through multi-level determination design, the problem of insufficient accuracy of output results caused by detection errors can be reduced, and the accuracy can be improved by more than 30%. It can also provide real-time information on the arrival and departure of aircraft on air bridges. In addition to providing the user with information on the charging system for aircraft stops, it can also allow the user to control the status of each terminal in real time.

The photography device of the present invention can be installed in front of the apron. As long as the object to be detected is not blocked by a large amount, a small amount of data sets can be supplemented as training data, and the present invention can be expanded to detect multiple operating procedures. In addition, by combining deep learning object detection and image processing foreground separation technology, the present invention can allow object detection to be less accurate due to less data sets, while still achieving better result output accuracy. Furthermore, the movement status of objects can be obtained through deep learning object detection and tracking algorithms. In addition, data such as airport environment, weather changes, light and shadow changes can be added according to customer needs to provide accuracy.

The above are only examples to illustrate the preferred embodiments of the present invention, and are not intended to limit the scope of implementation. All simple substitutions and equivalent changes made based on the patent scope of the present invention and the contents of the patent specification belong to the patent of the present invention. Application scope.

10:System 100:Camera device 110:Image recognition unit 111: Processing unit 112: Deep Learning Unit 120: Processing unit 130:Image combination unit 502, 504, 506: Box A, B, C, D: steps

Figure 1 is a schematic diagram of the system used to detect the behavior of aircraft on the tarmac; Figure 2 is an operation flow chart of one embodiment of the present invention; Figure 3 is an operation flow chart of deep learning technology and foreground and background separation technology in one embodiment of the present invention. Figures 4A and 4B are schematic diagrams of images obtained by deep learning technology and foreground and background separation technology in one embodiment of the present invention. Figure 5 is a schematic diagram of situation modes and judgment conditions in an embodiment of the present invention. Figures 6 to 11 are schematic diagrams of the present invention when determining the entry and departure of an aircraft. .

A, B, C, D: steps

Claims (8)

A method for detecting aircraft behavior on the tarmac, the method includes: (A) Identify at least one apron image to obtain a state condition represented as an aircraft area and an air bridge area in the at least one apron image, wherein the at least one apron image is generated by at least one camera device; (B) Based on the status condition, it is determined that the at least one apron image belongs to one of a plurality of situation modes; (C) Based on the multiple judgment conditions set in the situation mode, determine the spatial relationship and time between an aircraft object existing in the aircraft area and an air bridge object existing in the air bridge area in the at least one apron image relationship and produce a judgment result; and (D) Output an event image formed by combining the judgment result and the at least one apron image. As claimed in claim 1, the method for detecting the behavior of aircraft on the apron, the step (A) includes: Utilize at least one of a deep learning technology and a foreground and background separation technology to identify the spatial relationship and time relationship between the aircraft object in the aircraft area and the apron in the at least one apron image, and identify the air bridge area in the apron. The spatial relationship and time relationship between the air bridge object and the apron are used to obtain the status conditions. The method for detecting aircraft behavior on the tarmac as described in claim 2, wherein the foreground and background separation technology is implemented based on Gaussian Mixture. As for the method for detecting aircraft behavior on an apron as described in claim 2, the foreground and background separation technology includes binarizing the at least one apron image and noise filtering based on a threshold. The method for detecting aircraft behavior on the apron as described in claim 1, wherein the situation modes include an aircraft entry mode, an aircraft departure mode, an air bridge entry mode, an air bridge departure mode, and an output mode. The method for detecting the behavior of an aircraft on an apron as described in claim 5, wherein the status of the aircraft entry mode, the aircraft departure mode, the air bridge entry mode, the air bridge departure mode, and the output mode The relationship corresponds to the spatial relationship and time relationship between the aircraft object and the air bridge object, and are respectively defined as: The aircraft entry mode is defined as the aircraft object not entering and the aircraft object not leaving in the at least one apron image; The aircraft departure mode is defined as the aircraft object entering the at least one apron image and the aircraft object has not yet left; The air bridge entry mode is defined as when the aircraft object enters the at least one apron image, the aircraft object has not yet left, and the air bridge object has not yet connected to the aircraft object; The air bridge departure mode is defined as when the aircraft object enters the at least one apron image, the air bridge object is connected to the aircraft object, and the air bridge object does not leave the aircraft object; and The output mode is defined as the aircraft object entering and the aircraft object leaving in the at least one apron image. As for the method for detecting the behavior of aircraft on the apron as described in claim 6, the judgment conditions include: the moving direction and speed of the aircraft object in the aircraft area or the air bridge object in the air bridge area, And whether the aircraft object and the air bridge area exist, these judgment conditions are also used to decide whether to output the event image. A system for detecting the behavior of aircraft on the apron, used to implement the method for detecting the behavior of aircraft on the apron as described in any one of claims 1 to 7.