CN113344900B

CN113344900B - Airport runway intrusion detection method, airport runway intrusion detection device, storage medium and electronic device

Info

Publication number: CN113344900B
Application number: CN202110711431.9A
Authority: CN
Inventors: 崔磊; 王巍; 王谦; 肖旭; 赵晶晶
Original assignee: Beijing Sensetime Technology Development Co Ltd
Current assignee: Beijing Sensetime Technology Development Co Ltd
Priority date: 2021-06-25
Filing date: 2021-06-25
Publication date: 2023-04-18
Anticipated expiration: 2041-06-25
Also published as: CN113344900A

Abstract

The disclosure relates to an airport runway intrusion detection method, an airport runway intrusion detection device, a storage medium and an electronic device. The method includes acquiring a target image, a field of view of which includes an intrusion area determined based on an airport runway; cutting the target image to determine an image to be identified; performing target identification on the image to be identified based on a deep neural network to obtain first target position information, wherein the first target position information represents the position of a target in the image to be identified; obtaining at least two adjacent images to be recognized according to the cutting result, and determining second target position information based on the overlapping area between the adjacent images to be recognized and the first target position information, wherein the second target position information represents the position of a target in the target image; and determining the invasion result of the airport runway according to the second target position information and the invasion area. The intrusion detection method and the intrusion detection system can utilize the deep neural network to ensure the stability and reliability of intrusion detection.

Description

Airport runway intrusion detection method, airport runway intrusion detection device, storage medium and electronic device

Technical Field

The present disclosure relates to the field of intelligent airport management and control technologies, and in particular, to an airport runway intrusion detection method, an airport runway intrusion detection apparatus, a storage medium, and an electronic device.

Background

With the rapid development of global aviation business, the flight flow is continuously increased, the conflict between the air and the ground is more frequent, and particularly, the problem of runway safety is more serious. Runway safety is the basis for air transportation system safety, and airport runway incursion is a typical runway safety problem, and the resulting unsafe events are also increasing year by year. In the related art, the invasion of the airport runway can be monitored by arranging additional sensing equipment, and the invasion of the airport runway can also be monitored based on a manual method or a traditional visual algorithm, wherein the former needs to be additionally arranged with hardware, and the latter is low in monitoring precision.

Disclosure of Invention

In order to solve at least one technical problem, the present disclosure provides a technical solution for detecting an airport runway intrusion.

According to some embodiments of the present disclosure, there is provided an airport runway intrusion detection method, the method comprising: cutting the target image, and determining an image to be recognized according to a cutting result, wherein the image to be recognized comprises at least two adjacent images; performing target identification on the image to be identified based on a deep neural network to obtain first target position information, wherein the first target position information represents the position of a target in the image to be identified; determining second target position information based on the overlapping area between the adjacent images and the first target position information, wherein the second target position information represents the position of a target in the target image; and determining an intrusion result of the airport runway according to the second target position information and the intrusion area. Based on the configuration, the operation amount of target recognition is reduced by cutting the target image on the premise of not additionally introducing hardware cost, the target recognition based on the deep neural network is performed on the cut image to be recognized, the target recognition accuracy can be improved by fully utilizing the advantages of the deep neural network, and the stability and reliability of intrusion detection are ensured.

In some possible embodiments, the determining second target position information based on the overlapping area between the adjacent images and the first target position information includes: mapping the first target position information to the target image to obtain target position mapping information; and performing fusion processing on the target position mapping information based on the overlapping area between the adjacent images to obtain second target position information. Based on the above configuration, the accuracy of the second target position information is improved by the fusion process.

In some possible embodiments, the target position mapping information includes at least two detection frames, each detection frame represents a position of a corresponding target, and the fusion processing on the target position mapping information based on a coincidence area between the adjacent images to obtain the second target position information includes: acquiring any first detection frame and any second detection frame, wherein the first detection frame and the second detection frame are two detection frames intersected in the target position mapping information; acquiring the intersection area of the first detection frame and the second detection frame; determining a first image to be recognized according to the first detection frame, wherein the first detection frame is obtained by mapping first target position information corresponding to the first image to be recognized to the target image; determining a second image to be recognized according to the second detection frame, wherein the second detection frame is obtained by mapping first target position information corresponding to the second image to be recognized to the target image; acquiring the area of the overlapping area of the first image to be recognized and the second image to be recognized; in response to the fact that the intersection area and the area of the overlapping area accord with a preset condition, fusing the first detection frame and the second detection frame; and obtaining the second target position information according to the fusion result. Based on the configuration, for any two intersected detection frames in the target position mapping information, whether the two intersected detection frames represent the same target or not can be accurately judged according to the coincidence condition of the related images to be identified, and the judgment accuracy is high, so that redundant detection frames do not exist in the obtained second target position information, and the accuracy of the second target position information is improved.

In some possible embodiments, the second target location information includes at least one detection box, and the determining the intrusion result of the airport runway according to the second target location information and the intrusion area includes: determining a detection frame intersecting the intrusion area in the second target position information as a first target detection frame; in response to the first number of object detection boxes being zero, determining that there is no intrusion of an object into the airport runway; and determining that the condition that the target invades the airport runway exists in response to the number of the first target detection frames being not zero. Based on the configuration, whether the situation that the target invades the airport runway exists can be judged through the intersection situation of the detection frame and the invasion area in the second target position information, the criterion is simple, the execution efficiency is high, and the invasion detection result can be obtained quickly and accurately.

In some possible embodiments, the target image is a current frame image of a surveillance video stream, the field of view of the target image further includes a real-time identification area determined based on an airport runway, and the method further includes: determining a detection frame intersected with the real-time identification area in the second target position information as a second target detection frame; tracking the second target position information based on the surveillance video stream in response to the number of second target detection frames being zero; and responding to the condition that the number of the second target detection frames is not zero, updating the target image according to the next frame image of the monitoring video stream, carrying out target identification on the updated target image, and determining the intrusion result of the airport runway based on the identification result. Based on the configuration, the starting frequency of target identification can be reduced in combination with target tracking, the burden of electronic equipment is reduced, and the purpose of real-time intrusion detection is achieved.

In some possible embodiments, the cutting the target image and determining the image to be recognized according to the cutting result includes: determining each image in the cutting result as the image to be identified; or, according to the invasion area, determining the image to be identified in the cutting result. Based on the configuration, the calculation pressure of the electronic equipment can be reduced, and the target which can generate the intrusion behavior in the target image can be quickly identified.

In some possible embodiments, the deep neural network includes a feature extraction network, a classification recognition network, and a recognition result output network, and the performing target recognition on the image to be recognized based on the deep neural network to obtain first target location information includes: performing feature extraction on the image to be recognized based on the feature extraction network to obtain a feature map; classifying and identifying the characteristic graph based on the classification and identification network to obtain a classification and identification result, wherein the classification and identification result represents a target in the image to be identified and the probability of the target belonging to a preset target; and outputting the network to screen the classification recognition result based on the recognition result to obtain the first target position information. Based on the configuration, the target recognition can be carried out on the image to be recognized through the deep neural network, the position of the target in the image to be recognized is obtained, the interference is not easy to happen, and the recognition result is stable and reliable.

In some possible embodiments, the method further comprises: judging whether a target which is not identified exists or not according to the second target position information; in response to the presence of the missing identified target, updating the deep neural network in accordance with the missing identified target. Based on the configuration, the missing recognition probability can be reduced by continuously optimizing the deep neural network, and the accuracy and the stability of the target recognition of the deep neural network are improved.

In some possible embodiments, the second target location information includes at least one detection box, the method further comprising: acquiring a detection frame in the second target position information; and displaying the detection frame on the target image. Based on the configuration, the target image and the target in the target image are visually expressed, so that airport management personnel can conveniently know the invasion condition of the airport runway in real time.

In some possible embodiments, the method further comprises: and if the target invades the airport runway, sending alarm information. Based on the configuration, airport management personnel can conveniently know the invasion condition of the airport runway in time, and the safety risk is reduced.

In accordance with further embodiments of the present disclosure, there is provided an airport runway incursion detection apparatus, comprising: the system comprises a target image acquisition module, a target image acquisition module and a target image processing module, wherein the view field of the target image comprises an invasion area determined based on an airport runway; the image segmentation module is used for cutting the target image and determining an image to be identified according to a cutting result, wherein the image to be identified comprises at least two adjacent images; the target identification module is used for carrying out target identification on the image to be identified based on a deep neural network to obtain first target position information, and the first target position information represents the position of a target in the image to be identified; the recognition result integration module is used for determining second target position information based on the overlapping area between the adjacent images and the first target position information, and the second target position information represents the position of a target in the target image; and the invasion judging module is used for determining the invasion result of the airport runway according to the second target position information and the invasion area.

In some possible embodiments, the identification result integration module includes: the mapping unit is used for mapping the first target position information to the target image to obtain target position mapping information; and the fusion unit is used for carrying out fusion processing on the target position mapping information based on the overlapping area between the images to obtain the second target position information.

In some possible embodiments, the target position mapping information includes at least two detection frames, each of the detection frames represents a position of a corresponding target, and the fusion unit includes: a fusion target obtaining unit, configured to obtain any first detection frame and any second detection frame, where the first detection frame and the second detection frame are two detection frames that intersect in the target position mapping information; an intersection area acquisition unit configured to acquire an intersection area of the first detection frame and the second detection frame; a first to-be-recognized image determining unit, configured to determine a first to-be-recognized image according to the first detection frame, where the first detection frame is obtained by mapping first target position information corresponding to the first to-be-recognized image to the target image; the second image to be recognized determining unit is used for determining a second image to be recognized according to the second detection frame, and the second detection frame is obtained by mapping first target position information corresponding to the second image to be recognized to the target image; an overlapping area acquiring unit, configured to acquire an area of an overlapping area of the first image to be recognized and the second image to be recognized; the fusion execution unit is used for fusing the first detection frame and the second detection frame in response to the fact that the intersection area and the area of the overlapping area accord with a preset condition; and the second target position information determining unit is used for obtaining the second target position information according to the fusion result.

In some possible embodiments, the second target location information includes at least one detection box, and the intrusion determination module includes: a first target detection frame determining unit configured to determine, as a first target detection frame, a detection frame that intersects the intrusion area in the second target position information; an intrusion judgment unit, which is used for responding to the first target detection frame number being zero and judging that no target intrudes into the airport runway; and determining that the condition that the target invades the airport runway exists in response to the number of the first target detection frames being not zero.

In some possible embodiments, the target image is a current frame image of a surveillance video stream, the field of view of the target image further includes a real-time identification area determined based on an airport runway, and the apparatus further includes: a second target detection frame determining module, configured to determine, as a second target detection frame, a detection frame that intersects with the real-time identification area in the second target location information; a tracking module, configured to track, based on the surveillance video stream, the second target position information in response to that the number of the second target detection frames is zero; and the real-time identification module is used for responding that the number of the second target detection frames is not zero, updating the target image according to the next frame image of the monitoring video stream, carrying out target identification on the updated target image, and determining the intrusion result of the airport runway based on the identification result.

In some possible embodiments, the image segmentation module is configured to determine each image in the cutting result as the image to be identified; or, according to the invasion area, determining the image to be identified in the cutting result.

In some possible embodiments, the deep neural network includes a feature extraction network, a classification recognition network, and a recognition result output network, and the target recognition module includes: the feature extraction unit is used for extracting features of the image to be identified based on the feature extraction network to obtain a feature map; the recognition unit is used for carrying out classification recognition on the feature map based on the classification recognition network to obtain a classification recognition result, and the classification recognition result represents a target in the image to be recognized and the probability that the target belongs to a preset target; and the recognition result output unit is used for screening the classification recognition result based on the recognition result output network to obtain the first target position information.

In some possible embodiments, the apparatus further includes a neural network optimization module, configured to determine whether there is a missing identified target according to the second target location information; in response to the presence of the missing identified target, updating the deep neural network in accordance with the missing identified target.

In some possible embodiments, the second target location information includes at least one detection frame, and the apparatus further includes a display module configured to obtain the detection frame in the second target location information; and displaying the detection frame on the target image.

In some possible embodiments, the apparatus further comprises an alert module for issuing an alert message in response to the presence of an intrusion of the object into the airport runway.

According to still further embodiments of the present disclosure, there is provided an electronic device comprising at least one processor, and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements a method of airport runway intrusion detection according to any of the first aspects by executing the instructions stored by the memory.

According to still further embodiments of the present disclosure, there is provided a computer-readable storage medium having at least one instruction or at least one program stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement a method for airport runway intrusion detection according to any of the first aspects.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

In order to more clearly illustrate the embodiments of the present specification or the technical solutions and advantages of the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the specification, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 illustrates a flow chart of an airport runway intrusion detection method according to an embodiment of the present disclosure;

FIG. 2 illustrates a schematic view of a target image according to an embodiment of the present disclosure;

FIG. 3 shows a schematic diagram of a cut of a target image according to an embodiment of the present disclosure;

FIG. 4 illustrates a flowchart of a method of determining second target location information, according to an embodiment of the present disclosure;

FIG. 5 is a schematic flow chart diagram illustrating a target location mapping information fusion method according to an embodiment of the present disclosure;

FIG. 6 illustrates a flow chart of a method of reducing a target recognition start frequency based on target tracking according to an embodiment of the present disclosure;

FIG. 7 illustrates a flow diagram of a method of optimizing a deep neural network, in accordance with an embodiment of the present disclosure;

fig. 8 is a schematic flowchart illustrating a process of performing target recognition on the image to be recognized based on the deep neural network to obtain first target location information according to an embodiment of the present disclosure;

FIG. 9 shows a block diagram of an airport runway intrusion detection apparatus, according to an embodiment of the present disclosure;

FIG. 10 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure;

FIG. 11 shows a block diagram of another electronic device in accordance with an embodiment of the disclosure.

Detailed Description

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments in the present description, belong to the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of a, B, and C, and may mean including any one or more elements selected from the group consisting of a, B, and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

In an intelligent airport management and control scene, the invasion monitoring of an airport runway is needed, and the purpose of avoiding safety accidents is achieved. Intrusion monitoring on airport runways can be enhanced in the related art by arranging an additional sensor network, which obviously introduces additional hardware cost; the intrusion monitoring of the airport runways may also be based on a variety of methods, such as manual monitoring or traditional vision, but these monitoring are of low automation, have limited real-time and have limited accuracy in identifying objects entering the airport runways. Furthermore, the recognition accuracy of the target in the related art is easily influenced by related factors, and the stability is poor. For example, if the target in the target image is too small or too large, the lighting condition for capturing the target image is poor, or the weather environment is poor, the target recognition accuracy may be significantly reduced.

In view of this, the embodiments of the present disclosure provide an airport runway intrusion detection method, which relies on a deep neural network to identify targets in target images under various conditions without additionally introducing hardware cost, and has a high identification rate and high stability. By cutting the target image and carrying out target recognition on the cut image to be recognized, the calculation amount of target recognition can be reduced, and the recognition speed is improved. And judging whether the target invades the airport runway according to the recognition result of the image to be recognized, thereby achieving the purpose of carrying out high-accuracy and high-stability real-time invasion detection on the airport runway.

The airport runway intrusion detection method provided by the embodiment of the present disclosure may be executed by a terminal device, a server, or other types of electronic devices, where the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementations, the airport runway intrusion detection method may be implemented by a processor invoking computer readable instructions stored in a memory. The airport runway intrusion detection method according to the embodiment of the present disclosure will be described below by taking an electronic device as an execution subject. The airport runway intrusion detection method is implemented by a processor invoking computer readable instructions stored in a memory.

Fig. 1 shows a flow chart of an airport runway intrusion detection method according to an embodiment of the present disclosure, as shown in fig. 1, the method includes:

s10: an object image is acquired, and a field of view of the object image includes an intrusion area determined based on an airport runway.

In some possible embodiments, the target image may be from a surveillance video stream. The surveillance video stream may be obtained by an electronic device. Alternatively, the electronic device may obtain the monitoring video stream from another device, for example, the electronic device may obtain the monitoring video stream from an image capturing device, a monitoring device, or the like. The disclosed embodiments do not limit the source of the surveillance video stream. In an implementation scene, the cloud deck and the camera device of the airport can be reused to shoot the airport runway to obtain the monitoring video stream, so that the hardware cost can be saved, and the reuse rate of the airport device can be improved. In the embodiment of the present disclosure, intrusion detection may be performed on a single monitoring video stream, or intrusion detection may be performed on multiple monitoring video streams concurrently, which is not limited to this.

The target image is not limited in the embodiments of the present disclosure. In an embodiment, the target image may be a current frame image of a surveillance video stream, and the airport runway intrusion detection method according to the embodiment of the present disclosure is performed on each frame image of the surveillance video stream. In another embodiment, the target image may be an image corresponding to a frame meeting a preset condition in the monitoring video stream, and may also meet a real-time requirement for airport runway intrusion detection, and reduce a computational pressure of an electronic device to some extent. For example, the preset condition may be that an image corresponding to a key frame in the monitored video stream is determined as the target image, or an image whose image identifier meets a preset requirement in the monitored video stream is determined as the target image. For example, each frame of image in the surveillance video stream may be sequentially numbered to obtain a corresponding image identifier, and if the image identifier can be evenly divided by a preset integer, the image corresponding to the image identifier is determined as the target image.

The target image in the disclosed embodiments should include at least an intrusion area determined based on the airport runway, which may be located at the entrance, exit, edge of the airport runway, or around the airport runway. The specific position of the invasion area is not limited by the embodiment of the disclosure, and can be determined according to actual needs. Please refer to fig. 2, which illustrates a schematic diagram of a target image according to an embodiment of the present disclosure. The dotted line frame in the target image is an intrusion area, and if the target enters the intrusion area, it can be determined that the airport runway is intruded. The disclosed embodiments do not limit the target, for example, the target may be a vehicle, a pedestrian, an airplane, etc., and any object that may enter the above-mentioned intrusion area may be the target in the disclosed embodiments.

S20: and cutting the target image, and determining an image to be recognized according to a cutting result, wherein the image to be recognized comprises at least two adjacent images.

In the embodiment of the disclosure, the target image can be cut to obtain a plurality of images. The embodiment of the present disclosure does not limit the specific cutting manner, for example, the target image may be cut into 4 images, 9 images, or 16 images, and the sizes of the cut images may be the same or different, and the embodiment of the present disclosure is not limited specifically. In the embodiment of the present disclosure, there is an overlapping area between adjacent images. Please refer to fig. 3, which illustrates a schematic diagram of cutting a target image according to an embodiment of the present disclosure. Images 1 to 9 are cut out in fig. 3, and the elongated rectangles between adjacent images represent overlapping regions. Illustratively, image 1 and image 2 are two adjacent images, and the filled rectangle between image 1 and image 2 represents the overlapping area between image 1 and image 2; the

images

4 and 7 are two adjacent images, and the filled rectangle between the

images

4 and 7 characterizes the overlap area between the

images

4 and 7.

In one embodiment, each image in the cutting result may be determined as the image to be recognized.

In another embodiment, the image to be recognized may be determined in the cutting result based on the invaded region.

In order to reduce the computational stress of the electronic equipment and quickly identify a target which may generate an intrusion behavior in the target image, only an image at a position having a distance smaller than a preset threshold value from the intrusion area may be determined as an image to be identified. Taking fig. 3 as an example, if the invaded region is in the image 1 at the upper left corner, and the image 9 at the lower right corner is far away from the invaded region and can not be recognized, the images 1 to 8 are determined as the images to be recognized.

Based on the above configuration, it is possible to perform object recognition by selecting a partial image as an image to be recognized, quickly determine an object that may cause intrusion behavior, increase intrusion detection speed, and reduce electronic device stress.

S30: and performing target identification on the image to be identified based on a deep neural network to obtain first target position information, wherein the first target position information represents the position of a target in the image to be identified.

In the embodiment of the disclosure, each image to be recognized for target recognition corresponds to one piece of first target position information, and the first target position information can represent the recognition result of the target through the detection frame. If the first target position information does not comprise a detection frame, representing that no target exists in the image to be identified; if the first target location information includes at least one detection box, each detection box may be characterized by a row vector including 5 elements. For example, the row vector may include an upper left abscissa of the detection box, an upper left ordinate of the detection box, a lower right abscissa of the detection box, a lower right ordinate of the detection box, and a detection box score characterizing a probability that a target in the detection box is a preset target. The preset target may be an object that can be identified by the deep neural network and may cause an intrusion behavior, such as an airplane, a vehicle, a pedestrian, and the like, which is not limited by the embodiment of the present disclosure. The method for acquiring the first target location information is described in detail below, and is not described herein again.

S40: second target position information is determined based on the overlapping area between adjacent images and the first target position information, the second target position information representing the position of the target in the target image.

In the embodiment of the present disclosure, the first target position information represents a position of a target in a corresponding image to be recognized, and based on the first target position information and a position relationship between the image to be recognized and a target image, the second target position information may be determined, where the second target position information represents a position of the target in the target image.

In one embodiment, please refer to fig. 4, which illustrates a flowchart of a method for determining second target location information according to an embodiment of the present disclosure. The determining second target position information based on the overlapping area between the adjacent images and the first target position information includes:

s41: and mapping the first target position information to the target image to obtain target position mapping information.

In one embodiment, for any first target position information, the position relationship between the corresponding image to be recognized and the target image may be determined, and the first target position information is mapped to the target image based on the position relationship, so as to obtain the target position mapping information. For example, if the target recognition is performed on all 9 images in fig. 3, 9 pieces of first target position information are generated, and all the 9 pieces of first target position information are mapped to the target image, so that target position mapping information can be obtained.

In an embodiment, if the target position mapping information includes no detection frame or only one detection frame, the representation target image may include no target or only one target, and this case may not perform step S42. If the target location mapping information includes at least two detection frames, step S42 is performed. Based on the above configuration, the activation frequency of step S42 can be reduced, and the intrusion detection speed can be increased.

S42: and performing fusion processing on the target position mapping information based on the overlapping area between the adjacent images to obtain the second target position information.

Since the target image is cut, a situation that the same target appears in different images to be recognized at the same time may exist, taking fig. 3 as an example, a head of the target may appear in the image 1 and a tail of the target appears in the image 2, the first target position information corresponding to the image 1 includes the target, and the first target position information corresponding to the image 2 also includes the target, so that the target may be mistakenly regarded as two targets in the target position mapping information.

In an embodiment, a more reasonable area threshold may be set according to the overlapping area between each adjacent to-be-recognized image, the intersection area of any two detection frames in the target position mapping information where there is an intersection condition is calculated, and if the area is greater than the area threshold, the two detection frames are directly fused. Of course, the area threshold is not limited by the embodiments of the present disclosure.

In another embodiment, please refer to fig. 5, which illustrates a flowchart of a target location mapping information fusion method according to an embodiment of the present disclosure. The obtaining the second target position information by performing fusion processing on the target position mapping information based on the overlapping area between the adjacent images includes:

s421: and acquiring any first detection frame and any second detection frame, wherein the first detection frame and the second detection frame are two detection frames which are intersected in the target position mapping information.

The embodiment of the present disclosure determines one of two arbitrarily intersected detection frames as a first detection frame and the other as a second detection frame, and performs steps S422 to S427.

S422: and acquiring the intersection area of the first detection frame and the second detection frame.

In the embodiment of the present disclosure, the intersection area is an area of an overlapping area of the first detection frame and the second detection frame.

S423: and determining a first image to be recognized according to the first detection frame, wherein the first detection frame is obtained by mapping first target position information corresponding to the first image to be recognized to the target image.

Illustratively, if the first target position information corresponding to the image 1 to be recognized includes a detection frame 1, a detection frame 2, and a detection frame 3, and the first target position information is mapped to the target image to obtain a detection frame 10, a detection frame 20, and a detection frame 30, where the detection frame 30 is the first detection frame, and the image 1 to be recognized is the first image to be recognized.

S424: and determining a second image to be recognized according to the second detection frame, wherein the second detection frame is obtained by mapping first target position information corresponding to the second image to be recognized to the target image.

The second to-be-recognized image determination method is the same as the first to-be-recognized image determination method, and is not described herein again.

S425: and acquiring the area of the overlapping region of the first image to be recognized and the second image to be recognized.

S426: and fusing the first detection frame and the second detection frame in response to the fact that the intersection area and the area of the overlapped area meet a preset condition.

The embodiment of the present disclosure does not set the preset condition. In an embodiment, a ratio of the intersection area to the area of the overlap area may be calculated, and if the ratio is greater than a preset first ratio, it is determined that the target in the first detection frame and the target in the second detection frame represent the same target, and the first detection frame and the second detection frame are fused. In another embodiment, a second ratio may be determined according to the area of the overlapping area, and if the ratio of the intersection area to the area of the first detection frame is greater than the second ratio, it is determined that the target in the first detection frame and the target in the second detection frame represent the same target, and the first detection frame and the second detection frame are fused. The first ratio and the second ratio are not limited in the embodiments of the present disclosure, and an appropriate value may be selected according to actual needs. For example, a first minimum rectangle may be determined according to a first target in the first detection frame and a second target in the second detection frame, where the first minimum rectangle is a rectangle that can minimize an area included in both the first target and the second target, and the first minimum rectangle is determined as a fusion result. For another example, the second minimum rectangle may be determined directly from the first detection frame and the second detection frame, and the second minimum rectangle may be a rectangle having a smallest area included in both the first detection frame and the second detection frame, and the second minimum rectangle may be determined as the fusion result.

S427: and obtaining the second target position information according to the fusion result.

After the above processing is performed, different detection frames representing the same target in the target position mapping information may be fused into a detection frame uniquely corresponding to the target, thereby obtaining second target position information. Of course, the embodiments of the present disclosure do not limit the specific method of fusion.

Based on the configuration, for any two intersected detection frames in the target position mapping information, whether the two intersected detection frames represent the same target or not can be accurately judged according to the coincidence condition of the related images to be recognized, and the judgment accuracy is high, so that redundant detection frames do not exist in the obtained second target position information, and the accuracy of the second target position information is improved.

S50: and determining an intrusion result of the airport runway according to the second target position information and the intrusion area.

In one embodiment, the second target location information characterizes the location of the target in the target image by the detection box. If the second target position information does not comprise the detection frame, representing that no target exists in the target image, and judging that no target invades the airport runway; if the second target location information includes at least one detection box, each of the detection boxes may be characterized by a row vector including 5 elements, the exemplary row vector may include an upper left abscissa of the detection box, an upper left ordinate of the detection box, a lower right abscissa of the detection box, a lower right ordinate of the detection box, and a detection box score, where the detection box score characterizes a probability that a target in the detection box is a preset target.

If the second target position information includes at least one detection frame, the determining an intrusion result of the airport runway based on the second target position information and the intrusion area includes:

s51: and determining a detection frame intersecting the intrusion area in the second object position information as a first object detection frame.

In the disclosed embodiment, the objects in the first object detection box may be considered to produce an intrusion into the airport runway.

S52: and determining that no object invades the airport runway in response to the number of the first object detection frames being zero.

S53: and determining that the object invades the airport runway in response to the number of the first object detection frames being not zero.

In an embodiment, each first target detection box may also be fed back to the upper layer service platform, and further processing of the upper layer service platform is triggered, for example, taking the target as an airplane, the target may query relevant information of an intruding airplane, and perform airplane management and control, deployment, and the like, which is not limited in this disclosure.

In another embodiment, if it is determined that there is a target intruding into the airport runway, an alarm message is sent, where the alarm message may be a text message, a sound message, or a photoelectric message, and the alarm message may be sent directly or may be pushed to an electronic device corresponding to a relevant airport administrator in the form of a message, which is not limited in the embodiment of the present disclosure.

In another embodiment, if it is determined that there is an intrusion of the object into the airport runway, a corresponding alarm mode may be selected to alarm according to the type and/or the intrusion degree of the object. For example, when the target is identified on the image to be identified based on the deep neural network, if the target is identified, the type of the target is also determined, and the target is determined to be a person, an airplane, a vehicle and the like; if the target invades the airport runway, a corresponding alarm mode can be selected according to the type of the target to give an alarm, if a person invades the airport runway, the alarm is given by adopting the text information and the sound information, and if an airplane invades the airport runway, the alarm is given by adopting the text information. For another example, when it is determined that there is an intrusion of the target, the intrusion degree of the target may be determined, and whether the target is partially or completely located in the airport runway; if the target is partially in the airport runway, the alarm is given by using the character information, and if the target is completely in the airport runway, the alarm is given by using the character information and the sound information. Of course, the above two modes can also be used in combination, and are not described herein again.

In another embodiment, log information may be generated according to the invasion of the target into the airport runway, and the log information may be stored for performing related log analysis.

Based on the configuration, whether the situation that the target invades the airport runway exists can be judged through the intersection situation of the detection frame and the invasion area in the second target position information, the criterion is simple, the execution efficiency is high, and the invasion detection result can be obtained quickly and accurately.

In one embodiment, the detection frame in the second target position information may also be obtained; and displaying the detection frame on the target image. Referring to fig. 2, the target image shown in fig. 2 includes three targets (airplanes), and detection frames corresponding to the three targets are displayed on the target image, so that the target image and the targets in the target image are visually expressed through the detection frames, and airport managers can conveniently know the invasion condition of the airport runway in real time.

According to the airport runway intrusion detection method provided by the embodiment of the disclosure, on the premise of not additionally introducing hardware cost, the calculation amount of target identification is reduced by cutting the target image, the target identification based on the deep neural network is performed on the cut image to be identified, the target identification accuracy can be improved by fully utilizing the advantages of the deep neural network, and the stability and reliability of intrusion detection are ensured.

The airport runway intrusion detection method in the embodiment of the disclosure can carry out intrusion detection on at least one monitoring video stream in real time, the intrusion detection depends on target identification, and the target identification depends on a deep neural network, which may generate larger operation pressure. In order to reduce the starting frequency of target identification, and simultaneously detect the targets invading the airport runway in real time. The embodiment of the present disclosure may reduce the frequency of starting the target recognition based on the target tracking in a case where the second target location information includes at least one detection frame. Referring to fig. 6, a flowchart of a method for reducing target recognition start frequency based on target tracking according to an embodiment of the disclosure is shown, including:

and S60, determining a detection frame which is intersected with the real-time identification area in the second target position information as a second target detection frame.

The real-time identification area in the disclosed embodiments is determined based on the airport runway and is located in the field of view of the target image. The real-time recognition area may be considered to be an area that is a close distance from the invaded area or an area surrounding the invaded area, and the specific location of the real-time recognition area is not limited by the disclosed embodiments.

If the number of second target detection frames is zero, it can be considered that there is no target intruding into the real-time recognition area, and if the number of second target detection frames is not zero, it can be considered that there is a target intruding into the real-time recognition area.

If an object intrudes into the real-time identified area, it may be considered that the object is likely to be about to act to intrude into the airport runway, the behavior of the object should be closely monitored in real time; if the target does not invade the real-time identification area, the target can be considered not to have the behavior of invading the airport runway temporarily, the target can be tracked, and the real-time close monitoring can be carried out on the target after the target invades the real-time identification area. The embodiment of the disclosure can track the target based on a target tracking algorithm in the related art, the target tracking is faster than the target identification execution speed, the operation pressure generated on the electronic equipment is relatively small, the starting frequency of the target identification can be reduced by combining the target tracking, and the real-time monitoring of the behavior of the target invading the airport runway is not influenced.

And S70, responding to the fact that the number of the second target detection frames is zero, and tracking the second target position information based on the monitoring video stream.

The tracking of the second target position information by the embodiment of the present disclosure may be understood as tracking a target indicated by the second target position information. In this case, there is no target intruding into the real-time recognition area, and it may be considered that the target in the target image is far from the intruding area, and in this case, the target in the target image may be continuously tracked based on the monitoring video stream until the real-time recognition trigger condition is satisfied. In the embodiment of the present disclosure, the target image may be a current frame image of the monitoring video stream, and the continuous tracking process is to track the target indicated by the second target position information in a subsequent frame image. The embodiments of the present disclosure do not limit the real-time recognition triggering condition, and for example, the real-time recognition triggering condition may be that the target invades the real-time recognition area, or that a preset real-time recognition starting time is reached. When the target recognition triggering condition is satisfied, the target image may be updated according to a next frame image of the surveillance video stream, the updated target image may be subjected to target recognition, and an intrusion result of the airport runway may be determined based on the recognition result, that is, steps S10 to S50 may be repeatedly performed for the next frame image.

And S80, responding to the condition that the number of the second target detection frames is not zero, updating the target image according to the next frame image of the monitoring video stream, carrying out target identification on the updated target image, and determining the intrusion result of the airport runway based on the identification result.

This step is the same as the execution logic for which the target identification trigger condition is satisfied, and is not described herein again.

Based on the configuration, the starting frequency of target identification can be reduced in combination with target tracking, the burden of electronic equipment is reduced, and the purpose of real-time intrusion detection is achieved.

In the embodiment of the disclosure, the target image is subjected to target recognition based on the deep neural network, and the deep neural network can improve the target recognition capability through continuous optimization. Referring to fig. 7, a flowchart of a method for optimizing a deep neural network according to an embodiment of the present disclosure is shown, including:

and S60-1, judging whether the object which is not identified exists or not according to the second object position information.

The embodiment of the disclosure does not limit the discovery method of the target which is not identified, and the target can be found manually or according to the relevant airport management and control information.

S70-1, responding to the existence of the object which is not identified, and obtaining a second sample image and a label corresponding to the second sample image according to the object which is not identified; updating the deep neural network based on the second sample image and the label corresponding to the second sample image.

For the target that is not recognized, the embodiment of the present disclosure may determine the target image where the target that is not recognized is located as the second sample image, and determine a label corresponding to the second sample image, where the label represents a position of the target in the second sample image and a corresponding preset target.

In the embodiment of the present disclosure, the second sample image and the label corresponding to the second sample image form a training sample of the deep neural network, and the parameter of the deep neural network can be updated according to the training sample, so that the updated deep neural network has the capability of identifying the target that is not identified, the accuracy of identifying the target can be improved by continuously optimizing the deep neural network, and the probability of missing identification is reduced. Taking an aircraft as an example, in practical application, many factors such as the size, the position, the posture, the aircraft model, the weather condition, the illumination condition and the like of the aircraft in a target image may affect the target recognition accuracy to cause missing recognition, and by generating a training sample under the condition of missing recognition, the influence of the factors on the deep neural network can be gradually reduced by continuously optimizing the deep neural network, so that the accuracy and the stability of the target recognition performed by the deep neural network are improved.

The specific structure of the deep neural network is not limited in the embodiments of the present disclosure, and it may be obtained according to at least one of a YOLO (young Only Look Once) model, a multi-class Single-rod Detector (SSD), or a fast target recognition convolutional neural network R-CNN, fastR-CNN, and Faster RCNN series.

In one embodiment, the deep neural network may be obtained by improving a YOLO model as a base model. For example, the structure may include an input layer, a convolution layer, a normalization layer, an active layer, and the like, which are not described in detail in this disclosure.

In another embodiment, the deep neural network may include a feature extraction network, a classification recognition network, and a recognition result output network, which are sequentially connected. Please refer to fig. 8, which illustrates a schematic flow chart of performing target recognition on the image to be recognized based on the deep neural network to obtain first target location information according to an embodiment of the present disclosure, including:

and S101, extracting the features of the image to be identified based on the feature extraction network to obtain a feature map.

The specific structure of the feature extraction network is not limited in the embodiments of the present disclosure, and for example, the feature extraction network may include at least one convolutional layer, and each convolutional layer may be formed by a weight and a bias term of a convolutional kernel. In a convolutional layer, input data is first subjected to convolutional calculation through a convolutional kernel, and then an output feature map is obtained through an activation function, wherein the feature map can be a result obtained by combining feature information of a plurality of channels through convolution.

And S102, classifying and identifying the characteristic graph based on the classification and identification network to obtain a classification and identification result, wherein the classification and identification result represents the target in the image to be identified and the probability of the target belonging to a preset target.

The specific structure of the classification identification network is not limited in the embodiments of the present disclosure, and may include, for example, an upsampling layer, a convolution set, and a full-link layer. Taking the preset target as an airplane as an example, the target detected in the embodiment of the present disclosure is an airplane, and therefore the classification recognition result may directly represent the position of each target and the probability that each target belongs to the airplane, and if the probability is higher than a preset threshold, the target may be considered as an airplane.

And S103, screening the classification recognition result based on the recognition result output network to obtain the first target position information.

In one embodiment, the recognition result output network filters the classification recognition result through a preset probability threshold. If the probability corresponding to the target in the classification recognition result is greater than the probability threshold, the target is reserved, otherwise, the target is deleted. And generating first target position information according to the screened targets. The specific content of the first target location information refers to the foregoing, and is not described herein again. The specific numerical value of the probability threshold is not limited in the embodiments of the present disclosure, and may be set according to practical experience.

Based on the configuration, the target identification can be carried out on the image to be identified through the deep neural network, the position of the target in the image to be identified is obtained, the interference is not easy to happen, and the identification result is stable and reliable.

The following describes a process for training a neural network, showing a method of training a neural network, the method comprising:

s201, a first sample image and a label corresponding to the first sample image are obtained, wherein the label represents an object in the first sample image and whether the object belongs to a preset object or not.

The number and the acquisition method of the first sample images are not limited by the embodiments of the present disclosure. The target recognition capability of the deep neural network can be improved in a mode of enriching the first sample image. Taking the preset target as an airplane as an example, the first sample image can cover various airplane models, airplane positions, airplane postures and airplane visual angles, and the recognition capability of the deep neural network on airplanes in different natural environments can be improved by acquiring the first sample image under different weather conditions and different illumination conditions.

S202, extracting the characteristics of the first sample image based on the characteristic extraction network to obtain a sample characteristic diagram.

And S203, classifying and identifying the sample characteristic graph based on the classification and identification network to obtain a sample classification and identification result.

And S204, calculating loss according to the sample classification and identification result and the label.

Still taking the preset target as an airplane as an example, the loss may include a classification loss and a location loss, where the classification loss is used to measure a content difference between a probability that the target belongs to the airplane and a tag corresponding to the target in the sample classification recognition result. The smaller the classification loss, the more accurate the type identification of the aircraft is. During specific calculation, a preset classification loss function can be adopted for realization. The position loss is used for measuring the position difference between the position of the target in the sample classification recognition result and the position of the label corresponding to the target. The smaller the position loss, the more accurate the positioning of the aircraft. During specific calculation, a preset positioning loss function can be adopted.

S205, based on the loss, feeding back and adjusting parameters of the feature extraction network and the classification identification network.

In some possible embodiments, a back propagation method may be adopted, and the feature extraction network and the classification recognition network are jointly trained based on the loss until the loss converges to a preset value, and after the training is finished, it is confirmed that the deep neural network at this time meets the requirements, and the target detection can be achieved. The specific value and setting method of the preset value are not limited in this disclosure.

Based on the configuration, the deep neural network can be trained, so that the deep neural network has the capability of stably and accurately identifying the target.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted.

In addition, the present disclosure also provides an airport runway intrusion detection apparatus, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any one of the airport runway intrusion detection methods provided by the present disclosure, and the corresponding technical solutions and descriptions and corresponding descriptions in the methods section are omitted for brevity.

FIG. 9 illustrates an airport runway intrusion detection device according to an embodiment of the disclosure; as shown in fig. 9, the above apparatus includes:

a target image acquisition module 10, configured to acquire a target image, where a field of view of the target image includes an intrusion area determined based on an airport runway;

the image segmentation module 20 is configured to segment the target image, and determine an image to be identified according to a cutting result, where the image to be identified includes at least two adjacent images;

a target identification module 30, configured to perform target identification on the image to be identified based on a deep neural network to obtain first target location information, where the first target location information represents a location of a target in the image to be identified;

the recognition result integration module 40 is configured to obtain at least two adjacent images to be recognized according to a cutting result, and determine second target position information based on a coincidence region between the adjacent images and the first target position information, where the second target position information represents a position of a target in the target image;

and an intrusion determination module 50 for determining an intrusion result of the airport runway according to the second target location information and the intrusion area.

In some possible embodiments, the identification result integration module includes: a mapping unit, configured to map the first target location information to the target image to obtain target location mapping information; and a fusion unit configured to perform fusion processing on the target position mapping information based on a region of overlap between the adjacent images to obtain the second target position information.

In some possible embodiments, the target position mapping information includes at least two detection boxes, each of the detection boxes represents a position of a corresponding target, and the fusion unit includes: a fusion target obtaining unit, configured to obtain any first detection frame and any second detection frame, where the first detection frame and the second detection frame are two detection frames that intersect in the target position mapping information; an intersection area acquisition unit configured to acquire an intersection area of the first detection frame and the second detection frame; a first to-be-recognized image determining unit configured to determine a first to-be-recognized image based on the first detection frame, where the first detection frame is obtained by mapping first target position information corresponding to the first to-be-recognized image to the target image; a second to-be-identified image determining unit, configured to determine a second to-be-identified image according to the second detection frame, where the second detection frame is obtained by mapping first target position information corresponding to the second to-be-identified image to the target image; an overlapping area acquiring unit configured to acquire an area of an overlapping area of the first image to be recognized and the second image to be recognized; a fusion execution unit, configured to fuse the first detection frame and the second detection frame in response to that the intersection area and the area of the overlap area meet a preset condition; and the second target position information determining unit is used for obtaining the second target position information according to the fusion result.

In some possible embodiments, the second target location information includes at least one detection frame, and the intrusion determination module includes: a first target detection frame specifying unit configured to specify, as a first target detection frame, a detection frame that intersects the intrusion area in the second target position information; an intrusion determination unit configured to determine that there is no intrusion of an object into the airport runway in response to zero number of the first object detection frames; and determining that the object invades the airport runway in response to the number of the first object detection frames being not zero.

In some possible embodiments, the target image is a current frame image of a surveillance video stream, the field of view of the target image further includes a real-time identification area determined based on an airport runway, and the apparatus further includes: a second target detection frame determining module, configured to determine, as a second target detection frame, a detection frame that intersects with the real-time identification area in the second target location information; a tracking module, configured to track, in response to that the number of the second target detection frames is zero, the second target position information based on the surveillance video stream; and the real-time identification module is used for responding that the number of the second target detection frames is not zero, updating the target image according to the next frame image of the monitoring video stream, carrying out target identification on the updated target image, and determining the intrusion result of the airport runway based on the identification result.

In some possible embodiments, the image segmentation module is configured to determine each image in the cutting result as the image to be identified; or, the image to be recognized is determined in the cutting result according to the invasion area.

In some possible embodiments, the deep neural network includes a feature extraction network, a classification recognition network, and a recognition result output network, and the target recognition module includes: the characteristic extraction unit is used for extracting the characteristics of the image to be identified based on the characteristic extraction network to obtain a characteristic diagram; the recognition unit is used for carrying out classification recognition on the feature map based on the classification recognition network to obtain a classification recognition result, and the classification recognition result represents a target in the image to be recognized and the probability of the target belonging to a preset target; and the identification result output unit is used for screening the classification identification result based on the identification result output network to obtain the first target position information.

In some possible embodiments, the apparatus further includes a neural network optimization module, configured to determine whether there is a missing identified target according to the second target location information; and in response to the existence of the target which is not identified, updating the deep neural network according to the target which is not identified.

In some possible embodiments, the second target location information includes at least one detection frame, and the apparatus further includes a display module, configured to obtain the detection frame in the second target location information; and displaying the detection frame on the target image.

In some possible embodiments, the apparatus further comprises an alarm module for sending an alarm message in response to the presence of an intrusion of the object into the airport runway.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

The embodiment of the present disclosure also provides a computer-readable storage medium, where at least one instruction or at least one program is stored in the computer-readable storage medium, and the at least one instruction or the at least one program is loaded by a processor and executed to implement the method. The computer readable storage medium may be a non-volatile computer readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured as the method.

The electronic device may be provided as a terminal, server, or other form of device.

FIG. 10 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like terminal.

Referring to fig. 10, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user as described above. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of the touch or slide action but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as WiFi,2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 described above further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.

FIG. 11 shows a block diagram of another electronic device in accordance with an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server. Referring to fig. 11, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, that are executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system stored in memory 1932, such as Windows Server, mac OS XTM, unixTM, linuxTM, freeBSDTM, or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium, such as a memory 1932, is also provided that includes computer program instructions executable by a processing component 1922 of an electronic device 1900 to perform the above-described methods.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be interpreted as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An airport runway intrusion detection method, comprising:

acquiring a target image, wherein the view field of the target image comprises an invasion area determined based on an airport runway;

cutting the target image, and determining an image to be recognized according to a cutting result, wherein the image to be recognized comprises at least two adjacent images;

performing target identification on the image to be identified based on a deep neural network to obtain first target position information, wherein the first target position information represents the position of a target in the image to be identified;

mapping the first target position information to the target image to obtain target position mapping information, wherein the target position mapping information comprises at least two detection frames, and each detection frame represents the position of a corresponding target;

acquiring any first detection frame and any second detection frame, wherein the first detection frame and the second detection frame are two detection frames intersected in the target position mapping information;

acquiring the intersection area of the first detection frame and the second detection frame;

determining a first image to be recognized according to the first detection frame, wherein the first detection frame is obtained by mapping first target position information corresponding to the first image to be recognized to the target image;

determining a second image to be identified according to the second detection frame, wherein the second detection frame is obtained by mapping first target position information corresponding to the second image to be identified to the target image;

acquiring the area of the overlapping area of the first image to be recognized and the second image to be recognized;

fusing the first detection frame and the second detection frame in response to the intersection area and the area of the overlapped area meeting a preset condition;

obtaining second target position information according to the fusion result, wherein the second target position information represents the position of a target in the target image;

and determining an intrusion result of the airport runway according to the second target position information and the intrusion area.

2. The method of claim 1, wherein the second target location information includes at least one detection box, and wherein determining the intrusion outcome for the airport runway based on the second target location information and the intrusion area comprises:

determining a detection frame intersecting the intrusion area in the second target position information as a first target detection frame;

in response to the first number of object detection boxes being zero, determining that there is no intrusion of an object into the airport runway;

and determining that the condition that the target invades the airport runway exists in response to the number of the first target detection frames being not zero.

3. The method of claim 2, wherein the target image is a current frame image of a surveillance video stream, the target image further comprising a real-time identified area in the field of view determined based on airport runways, the method further comprising:

determining a detection frame intersecting the real-time identification area in the second target position information as a second target detection frame;

tracking the second target position information based on the surveillance video stream in response to the number of second target detection frames being zero;

and in response to the second target detection frame number not being zero, updating the target image according to the next frame image of the surveillance video stream, performing target identification on the updated target image, and determining an intrusion result of the airport runway based on an identification result.

4. The method according to any one of claims 1-3, wherein determining the image to be recognized according to the cutting result comprises:

determining each image in the cutting result as the image to be identified;

or the like, or, alternatively,

and determining the image to be identified in the cutting result according to the invasion area.

5. The method according to claim 1, wherein the deep neural network comprises a feature extraction network, a classification recognition network and a recognition result output network, and the performing target recognition on the image to be recognized based on the deep neural network to obtain first target location information comprises:

extracting the features of the image to be identified based on the feature extraction network to obtain a feature map;

classifying and identifying the feature map based on the classification and identification network to obtain a classification and identification result, wherein the classification and identification result represents a target in the image to be identified and the probability that the target belongs to a preset target;

and outputting the network to screen the classification recognition result based on the recognition result to obtain the first target position information.

6. The method of claim 1, further comprising:

judging whether a target which is not identified exists or not according to the second target position information;

in response to the presence of the missing identified target, updating the deep neural network in accordance with the missing identified target.

7. The method of claim 1, wherein the second target location information comprises at least one detection box, the method further comprising:

acquiring a detection frame in the second target position information;

and displaying the detection frame on the target image.

8. A method according to claim 2 or 3, characterized in that the method further comprises:

and sending alarm information in response to the condition that the target invades the airport runway.

9. An airport runway intrusion detection device, comprising:

the system comprises a target image acquisition module, a display module and a display module, wherein the target image acquisition module is used for acquiring a target image, and the view field of the target image comprises an invasion area determined based on an airport runway;

the image segmentation module is used for cutting the target image and determining an image to be identified according to a cutting result, wherein the image to be identified comprises at least two adjacent images;

the target identification module is used for carrying out target identification on the image to be identified based on a deep neural network to obtain first target position information, and the first target position information represents the position of a target in the image to be identified;

the recognition result integration module is used for mapping the first target position information to the target image to obtain target position mapping information, wherein the target position mapping information comprises at least two detection frames, and each detection frame represents the position of a corresponding target; acquiring any first detection frame and any second detection frame, wherein the first detection frame and the second detection frame are two detection frames intersected in the target position mapping information; acquiring the intersection area of the first detection frame and the second detection frame; determining a first image to be recognized according to the first detection frame, wherein the first detection frame is obtained by mapping first target position information corresponding to the first image to be recognized to the target image; determining a second image to be recognized according to the second detection frame, wherein the second detection frame is obtained by mapping first target position information corresponding to the second image to be recognized to the target image; acquiring the area of the overlapping area of the first image to be recognized and the second image to be recognized; fusing the first detection frame and the second detection frame in response to the intersection area and the area of the overlapped area meeting a preset condition; obtaining second target position information according to the fusion result, wherein the second target position information represents the position of a target in the target image;

and the invasion judging module is used for determining the invasion result of the airport runway according to the second target position information and the invasion area.

10. A computer-readable storage medium having at least one instruction or at least one program stored thereon, the at least one instruction or the at least one program being loaded into and executed by a processor to implement a method of airport runway intrusion detection according to any of claims 1-8.

11. An electronic device comprising at least one processor, and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to implement an airport runway intrusion detection method according to any of claims 1-8 by executing the instructions stored by the memory.