KR20220107494A - Drone detection system based on deep learning using SWIR camera - Google Patents

Abstract

The present invention relates to a drone detection system based on deep learning using a near-infrared camera. The present invention comprises: a near-infrared camera that performs in photographing a video to detect a drone object as a short-wavelength infrared method, and outputs the photographed video as an image; and an image analyzer that receives the image photographed and transmitted by the near-infrared camera, and analyzes the image based on deep learning to detect a drone object, a location of the drone object, and a movement of the drone object in the transmitted image. Therefore, the present invention is capable of having an effect of detecting the drone that is so small that it is difficult for a person to identify with the naked eye.

Description

Drone detection system based on deep learning using SWIR camera}

The present invention relates to a deep learning-based drone detection system using a near-infrared camera. In more detail, using a convolutional neural network (CNN), which is one of the deep learning technologies used in the image processing field, a small drone that is located at a long distance and is small enough for a human to be difficult to identify with the naked eye. DIP using a near-infrared camera that can detect and configure a system using only a SWIR camera without special devices or technologies such as radar, thereby reducing the cost for system configuration and improving portability. It relates to a learning-based drone detection system.

Recently, small unmanned aerial vehicles known as drones and related industries are developing rapidly.

The global drone market is rapidly growing from the initial military market to the commercial/hobby drone market.

Based on the expansion of the drone market, related technologies such as miniaturization of drones and battery expansion have also rapidly developed.

As drone technology develops, the malicious use of drones, including reckless invasion of privacy and the delivery of small bombs, and cases of drone abuse by individuals or groups related to chemical substances and small-scale terrorism have become a global problem.

In order to prevent this problem, recent research on anti-drone technology such as drone detection technology is being actively conducted.

Anti-drone technology is a combination of unmanned aerial vehicle detection technology and unmanned aerial vehicle neutralization technology.

In the case of unmanned aerial vehicle device detection technology among anti-drone technologies, drone detection technology that combines various sensors such as ultra-high-resolution radar, microphone, camera, and RF detection is being developed.

However, most of the existing drone detection technology is a detection technology through a radar fused with various sensors, and such radar-based detection has a problem that it is relatively expensive and requires a trained operating expert.

In addition, research using a general video camera image is also being conducted, but research is being conducted only on a case that is large enough for humans to visually recognize the shape of the drone.

Therefore, there is a need for a detection technology for a small drone image that is difficult to identify with the naked eye without using a special device and technology.

Prior art literature: KR Registered Patent Publication No. 10-1200974 (Notice on November 12, 2012)

The present invention has been devised to solve the above problems, and is located at a distance using a convolutional neural network (CNN), which is one of the deep learning technologies used in the image processing field, so that a person can see it with the naked eye. It can detect small drones that are small enough to be difficult to identify, and can configure the system using only a SWIR camera without special devices or technologies such as radar, thereby reducing the cost for system configuration and portability. It aims to provide a deep learning-based drone detection system using a near-infrared camera that can improve

A deep learning-based drone detection system using a near-infrared camera according to the present invention devised to achieve the above object is a near-infrared camera ( 20); An image analyzer (30) that receives an image captured by a near-infrared camera and transmitted, and analyzes an image based on deep learning to detect a drone object, a location of a drone object, and a movement of a drone object in the transmitted image; Receives information about the position and movement of the drone object detected by the image analyzer, remotely controls the operation of the near-infrared camera, displays the image transmitted from the near-infrared camera, and when the image analyzer detects the drone object, the near-infrared camera It includes an integrated controller 40 that superimposes the drone object detected by the image analyzer on the image transmitted from the image analyzer and displays it on the monitor. Learning data generation function 100 for generating learning data that can identify only the drone object in the image including the background through the labeling process of setting a bounding box for the; A data conversion function 200 for converting learning data into an input form for deep learning network learning in order to detect a drone object using a deep learning network from an image transmitted from a near-infrared camera and track its location; A deep learning network model including a convolutional neural network is included, and the training data converted in the data transformation function 200 is used as input data for the deep learning network model to detect a drone object on the image used as input data, a data learning function 300 for learning to track the location; In order to detect an actual drone using the deep learning network model learned by the data learning function 300, the image information acquired from the near-infrared camera is captured by time unit and divided into images, and the drone object included in the divided image is a drone detection application function 400 for detecting and outputting an image in which a bounding box is set for the detected drone object; and a state estimation function 500 for estimating the dynamic state of the drone object including the movement speed of the drone object detected in the form in which the bounding box is set on the image output from the drone detection application function 400 .

In addition, the image analyzer uses the result value of the state estimation function 500 to obtain and store the center coordinates of the bounding box including the drone object, and then connects the center coordinates of the bounding box included in each image with a straight line to determine the size of the drone object. It may include a path tracking function 600 for outputting a final image by tracking the path.

In addition, the learning data converted through the data conversion function 200 is stored in the form of a text file expressed as a real number, and the converted learning data is the class number, the horizontal axis coordinates for the center point of the bounding box containing the drone object, the bounding beat It may include including information about the vertical axis coordinates for the center point of the bounding box, the width of the bounding box, and the height of the bounding box.

According to the present invention, there is an effect that it is possible to detect a drone that is small enough for a human to be difficult to identify with the naked eye without using a special device and technology.

In addition, since the configuration of the entire system is relatively simple, the cost for system configuration can be reduced and portability can be improved.

1 is a diagram showing the configuration and signal flow of a deep learning-based drone detection system using a near-infrared camera according to a preferred embodiment of the present invention;
2 is a view showing the function of the image analyzer;
3 is a diagram illustrating a labeling process for a drag object in a learning data generation function;
4 is a diagram showing a state in which the learning data is converted into an input form for deep learning network learning in the data conversion function;
5 is a graph showing performance test results for a deep learning-based drone detection system using a near-infrared camera according to a preferred embodiment of the present invention;
6 is a view showing a state in which the path of the drone object is tracked by the path tracking function;
7 is a view showing the drone object detected by the image analyzer superimposed on the image transmitted from the near-infrared camera on the monitor of the integrated controller.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, it should be noted that in adding reference numerals to the components of each drawing, the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, preferred embodiments of the present invention will be described below, but the technical spirit of the present invention is not limited thereto or may be variously implemented by those skilled in the art without being limited thereto.

1 is a diagram showing the configuration and signal flow of a deep learning-based drone detection system using a near-infrared camera according to a preferred embodiment of the present invention, FIG. 2 is a diagram showing the function of an image analyzer, FIG. 3 is a learning data generation function A diagram showing a labeling process for a drag object in FIG. 4 is a diagram showing a state in which the learning data is converted into an input form for deep learning network learning in a data conversion function, FIG. 5 is a diagram according to a preferred embodiment of the present invention A diagram showing the results of a performance experiment for a deep learning-based drone detection system using a near-infrared camera as a graph, FIG. 6 is a diagram showing a state of tracking the path of a drone object by a path tracking function, and FIG. 7 is a transmission from the near-infrared camera It is a drawing that is displayed on the monitor of the integrated controller by superimposing the drone object detected by the image analyzer on one image.

A deep learning-based drone detection system using a near-infrared camera according to a preferred embodiment of the present invention includes a near-infrared camera 20 , an image analyzer 30 , and an integrated controller 40 , with reference to FIG. 1 .

First, the deep learning-based drone detection system using a near-infrared camera according to the present invention uses a convolutional neural network (CNN), which is one of the deep learning technologies used in the image processing field, and is It can detect small drones that are so small that they cannot be identified with the naked eye.

In addition, since a system can be configured using only a near-infrared camera without a special device or technology such as a radar, the cost for system configuration can be reduced and portability can be improved.

Hereinafter, components of a deep learning-based drone detection system using a near-infrared camera according to a preferred embodiment of the present invention will be described in detail.

Referring to FIG. 1 , the near-infrared camera 20 captures an image for detecting a drone object in a short-wavelength infrared method including a SWIR camera, and outputs the captured image as an image in the form of a thermal image.

1, the image analyzer 30 receives an image captured by the near-infrared camera 20 and transmitted, and based on deep learning to detect the drone object and the position of the drone object and the movement of the drone object in the transmitted image. Analyze the image with

Referring to FIG. 1 , the integrated controller 40 receives information about the position and movement of the drone object detected by the image analyzer 30 , and remotely controls the operation including pan-tilt of the near-infrared camera 20 . .

In addition, the integrated controller displays the image transmitted from the near-infrared camera 20, and when the image analyzer 30 detects the drone object, as shown in FIG. The drone object detected by the analyzer 30 is superimposed and displayed on the monitor.

Meanwhile, referring to FIG. 2 , the image analyzer 30 includes a learning data generation function 100 , a data conversion function 200 , a data learning function 300 , a drone detection application function 400 , and a state estimation function 500 . , a path tracking function 600 is included.

Referring to FIG. 1 , the learning data generating function 100 of the image analyzer 30 determines whether there is a drone object in a specific image in the data learning function 300 described below, and learning to detect the drone object create material

As shown in FIG. 3 , the learning data generation function 100 is a drone object in an image including a background through a labeling process of setting a bounding box for a drone object in a plurality of images including a background and a drone object. It generates training data that can identify only

The labeling process in the learning data generating function 100 determines whether the pixels of each object are connected to each other in order to distinguish each object included in the image for learning on the image.

At this time, each object included in the image is distinguished from each other by searching for a connected component, which is a set of pixels connected to each other in the image array, and labeling the pixels having the same connected component by designating a number.

In general, image information acquired by the near-infrared camera 20 may generate non-regular noise depending on the performance of the near-infrared camera 20 .

The size filtering process 150 is a process of removing non-regular noise included in the image acquired by the near-infrared camera 20. In the labeling process of the learning data generating function 100, a specific number of pixels from the pixels of the image to be labeled , and removes an area less than the specified number of pixels so that the data learning function 300 can improve the identification of the drone object included in the image.

The data conversion function 200 converts learning data into an input form for deep learning network learning in order to identify a drone object using a deep learning network and track its location.

The learning data converted through the data conversion function 200 is stored in the form of a text file expressed in real numbers as shown in FIG. 4, and the configuration of each row of the converted learning data is a class number (2), a drone object, respectively. It includes information about the horizontal axis coordinates (4) for the center point of the bounding box, the vertical axis coordinates (6) for the center point of the bounding box, the width (8) of the bounding box, and the height (10) of the bounding box.

Here, the class number (2) is expressed as a non-negative integer. If there is only one object in the current image, it has a fixed value of 0.

The horizontal axis coordinate (4) for the center point of the bounding box containing the drone object is expressed as a real number of 6 digits after the decimal point between 0 and 1. express

The ordinate coordinate (6) of the center point of the bounding foil is It is expressed as a real number of 6 digits after the decimal point between 0 and 1, and it expresses where the coordinates of the upper left corner of the rectangular box are on the vertical axis in proportion to the entire image.

The width (8) of the bounding box is It is expressed as a real number of 6 digits after the decimal point between 0 and 1, and when the size of the entire image is 1, the width of the bounding box is expressed in proportion to the entire image.

The height (10) of the bounding box is It is expressed as a real number of 6 digits after the decimal point between 0 and 1, and when the size of the entire image is 1, the height of the bounding box is expressed in proportion to the entire image.

The data learning function 300 is deep learning including R-CNN (Regions with CNN), Faster R-CNN, Faster R-CNN, and YOLO (You Only Look Once) including a convolutional neural network (CNN). Learning to detect a drone object on an image used as input data by using the training data converted by the data conversion function 200 as input data for the deep learning network model, including a network model, and to track its location proceed with

The drone detection application function 400 is a process for detecting an actual drone using the deep learning network model learned by the data learning function 300, and captures and divides the image information obtained from the near-infrared camera 20 in units of time. A drone object included in an image is detected, and an image in which a bounding box is set for the detected drone object is output.

On the other hand, the drone detection application function 400 undergoes a standardization process of dividing the continuous image acquired by the near-infrared camera 20 into a finite number of pixels.

Then, a quantization process of expressing a discrete integer value is performed in order to determine the grayscale value of the divided pixels having continuous grayscale values for the image that has undergone the standardization process.

In addition, by removing noise that may occur in the quantization process or in the process of amplifying the image signal acquired by the near-infrared camera 20, the drone detection application function 400 increases the accuracy for detecting a real drone object. A noise canceling function may be included.

The background clutter removal function 350 may be included in the drone detection application function 400, and it extracts the background by separating the drone object and the background from the image obtained from the actual near-infrared camera 20, and removes the extracted background. Pre-processing is performed to extract pixel information about the drone object in the dynamic state.

The background clutter removal process 350 can be implemented using a Gaussian model, obtains a relatively motionless background in an image compared to a drone object whose position continuously changes due to movement, and initially selects a background from a given image. After creating an appropriate background from images specified as or continuously inputted, it separates only the drone object excluding the background through comparison with the image acquired from the current movie camera.

On the other hand, when the trained deep learning network model detects a drone object in an image obtained from the near-infrared camera 20, it divides the image into image frame units and detects the drone object only in a single image frame, so there is a high possibility of false detection.

In addition, since movement information of a dynamic object such as a drone has image noise and uncertainty of movement state transition, it is necessary to robustly estimate the state.

Therefore, in the present invention, in order to solve this problem, a drone object is recursively displayed on an image output from a deep learning network model by using a recursive filter including a Kalman filter and an extended Kalman filter (EKF). A state estimation function 500 for detecting (recursively) and estimating the state of the drone object including the movement path is included.

The state estimation function 500 may also be used to estimate the dynamic state of the drone object acquired by the near-infrared camera 20 .

The state estimation function 500 includes the movement speed of the drone object detected in the form of a bounding box set on the image output from the drone detection application function 400 that detects the actual drone by applying the learned deep learning network model. Estimate the dynamic state of the drone object.

The state estimation function 500 stores and uses images output from the learned deep learning network model for each time step.

The state estimation function 500 receives an image at a previous time step from among a plurality of images output by the deep learning network model learned in the drone detection application function 400 as an input and recursively performs a drone object. Detect and estimate the current state.

Referring to FIG. 6 , the path tracking function 600 obtains the center coordinates of the bounding box including the drone object using the result values output in the form of a plurality of images from the state estimation function 500, and is included in each result value The final image is output by tracing the path of the drone object by connecting the center coordinates of the bounding box with a straight line.

That is, on the image output from the state estimation function 500, the horizontal axis coordinates for the center point of the bounding box containing the drone object, the vertical axis coordinates for the center point of the bounding foil, the width of the bounding box, and information about the height of the bounding box The center coordinates of the bounding box containing the drone object are obtained using the method, and the center coordinates of the bounding box containing the drone object are obtained and stored in the memory.

5 is a result of performing a performance experiment using mAP (mean average precision), which is a performance evaluation index used in an object detection area in the field of image processing (computer vision), for a total of 95 minutes ( minute), it can be seen that the drone detection performance of 98.4% was obtained as a result of mAP evaluation using the images acquired from the near-infrared camera 20.

The above description is merely illustrative of the technical idea of the present invention, and various modifications, changes and substitutions are possible within the scope that does not depart from the essential characteristics of the present invention by those of ordinary skill in the art to which the present invention pertains. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are for explaining, not limiting, the technical spirit of the present invention, and the scope of the technical spirit of the present invention is not limited by these embodiments and the accompanying drawings. . The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

20 - Near infrared camera 30 - Image analyzer
40 - integrated controller

Claims (5)

a near-infrared camera 20 that performs image capturing for detecting a drone object in a short-wavelength infrared method and outputs the captured image as an image in the form of a thermal image;
An image analyzer (30) for receiving an image photographed and transmitted by the near-infrared camera (20), and analyzing the image based on deep learning to detect the drone object and the position of the drone object in the transmitted image, and the movement of the drone object;
Receives information about the position and movement of the drone object detected by the image analyzer 30, remotely controls the operation of the near-infrared camera 20, displays the image transmitted from the near-infrared camera 20, and uses the image analyzer ( When the drone object is detected in 30), the integrated controller 40 superimposes the drone object detected by the image analyzer 30 on the image transmitted from the near-infrared camera 20 and displays it on the monitor.
including,
The image analyzer 30
Through a labeling process of setting a bounding box for a drone object in a plurality of images including the background and drone object transmitted from the near-infrared camera 20, learning data that can identify only the drone object in the image including the background is generated. Learning data generation function (100);
A data conversion function 200 for converting learning data into an input form for deep learning network learning in order to detect a drone object using a deep learning network in the image transmitted from the near infrared camera 20 and track its location; and
A deep learning network model including a convolutional neural network is included, and the training data converted in the data transformation function 200 is used as input data for the deep learning network model to detect a drone object on the image used as input data, Data learning function 300 for learning to track the location
A deep learning-based drone detection system using a near-infrared camera. a near-infrared camera 20 for performing image capturing for detecting a drone object in a short-wavelength infrared method and outputting an image of the captured image;
An image analyzer (30) for receiving an image photographed and transmitted by the near-infrared camera (20), and analyzing the image based on deep learning to detect the drone object and the position of the drone object in the transmitted image, and the movement of the drone object;
Receives information about the position and movement of the drone object detected by the image analyzer 30, remotely controls the operation of the near-infrared camera 20, displays the image transmitted from the near-infrared camera 20, and uses the image analyzer ( When the drone object is detected in 30), the integrated controller 40 superimposes the drone object detected by the image analyzer 30 on the image transmitted from the near-infrared camera 20 and displays it on the monitor.
including,
The image analyzer 30
Through a labeling process of setting a bounding box for a drone object in a plurality of images including the background and drone object transmitted from the near-infrared camera 20, learning data that can identify only the drone object in the image including the background is generated. Learning data generation function (100);
A data conversion function 200 for converting learning data into an input form for deep learning network learning in order to detect a drone object using a deep learning network in the image transmitted from the near infrared camera 20 and track its location; and
A deep learning network model including a convolutional neural network is included, and the training data converted in the data transformation function 200 is used as input data for the deep learning network model to detect a drone object on the image used as input data, a data learning function 300 for learning to track the location; and
In order to detect a real drone using the deep learning network model learned by the data learning function 300, the image information obtained from the near infrared camera 20 is captured by time unit and divided into images, and the Drone detection application function 400 that detects a drone object and outputs an image in which a bounding box is set for the detected drone object
A deep learning-based drone detection system using a near-infrared camera. a near-infrared camera 20 for performing image capturing for detecting a drone object in a short-wavelength infrared method and outputting an image of the captured image;
An image analyzer (30) for receiving an image photographed and transmitted by the near-infrared camera (20), and analyzing the image based on deep learning to detect the drone object and the position of the drone object in the transmitted image, and the movement of the drone object;
Receives information about the position and movement of the drone object detected by the image analyzer 30, remotely controls the operation of the near-infrared camera 20, displays the image transmitted from the near-infrared camera 20, and uses the image analyzer ( When the drone object is detected in 30), the integrated controller 40 superimposes the drone object detected by the image analyzer 30 on the image transmitted from the near-infrared camera 20 and displays it on the monitor.
including,
The image analyzer 30
Through a labeling process of setting a bounding box for a drone object in a plurality of images including the background and drone object transmitted from the near-infrared camera 20, learning data that can identify only the drone object in the image including the background is generated. Learning data generation function (100);
A data conversion function 200 for converting learning data into an input form for deep learning network learning in order to detect a drone object using a deep learning network in the image transmitted from the near infrared camera 20 and track its location; and
A deep learning network model including a convolutional neural network is included, and the training data converted in the data transformation function 200 is used as input data for the deep learning network model to detect a drone object on the image used as input data, a data learning function 300 for learning to track the location; and
In order to detect an actual drone using the deep learning network model learned by the data learning function 300, the image information acquired from the near infrared camera 20 is captured by time unit and divided into images, and the a drone detection application function 400 that detects a drone object and outputs an image in which a bounding box is set for the detected drone object; and
State estimation function (500) for estimating the dynamic state of the drone object including the movement speed of the drone object detected in the form of a bounding box on the image output from the drone detection application function (400)
A deep learning-based drone detection system using a near-infrared camera. 4. The method of claim 3,
The image analyzer 30 obtains and stores the center coordinates of the bounding box including the drone object by using the result value of the state estimation function 500, and then connects the center coordinates of the bounding box included in each image with a straight line to obtain the drone object. Path tracking function (600) to output the final image by tracing the path of
A deep learning-based drone detection system using a near-infrared camera. According to claim 1,
The learning data converted through the data conversion function 200 is stored in the form of a text file expressed as a real number, and the converted learning data is the class number, the horizontal axis coordinates for the center point of the bounding box containing the drone object, and the center point of the bounding foil. Including information about the vertical axis coordinates, the width of the bounding box, and the height of the bounding box
A deep learning-based drone detection system using a near-infrared camera.

Images