KR102583686B1 - Method and apparatus for detecting objects - Google Patents

Abstract

The object detection device performs image preprocessing to change the statistical characteristics of the input image using a preprocessing neural network learned to improve object detection performance, detects an object from the preprocessed image, and detects the object. The detection results are fed back to the preprocessing neural network.

Description

Object detection method and apparatus {METHOD AND APPARATUS FOR DETECTING OBJECTS}

The present invention relates to an object detection method and device. In particular, an object detection method that can improve ship detection performance by preprocessing marine images with various weather effects into a form suitable for machine learning for ship detection rather than the human eye. and devices.

On operating ships, coastal posts, control centers, etc., methods are needed to detect nearby ships for various purposes, such as traffic control or maritime accident support. To this end, communication-based identification methods such as automatic identification system (AIS) are applied to vessels over a certain size and location transmitting devices (V-pass) to small vessels. There are cases where private vessel information cannot be obtained due to intentional non-use. Additionally, in the case of enemy countries and warships of other countries, it is difficult to obtain information based on such communication. In this case, technologies using synthetic aperture radar and satellite images have been proposed, but it takes a long time to acquire the information, so vessel detection on marine optical images is necessary.

Unlike land, the marine environment has many environmental factors such as sea fog, wake, and reflected light from the water, and it is also difficult to obtain data. In the case of marine images, the goal is to improve the image quality by removing sea fog. In general, sea fog is removed by analyzing the characteristics of the sea fog, separating sea fog and cloud light, and then compensating for brightness accordingly. However, there is a problem of worsening distortion when there is an extreme difference in brightness within the image, and since it is a preprocessing technology suitable for viewing with the human eye, it is unsuitable as a technology for detecting objects in a resolution optical image. In addition, methods for preprocessing marine images using traditional image processing techniques have been proposed, but they generally focus on simply removing weather effects.

Most image preprocessing techniques, such as denoising, blurring, and fog removal, are for output optimized for the human eye, and it is difficult to use these preprocessing techniques to improve machine learning performance. This is also a problem with most existing preprocessing technologies. Additionally, to solve the lack of learning data, there are data augmentation techniques such as adding geometric changes to images or changing colors, but these methods also rely on human intuition and are vulnerable to changes in the angle (white balance, etc.) of the augmentation techniques.

In addition, the active learning technique, which selectively learns data with a high learning effect among a large amount of data, showed significant results, confirming the existence of effective learning data for machine learning, but this was achieved by selectively learning some of the existing data. It is a learning skill.

The problem to be solved by the present invention is to provide an object detection method and device that can improve object detection performance from images captured in a maritime environment where data is difficult to secure.

According to one embodiment of the present invention, a method for detecting an object from an image input from an object detection device is provided. The object detection method includes performing image preprocessing to change the statistical characteristics of the input image using a preprocessing neural network learned to improve object detection performance, detecting an object from the preprocessed image, And it includes the step of feeding back the detection result of the object to the preprocessing neural network.

The detecting step may include detecting the object from the image preprocessed using an object detection neural network learned to detect the object.

The feedback step may include training the preprocessing neural network to change statistical characteristics of the image in a direction to improve object detection performance based on the object detection result.

The detection result includes a class of the detected object and a loss value for the detected object, and the feedback step may include training the preprocessing neural network in a direction to reduce the loss value.

The statistical characteristic may include at least one of color value, saturation value, brightness value, white balance, and probability distribution itself.

The image may include an image taken at sea, and the object may include a ship.

According to another embodiment of the present invention, an object detection device for detecting an object from an input image is provided. The object detection device includes a pre-processing unit, a detection unit, and a feedback processing unit. The preprocessor modulates the input image using a preprocessing neural network learned to modulate the image into a form optimized for neural network-based object detection and outputs the modulated image. The detection unit performs the neural network-based object detection from the image output from the preprocessor. And the feedback processing unit transmits the object detection result of the detection unit to the pre-processing unit.

The preprocessor may use the input image and object detection results detected from the input image to retrain the preprocessing neural network to improve object detection performance based on the neural network.

The preprocessing neural network may change statistical characteristics of the image to modulate the image.

The detection result includes a class of the detected object and a loss value for the detected object, and the preprocessor may retrain the preprocessing neural network in a direction to reduce the loss value.

According to an embodiment of the present invention, image preprocessing is performed to change the statistical characteristics of the input marine optical image using a learned neural network in the direction of improving vessel detection performance, and then vessel detection is attempted on the preprocessed image, Ship detection performance can be improved.

That is, according to the embodiment of the present invention, an image preprocessing technique that aims only to improve vessel detection performance is applied without relying on human intuition or judgment on qualitative results visible to the naked eye, so it is optimized for the application site. The preprocessing technique can be automatically derived based on data accumulation. This is a technology that can be used to detect ships through optical images that were impossible to detect/identify with existing communications/radar, and can be used in maritime surveillance applications.

1 is a diagram showing an object detection device according to an embodiment of the present invention.
Figure 2 is a diagram showing a learning method of a preprocessing neural network according to an embodiment of the present invention.
Figure 3 is a flowchart showing an object detection method according to an embodiment of the present invention.
Figure 4 is a diagram showing an object detection device according to another embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification and claims, when a part is said to “include” a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Now, the object detection method and device according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a diagram showing an object detection device according to an embodiment of the present invention.

Referring to FIG. 1 , the object detection apparatus 100 includes a pre-processing unit 110, a detection unit 120, and a feedback processing unit 130. The object detection device 100 may further include a storage unit 140.

The preprocessing unit 110 preprocesses the input image and transmits the preprocessed image to the detection unit 120. An object according to an embodiment of the present invention may be a ship, but is not limited to people or structures. Additionally, the input image may be a marine image captured in a marine environment, but is not limited to this. Most of the existing image pre-processing techniques such as denoising, blurring, and fog removal applied to images captured in the marine environment are methods for outputting images optimized for the human eye. These image pre-processing techniques It is difficult to use them to improve machine learning performance.

The preprocessor 110 according to an embodiment of the present invention performs an image preprocessing process that changes the statistical characteristics of the image to optimize object detection using a neural network corresponding to a machine learning model. Statistical characteristics of images include, for example, pixel color value, pixel brightness value, pixel saturation value, average value of pixel characteristic value, color distribution, white balance, probability distribution of characteristic value, etc. It may be included, and the preprocessor 110 changes at least one element of the input image to optimize object detection using a neural network. The preprocessor 110 uses a neural network that learns image preprocessing itself to change the statistical characteristics of the image to optimize object detection using a neural network. Below, the neural network learned for image preprocessing is called a preprocessing neural network. If the Generative Adversarial Networks used in existing techniques such as style transfer use a neural network that creates a distribution similar to the target distribution, the preprocessor 110 according to an embodiment of the present invention By using a preprocessing neural network that changes the statistical characteristics of the image into a form that is advantageous for improving object detection performance using a neural network, the input image is optimized for object detection using a neural network, unlike existing preprocessing techniques that find feature points in the input image. Modify and output as much as possible.

In other words, changing the statistical characteristics of the image input from the preprocessor 110 into a form advantageous for improving object detection performance using a neural network does not mean changing the input image into a form suitable for viewing with the human eye. This means preprocessing the input image to form a form suitable for a neural network for object detection, and may include modulating an image that is completely different from the input image.

The preprocessor 110 uses the training image and the object detection results using a neural network for the learning image to train the preprocessing neural network to change the statistical characteristics of the image to optimize object detection using the neural network, thereby creating a neural network for object detection. Create a specialized preprocessing neural network. Additionally, the pre-processing unit 110 may use the detection results of the detection unit 120 fed back through the feedback processing unit 130 to retrain the pre-processing neural network in a way that improves the object detection performance of the neural network for object detection. The pre-processing unit 110 may determine the learning direction of the pre-processing neural network based on the detection result of the detection unit 120 and update the parameters of the pre-processing neural network. The preprocessing unit 110 may determine the learning direction of the preprocessing neural network depending on whether the detection result of the detection unit 120 is positive or negative. The preprocessing unit 110 may train the preprocessing neural network in a way that minimizes the loss value for the detected object of the detection unit 120.

In this way, the object detection performance of the detection unit 120 can be improved through image pre-processing using a pre-processing neural network specialized for ship detection.

The detection unit 120 performs vessel detection by applying a neural network-based vessel detection technique to the image preprocessed by the preprocessor 110 and measures vessel detection performance. The detection unit 120 can detect ships using a neural network such as Cascade R-CNN (Region-based Convolutional Neural Network), and can also apply ship detection techniques using various machine learning. The detection unit 120 may measure the object class and loss value for the detected object during the vessel detection process. Object class refers to the type of object that is the subject of identification, and can be distinguished as a person, ship, or structure, and can be distinguished in more detail as needed. For example, in the case of a ship, the object class may be set to a merchant ship, warship, fishing ship, etc. Additionally, the loss value for the detected object represents the difference between the output value of the neural network and the actual object class value. At this time, the detection unit 120 can measure the object class and the loss value for the detected object depending on whether the preprocessing neural network of the preprocessing unit 110 is applied, and through this, determine whether the image preprocessing technique using the applied preprocessing neural network is effective. can do.

The feedback processing unit 130 transmits the detection result of the detection unit 120 to the preprocessing unit 110. Additionally, the feedback processing unit 130 may store the detection result of the detection unit 120 in the storage unit 140 in response to the input image. The data stored in this way can be used as learning data later, and the neural network of the detection unit 120 can be updated.

Figure 2 is a diagram showing a learning method of a preprocessing neural network according to an embodiment of the present invention.

Referring to FIG. 2, the preprocessor 110 uses input images and real object detection results for the input images as learning data for the preprocessing neural network.

The preprocessor 110 receives the input image and the object detection result for the input image (S210).

The preprocessor 110 determines the learning direction of the preprocessing neural network based on the input image and the object detection result for the input image (S220). The preprocessor 110 may determine the learning direction of the preprocessing neural network depending on whether the actual object detection result is positive or negative. A positive indicates that the object was detected correctly, and a negative indicates that it was not detected.

The preprocessor 110 trains a preprocessing neural network that converts the input image into a form optimized for neural network-based object detection based on the determined learning direction (S230). For example, if the loss value of the object detection result is smaller than before, learning is done in the direction of improving object detection performance, and if the loss value of the object detection result is larger than before, learning is done in the direction of worsening object detection performance. It can be said that it has happened. The preprocessor 110 may train a preprocessing neural network to improve object detection performance based on the input image and the actual object detection result for the input image. As a method of learning a preprocessing neural network in the direction of improving object detection performance, the preprocessor 110 minimizes the loss of the bounding box and the predicted bounding box corresponding to the ground truth of the object coordinates in the image, A preprocessing neural network can be trained in a direction that minimizes the loss of the object's center coordinates and maximizes IOU (Intersection over Union) for segmentation values.

In the preprocessing neural network learned in this way, when an image is input, the input image is modulated to ultimately improve object detection performance using the image preprocessed by the detection unit 120.

Figure 3 is a flowchart showing an object detection method according to an embodiment of the present invention.

Referring to FIG. 3, when the preprocessor 110 receives an input image for detecting an object (S310), it changes the statistical characteristics of the input image using the image preprocessing technique of the learned preprocessing neural network and outputs it (S320). .

The detection unit 120 detects an object from the image output from the preprocessor 110 using a neural network-based object detection technique (S330).

The detection unit 120 transmits the object detection result to the pre-processing unit 110 through the feedback processing unit 130. The detection unit 120 may transmit the detected object class and the loss value for the detected object to the preprocessor 110 through the feedback processor 130 as the object detection result.

The preprocessor 110 may learn a preprocessing neural network based on the input image and the object detection result for the input image (S340). That is, the preprocessor 110 may update the preprocessing neural network to improve neural network-based object detection performance based on the input image and the object detection result for the input image.

FIG. 4 is a diagram illustrating an object detection device according to another embodiment of the present invention, and represents a system that can be used to perform at least some of the object detection devices and methods described with reference to FIGS. 1 to 3.

Referring to FIG. 4 , the object detection device 400 includes a processor 410, a memory 420, a storage device 430, and an input/output (I/O) interface 440.

The processor 410 may be implemented as a central processing unit (CPU), other chipset, microprocessor, etc.

The memory 420 is a medium such as RAM, such as dynamic random access memory (DRAM), rambus DRAM (RDRAM), synchronous DRAM (SDRAM), and static RAM (static RAM, SRAM). It can be implemented as:

The storage device 430 may be a hard disk, compact disk read only memory (CD-ROM), CD rewritable (CD-RW), digital video disk ROM (DVD-ROM), DVD-RAM, or DVD-RW disk. , may be implemented as permanent or volatile storage devices such as optical disks such as Blu-ray disks, flash memory, and various types of RAM.

I/O interface 440 allows processor 410 and/or memory 420 to access storage device 430. Additionally, the I/O interface 440 may provide an interface with an external user, for example.

The memory 420 or the storage device 430 may include a storage unit 140.

The processor 410 may perform the object detection function described in FIGS. 1 to 3, and may store program instructions for implementing at least some of the functions of the preprocessor 110, the detection unit 120, and the feedback processing unit 130. By loading it at 420, it can be controlled so that the operations described with reference to FIGS. 1 to 3 are performed. And these program commands may be stored in the storage device 430, or may be stored in another system connected to a network.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements can be made by those skilled in the art using the basic concept of the present invention defined in the following claims. It falls within the scope of rights.

Claims (10)

A method of detecting an object from an image input from an object detection device,
Performing image preprocessing to change the statistical characteristics of the input image using a preprocessing neural network learned to improve object detection performance,
Detecting an object from the image preprocessed image, and
Feeding back the detection result of the object to the preprocessing neural network,
The detection result includes a class of the detected object and a loss value for the detected object,
The class of the object represents the type of object that is the subject of identification,
The loss value represents the difference between the output value of the neural network and the actual object class,
The feedback step includes training the preprocessing neural network in a direction to reduce the loss value, and
An object detection method where the statistical characteristics include color value, saturation value, brightness value, white balance, and probability distribution. In paragraph 1:
The detecting step includes detecting the object from the image preprocessed using an object detection neural network learned to detect the object. In paragraph 1:
The feedback step includes training the preprocessing neural network to change statistical characteristics of the image in a direction to improve object detection performance based on the object detection result. delete delete In paragraph 1:
An object detection method wherein the image includes an image taken at sea, and the object includes a ship. An object detection device that detects an object from an input image,
A preprocessor that performs image preprocessing to change the statistical characteristics of the input image using a preprocessing neural network learned to improve object detection performance;
A detection unit that detects an object from the pre-processed image, and
A feedback processing unit that feeds back the detection result of the object to the pre-processing neural network,
The detection result includes a class of the detected object and a loss value for the detected object,
The class of the object represents the type of object that is the subject of identification,
The loss value represents the difference between the output value of the neural network and the actual object class,
The feedback processing unit trains the preprocessing neural network to reduce the loss value, and
The statistical characteristics include color value, saturation value, brightness value, white balance, and probability distribution. In paragraph 7:
The detection unit detects the object from the pre-processed image using an object detection neural network learned to detect the object. The object detection device of claim 7, wherein the feedback processing unit trains the preprocessing neural network to change statistical characteristics of the image in a direction that improves object detection performance based on the detection result of the object. delete

Images