KR20200009657A - Apparatus and method for eliminating obstacles in image based on learning-machine - Google Patents

Abstract

The present invention relates to a technology for eliminating an obstacle included in an image, and more specifically, to an apparatus for eliminating an obstacle in an image based on machine learning, configured to predict location information of an obstacle in an image based on machine learning and eliminate the obstacle and restore the image so as to obtain a clear image, and a method thereof. According to the present invention, the apparatus for eliminating an obstacle in an image based on machine learning comprises: a detection unit configured to detect an obstacle in an image by analyzing an input image according to a first model that is pre-learned based on deep learning; an auto focus setting unit configured to focus on a portion excluding the obstacle in the image by using location information of the detected obstacle; and an image restoration unit configured to output an image free of the obstacle according to a second model that is pre-learned based on deep learning with respect to the focused image.

Description

Apparatus and method for eliminating obstacles in image based on learning-machine}

The present invention relates to a technique for removing an obstacle included in an image, and more specifically, based on machine learning, a machine learning base capable of obtaining a clear image by removing an obstacle and restoring the image by predicting the position information of the obstacle in the image. An apparatus and method for removing obstacles in an image are provided.

Due to security, safety, or military conditions, the installation of a sensor such as a camera or a CCTV in a specific place is often performed outside. In particular, when taking a picture or video using a camera or CCTV outside the barbed wire, it is difficult to obtain a perfect image by the barbed wire.

For example, if soldiers are to monitor enemy forces outside the barbed wire during the operation, if a net is installed for safety at a stadium such as a baseball field, or inside an obstacle such as barbed wire or net such as watching outside the barbed wire of the zoo. You can't go inside and shoot video outside.

In this case, if the focus is on the barbed wire or the mesh, even if the desired object is blurred or the focus is on the target, it is impossible to obtain a perfect image by covering the barbed wire or the mesh.

This problem can occur in various environments.

That is, not only when a camera or CCTV is fixedly installed toward the fixed barbed wire, but also when the camera or CCTV moves toward the fixed barbed wire, or when the camera or CCTV moves while the barbed wire moves.

US Patent No. 9542735

The present invention was devised to solve the above problems, and an object of the present invention is to obtain a clear image from which obstacles are removed from an image having obstacles such as barbed wire or mesh.

To this end, the obstacle elimination apparatus of the machine learning-based image according to the present invention is a detection unit for detecting an obstacle in the image by analyzing the input image according to the pre-learned first model based on deep learning (learning), and the detected An auto focus setting unit that focuses on an area except for an obstacle in the image by using the position information of the obstacle, and an image outputting an image without obstacles according to a second model that is pre-learned based on deep learning on the focused image. It includes a recovery unit.

In addition, the method for removing an obstacle of a machine learning-based image according to the present invention is a method of removing an obstacle included in an image based on machine learning in an obstacle removing apparatus of an image, the first being pre-learned on the basis of deep learning. An obstacle detection step of detecting an obstacle in the image by analyzing an input image according to a model, an auto focus setting step of focusing on an area except an obstacle in the image using the detected position information of the obstacle, and the focused image And an image reconstruction step of outputting an obstacle-free image according to a second model that is pre-trained on the basis of deep learning.

As described above, the present invention has the effect of obtaining a clear image free of obstacles by detecting obstacles in the image based on deep learning and removing the obstacles in the image by automatically focusing on a portion that is not an obstacle.

1 is an internal configuration of the obstacle removing device of the machine learning based image according to the present invention.
2 is a view showing a state of the image processed step by step in the obstacle removal process of the machine learning based image according to the present invention.
Figure 3 is an internal configuration of the detector in the obstacle removing apparatus of the machine learning based image according to the present invention.
4 is a view for explaining a processing procedure in the visualization mask generator according to the present invention.
5 is a flowchart of a method for removing an obstacle in a machine learning based image according to the present invention.

 Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The construction of the present invention and the effects thereof will be clearly understood through the following detailed description.

1 schematically illustrates an internal configuration of an obstacle removing apparatus of a machine learning based image according to the present invention.

Referring to FIG. 1, the obstacle removing apparatus of the machine learning-based image includes a detector 10, an auto focus setting unit 20, an image restoring unit 30, an image quality improving unit 40, and the like.

The detector 10 receives an image from a camera or a CCTV and detects an obstacle included in the image. Obstacles included in the image are typically a barbed wire, a mesh, etc. In the embodiment of the present invention, assuming the barbed wire, mesh, etc. as an obstacle, these obstacles are collectively referred to as a mesh (mesh).

In order to find such a mesh in an image, the present invention proposes a deep-learning model based on weakly supervised learning. That is, the learning unit 10 is pre-learned based on deep learning by preparing an image having a mesh and an image having no mesh as a learning image.

The detector 10 analyzes an image according to a model trained on a deep learning basis, and classifies an image with a mesh as 0 and classifies an image without a mesh as 1. The deep learning-based learning model applied to the detector 10 is called a first model.

The auto focus setting unit 20 performs an auto focus function on the image by using the position information of the mesh detected by the detection unit 10. That is, since the auto focus setting unit 20 knows the position information of the mesh, the auto focus setting unit 20 focuses on the portion except the mesh portion in the image.

When the camera focuses on the non-mesh part, the mesh obstacles in the image become blurred and the non-mesh obstacles become clear.

The image reconstructor 30 generates an image without a mesh from the image focused by the auto focus setting unit 20.

In order to generate a mesh-free image by removing the mesh in the image, the present invention uses an auto-encoder technique, which is one of deep learning.

The auto encoder is a neural network structure having a hidden layer, and is a method of learning to output an input image as it is. That is, the image reconstruction unit 30 is trained by using an auto encoder method by preparing a pair of images having a mesh and an image having no mesh as a training image.

The image reconstructor 30 may output an image without a mesh from the auto-focused image according to the learned model based on the otter encoder. The deep learning based learning model applied by the image reconstructor 30 is called a second model.

The image quality improving unit 40 removes a blur that occurs in an image having no mesh output from the image restoring unit 30.

Since the auto-encoder compresses a feature of an image and restores the image by using the feature, loss of information occurs and a blur phenomenon in which the image is blurred due to the loss of the information.

The image quality improving unit 40 may be separately installed at the output terminal of the image restoring unit 30, but may be included in the image restoring unit 30. That is, the image quality improving function of the image quality improving unit 40 may be implemented according to an image processing technique or may be learned at a time in an auto encoder structure.

Figure 2 shows the appearance of the image processed step by step in the obstacle removal process of the machine learning based image according to the present invention.

Referring to FIG. 2, (a) of FIG. 2 shows an image with a mesh photographed by a camera or CCTV, (b) shows that a mesh has been detected in the image, and (c) shows a focus on a part except the mesh. This alignment is shown, and (d) shows the image without the mesh being removed.

In particular, it can be seen that the mesh is blurred and the portions except the mesh are clearly visible by adjusting the focus (where the dotted red line points) to the non-mesh portion in FIG.

3 is a block diagram illustrating an internal configuration of a detector in an obstacle removing apparatus of a machine learning based image according to the present invention.

Referring to FIG. 3, the detector 10 includes a convolutional neural network (CNN) 101, a classifier 102, a visualization mask generator 103, a binarization module 104, and a segmentation module. 105, and the like.

When an image is input to the CNN 101 from a camera or a CCTV, the CNN 101 extracts a feature map from the input image.

The feature map extracted from the CNN 101 is input to the classifier 102, and the classifier 102 applies the feature map to the deep neural network to classify the image into a meshed or non-meshed image. Images with a mesh are classified as label 0, and images without a mesh are classified as label 1.

When classified as an image having a mesh, the visualization mask generator 103 outputs a visualization mask mapped to the same size as the input image using the feature map of the meshed image. In the visualization mask, the mesh obstacle portion has a relatively large value, and the non-mesh obstacle portion has a relatively small value.

The binarization module 104 binarizes the value of the visualization mask to 0 or 1, and the segmentation module 105 recognizes a portion of 1 as a mesh in the binarized image, and recognizes that the portion of 0 is not a mesh, By storing only the coordinate values, the mesh position information is obtained.

Figure 4 illustrates in detail the processing in the visualization mask generator according to the present invention.

Referring to FIG. 4, when an image is input to the CNN 101, the feature maps 12-1, 12-2, ..., are repeatedly performed by a convolution process and a sub-sampling process. 12-n).

When the last generated feature map 12-n is input to the classifier 102, the classifier 102 classifies the input image into an image having a mesh or an image having no mesh through the deep neural network.

As a result of the classification, when the image is classified as the meshed image, the visualization mask generator 103 uses the feature maps 12-1, 12-2, ..., 12-n of the meshed image to determine the final visualization mask 16. will generate -n).

Specifically, the last n th feature map 12-n is averaged in the visualization mask generator 103 to generate a first average feature map 14-1, and the size of the first average feature map 14-1 is increased. The first mask 16-1 is adjusted to be adjusted.

The first mask 16-1 is computed pointwise multiplication with a second average feature map 14-2, which is an average feature map of the n−1 th feature map 12-(n-1), and then performs a first pixel multiplication. The pixel-by-pixel product result 15-1 is generated, and the size of the first pixel-by-pixel product result 15-1 is adjusted to generate a second mask.

As described above, the process of multiplying the average feature map of the current layer and the mask of the previous layer in the feature map group of the meshed image by pixels is sequentially performed in the reverse direction.

That is, in the last step, the n-th average feature map 14-n, which is the average feature map of the n-th feature map 12-1, and the n-th mask 16- (n-1) are multiplied pixel by pixel. The product result 15- (n-1) of the n-th pixel is generated, and the size of the product result 15- (n-1) of the n-th pixel is adjusted to finally have the same size as the input image. The visualization mask 16-n mapped to is generated.

5 shows an obstacle removing process of a machine learning based image according to the present invention.

Referring to FIG. 5, first, an obstacle removing apparatus of a machine learning-based image receives an image and classifies the image into a label 0, that is, an image having an obstacle, and a label 1, that is, an image without an obstacle, according to the first model (S10). .

After checking the classification result of the image (S20), if the label is 0, a visualization mask mapped to the same size as the input image is generated through the above-described process (S30). The visualization mask has a relatively large value at the obstacle free portion and a relatively small value at the obstacle free portion.

When a visualization mask having a different value is generated in an area having an obstacle and a non-obstruction, a binary image is generated by binarizing the value of the visualization mask to 0 or 1 (S40).

When generating the binarized image, the part 1 is recognized as an obstacle in the binarized image, the part 0 is recognized as not an obstacle, and only the coordinate values of the part 1 are stored to obtain location information of the obstacle (S50).

When acquiring the position information of the obstacle, the machine learning-based image removal apparatus removes the obstacle according to the second model based on the auto encoder to restore the image without the obstacle (S70).

When restoring the image without the obstacle, the image quality of the image is improved by removing the blur phenomenon generated in the restored image (S80).

In the image quality improvement step S80, the second model of the image reconstruction step S70 may be used, or an image processing technique independent of the second model may be used.

The above description is merely illustrative of the present invention, and various modifications may be made by those skilled in the art without departing from the spirit of the present invention.

Therefore, the embodiments disclosed in the specification of the present invention are not intended to limit the present invention. The scope of the present invention should be construed by the claims below, and all techniques within the scope equivalent thereto will be construed as being included in the scope of the present invention.

1: Original video with obstacles 2: Video with obstacles detected
3: Auto Focus Image 4: Image with Obstacles Removed
10: detector 12: feature map
14: Average feature map 15: Pixel-by-pixel product result
16: mask 20: autofocus setting
30: image restoration unit 40: image quality improvement unit
101: CNN 102: classifier
103: visualization mask generator 104: binarization module
105: split module

Claims (10)

A detector configured to detect an obstacle in the image by analyzing an input image according to a first model that is pre-learned based on deep learning;
An auto focus setting unit which focuses on a portion excluding an obstacle in the image by using the detected position information of the obstacle;
And an image restoring unit configured to output an image without obstacles according to a second model that is pre-learned based on deep learning based on the focused image. In claim 1,
The detection unit extracts a feature map from an input image, applies the feature map to a deep neural network, classifies the image as an obstacle image and an image without an obstacle, and uses the feature map of the image with the obstacle in the image. Obstacle removal device of the machine learning-based image, characterized in that for predicting the position information of the obstacle. In claim 2,
The detection unit sequentially performs a pointwise multiplication operation of the average feature map of the current layer and the mask of the previous layer in the feature map group of the obstacle-free image in reverse order. Obstacle removal apparatus for machine learning based image, characterized in that for outputting a visualization mask (visualization mask) mapped to the same size. In claim 3,
The detecting unit performs a binarization on the visualization mask to generate a binarized image and extract position information of the obstacle from the binarized image. In claim 1,
The obstacle of the machine learning-based image, characterized in that it further comprises an image quality improvement unit for removing a blur phenomenon generated in the image without the obstacle output from the image restoring unit. In the method for removing obstacles included in the image on the basis of machine learning in the obstacle removal apparatus of the image,
An obstacle detection step of detecting an obstacle in the image by analyzing an input image according to a first model that is pre-learned based on deep learning;
An autofocus setting step of focusing on a portion excluding an obstacle in the image by using the detected position information of the obstacle;
And an image reconstruction step of outputting an image free of obstacles according to a second model that is pre-learned on a deep learning basis with respect to the focused image. In claim 6,
The obstacle detecting step may include extracting a feature map from an input image and applying the feature map to a deep neural network to classify the image with an obstacle and an image without an obstacle,
A method for removing obstacles in a machine learning-based image, comprising: predicting position information of an obstacle in an image by using a feature map of an obstacle image. In claim 7,
Predicting the position information of the obstacle includes reversely performing a pointwise multiplication operation of the average feature map of the current layer and the mask of the previous layer in the feature map group of the image with the obstacle. And sequentially generating a visualization mask mapped to the same size as the input image. In claim 8,
The step of predicting the position information of the obstacle is a method of removing obstacles of the machine learning-based image, characterized in that the binarization is performed on the visualization mask to generate a binarized image and extract position information of the obstacle from the binarized image. In claim 6,
The method of claim 1, further comprising an image quality improvement step of removing a blur phenomenon occurring in the image without the obstacle output in the image restoring step.

Images