KR102568907B1 - Data set producing system and method used by deeplearning for search dead zone in image - Google Patents

Abstract

The present invention relates to a data set generation method and data set generation system for machine learning for detecting occlusion areas with improved supplementation efficiency and accuracy for occlusion areas of a video image identified in the process of constructing spatial information in 3D, and a specified object image A learning image labeling step in which an image processing module retrieves learning image data including ? from a learning image storage module, and a labeling module divides and labels the object image from the learning image; a machine learning step in which a learning module deep-learns object image patterns of the learning image and the labeling image based on the shape, position, and size of the labeled object image in the learning image; A dataset generation step in which the learning module classifies the dataset generated through deep learning by designated object image and stores it in a dataset storage module; a dataset checking step in which an image processing module searches a corresponding dataset in the dataset storage module according to an identification code input to a target image to check a occluded area; an image analysis step in which an occluded area detection module analyzes the target image and confirms image shapes of pixels clustered in colors and intervals having a similarity within a specified range; From the image shape identified in the target image, the learning module detects a corresponding object image through dataset-based deep learning to update the dataset, and the occluded area detection module converts the area of the image shape detected as the designated object image to the occluded area. It is to include; a occluded area detection step designated as .

Description

Data set generation method and data set generation system for machine learning for occlusion detection

The present invention relates to a dataset generation method and a dataset generation system for machine learning for detecting occluded regions with improved supplementation efficiency and accuracy for occluded regions of a video image identified in the process of constructing spatial information in 3D.

Conventional spatial information 3D construction was performed by performing 3D modeling based on images collected using MMS (Mobile Mapping System) and aerial photography.

However, since the collected images are covered by obstacles such as roadside trees, people, and banners, a occluded area occurs during texturing, and the occluded area limits 3D modeling. Therefore, conventionally, in order to supplement the occluded area, after texturing, an operator directly detects the occluded area manually and compares the occluded area with the surrounding background to correct the occluded area.

However, in order to build 3D modeling spatial information for an arbitrary area, it was necessary to collect images from various parts of the area, and in the collected images, a considerable number of obstructed areas due to obstacles occurred. Therefore, the operator had to manually detect all the occluded areas and supplement the numerous detected occluded areas one by one. However, it is very disadvantageous in terms of time and efficiency in detecting and supplementing a considerable number of occluded areas in numerous images. Moreover, since manual supplementation of occluded areas lowers the accuracy of 3D modeling, 3D modeling technology based on collected images was urgently needed to be improved.

In addition, as video collection technology using unmanned equipment such as drones was introduced to the relevant technology field, it was possible to easily and quickly collect images from an arbitrary area at low cost, and the amount of images collected over time also increased. Therefore, as the number of modeling objects for constructing 3D spatial information increases, the occlusion area also greatly increases. Therefore, a technology capable of increasing the detection speed of the occlusion area while reliably increasing the complementation efficiency is urgently needed.

Prior art literature 1. Patent Publication No. 10-2018-0061803 (published on 06/08/2018)

Accordingly, the present invention is intended to solve the above problems, and in order to build 3D spatial information based on video images, it is possible to accurately detect occluded areas generated by unnecessary obstacles (objects) from video images (learning images) to increase restoration reliability. The task to be solved is to provide a method for generating a dataset for machine learning and a system for generating a dataset for detecting occluded regions.

In order to achieve the above object, the present invention,

A learning image labeling step in which an image processing module retrieves learning image data designated with an identification code based on a specific object image configured in the learning image from the learning image storage module, and the labeling module divides and labels the object image from the learning image;

a machine learning step in which a learning module deep-learns object image patterns of the learning image and the labeling image based on the shape, position, and size of the labeled object image in the learning image;

A dataset generation step of classifying the dataset generated by the deep learning of the object image pattern by the learning module according to the identification code of the designated object image and storing the dataset in a dataset storage module;

a dataset checking step of an image processing module retrieving a corresponding dataset from the dataset storage module based on the identification code input to the target image;

an image analysis step in which an occluded area detection module analyzes constituent pixels of the target image and confirms an image shape formed by pixels clustered in colors and intervals having a similarity within a designated range in the target image; and

From the image shape identified in the target image, the learning module detects an image shape corresponding to an object image of a dataset corresponding to an identification code of the target image in the target image, and the occluded area detection module detects the detected image shape an occluded area detection step of designating a occluded area and updating the dataset through deep learning using the target image whose image shape is detected as a training image of the dataset;

It is a method for generating a dataset for machine learning for detecting an occluded region that includes.

The present invention described above accurately and quickly automatically detects occluded areas through a machine learning method in video images (learning images) collected for constructing 3D spatial information, so that rapid restoration processing for the occluded areas is possible, and restoration efficiency is improved. It has the effect of constructing high-quality 3D spatial information by increasing the accuracy and accuracy.

1 is a block diagram showing an embodiment of a dataset generation system according to the present invention;
2 is a flowchart showing an embodiment of a method for generating a dataset according to the present invention;
3 is an image showing an embodiment of a learning image collected for generating a dataset according to the present invention,
4 is an image showing an example of the labeling of the learning image,
5 is an image showing an example of a occluded area detected in a target image by the dataset generation system according to the present invention;
6 is a diagram showing another example of a occluded area detected in a target image by the dataset generation system according to the present invention;
7 is an image showing a process for filtering an actual occluded area from a occluded area primarily detected by the dataset generation system according to the present invention;
8 is a diagram showing occluded areas filtered from a target image by the dataset generation system according to the present invention.

The features and effects of the present invention described above will become clear through the following detailed description in relation to the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily implement the technical idea of the present invention. There will be. Since the present invention can have various changes and various forms, specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosure, and it should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention. Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention.

Hereinafter, the present invention will be described in detail based on the accompanying drawings.

1 is a block diagram showing an embodiment of a dataset generation system according to the present invention.

Referring to FIG. 1, the dataset generation system of the present invention includes a learning image storage module 110 for storing learning image data for image-related machine learning (hereinafter referred to as 'deep learning'), and an object image designated in the learning image. A labeling storage module 120 for storing labeled image data that has been processed, and a dataset storage module 130 for storing training image data including designated object images and a dataset generated by deep learning of the corresponding labeling image data; The labeling module 140 divides and labels the designated object image from the learning image and generates a labeling image, and searches the learning image data including the designated object image in the learning image storage module 110, and checks the occluded area. The image processing module 150 searches for a corresponding dataset in the dataset storage module 130 according to the identification code input to the input target image, and the shape, position, and size of the object image labeled in the learning image are based on By deep learning the object image patterns of the learning image and the labeling image, a dataset for each designated object image is created, and an image shape corresponding to the object image is detected from the image shape of the target image through the dataset-based deep learning, A learning module 160 for updating a dataset, a target image input module 170 for inputting target image data for checking occluded regions, and analyzing the target image to cluster the similarity into colors and intervals within a specified range and an occluded area detection module 180 that checks the image shape of the pixels and designates an area of the image shape corresponding to the designated object image as the occluded area.

Detailed descriptions of the dataset generation system and method according to the present invention will be described below through examples.

2 is a flowchart showing an embodiment of a method for generating a dataset according to the present invention, FIG. 3 is an image showing an embodiment of a learning image collected for generating a dataset according to the present invention, and FIG. It is an image showing an example of the labeling of a learning image, and FIG. 5 is an image showing an example of a occluded region detected in a target image by the dataset generation system according to the present invention.

S11; Learning video search step

The learning image is an image collected through MMS (Mobile Mapping System) or an image collected by a worker through on-site shooting, and is stored in the learning image storage module 110.

Various types of object images are included in the learning image. However, the object images include trees, cars, fences, etc. that visually obscure buildings or structures necessary for constructing 3D spatial information, and as a result, occluded areas are created in the buildings or structures. Therefore, an object image that creates a occluded area must be extracted from the learning image, and the occluded area must be restored.

Since there are various types of object images that generate occlusion areas as described above, it is preferable to classify and store learning image data for each type. Therefore, the identification code of the object image that creates the occlusion area is set for each learning image data, and the learning image data is classified and stored. Since there may be two or more object images in the learning image, two or more identification codes may be set in the learning image data.

As shown in FIG. 3, the learning image is commonly composed of a plan card and a tree as object images, and an identification code is assigned to the plan card and the tree to store the corresponding learning image data in the learning image storage module 110.

The learning image storage module 110 stores the learning image data constituting the above configuration, and the image processing module 150 searches the learning image storage module 110 for a learning image including a designated object image to create a dataset. do. As described above, since an identification code is assigned to an object image, the image processing module 150 searches for a designated object image using the identification code.

S12; Training image labeling step

When the image processing module 150 searches for a learning image including a designated object image in the learning image storage module 110, the labeling module 140 divides the object image from the learning image and labels it.

In the present invention, labeling is an algorithm that identifies the object image by displaying it according to the shape of the object image shown in the learning image. In the drawing (a) of FIG. 4, when a worker marks an object image that has various colors and irregular shapes like a tree image and the background is visible through the object image as if it were painted, within the shape and display range formed by the mark, for forming a tree image The output color of the pixel is specified (labeled) with the tree image. Accordingly, when an output color formed by a labeling target having the same or similar shape as the mark by the labeling algorithm is detected in the target image, the image shape of the detected area is recognized as a tree image.

In (a) of FIG. 4 , the object image to be labeled includes a plan card marked with a shape in addition to a tree image marked with a green shape, and the labeling module 140 labels the object image with red for distinction from labeling the green tree image. As a result, two object images are set for the learning image in (a) of FIG. 4, and combination information formed by combining the arrangement position and size of the two object images is set in the labeling image.

Displaying object images for labeling is done manually, and programs such as Pixel Annotation Tool can be used. In this embodiment, labeling image data generated through labeling processing is generated as a file with a .json extension and stored in the labeling storage module 120 . The labeling module 140 may be applied with various types of programs, such as VGG Image Annotator, in addition to the exemplified program, and the extension of labeling image data may also be various.

In the present embodiment, the labeling module 140 may label other object images as well as designated object images by the operator.

S13; machine learning step

Based on the shape, position, and size of the object image labeled in the learning image, the learning module 160 deep-learns the pattern of the learning image and the labeling image.

Deep learning for learning images learns the overall image combination and structure of unspecified object images and backgrounds in addition to designated object images, and the labeling image learns only the shape, location, and size of the labeling image, that is, the specified object image. Therefore, it is possible to detect the image shape corresponding to the designated object image from the target image by learning the pattern combination of the object image through deep learning of the learning image and the labeling image.

Deep learning of this embodiment is performed through the ResNet method, but is not limited thereto and may be performed through the R-CNN method.

Since the deep learning technology related to video images is already known, a detailed description of the deep learning process for training images and labeling images will be omitted.

S14; Dataset creation steps

The learning module 160 classifies the dataset generated through deep learning for each designated object image and stores it in the dataset storage module 130. As is well known, deep learning for video images generates a reference frame for commonality, that is, a dataset by repeatedly learning a large number of learning data having commonality several times. Therefore, when a new video image is input, the deep learning algorithm can detect commonalities with the dataset.

The dataset generated through the deep learning is classified by designated object image and stored in the dataset storage module 130, so if the tree image is a designated object image, it is classified and stored as a dataset of tree images, and the plan card In the case of a designated object image, it is classified and stored as a plan card dataset, and in the case of a tree image and plan card designated object image, it is classified and saved as a dataset containing both tree images and plan cards.

In the end, when the tree image and the plan card in the target image for constructing the 3D spatial image are object images that create a occluded area, the dataset storage module 130 searches for a data set including the tree image and the plan card.

S15; Target image input step

The object image data for checking the occlusion area is received through the object image input module 170, and the identification code of the object image composed of the object image of the object image data (see FIG. 5(a)) is input.

S16; Dataset verification steps

The image processing module 150 checks the identification code and searches the dataset storage module 130 for a dataset of a corresponding object image.

As described above, the object image configured in the learning image is given an identification code for identification, and the learning image data is searched in the learning image storage module 110 using the identification code as a keyword, and labeling in the labeling storage module 120 Image data is retrieved, and a dataset is retrieved from the dataset storage module 130 .

S17; Steps to confirm additional learning

When a target image is input, an image shape corresponding to an object image in the target image is labeled through the labeling module 140, and the target image is deep-learned based on the corresponding dataset retrieved from the dataset storage module 130 to obtain data. update the set

S18; Image analysis steps

The occluded area detection module 180 analyzes the target image and confirms the image shape of pixels clustered in colors and intervals of similarity within a specified range.

Since the target image constitutes various image shapes, and since the image shape is a combination of various colors, the occluded area detection module 180 analyzes the set color of pixels outputting the target image and extracts each image shape configured in the target image. Check.

In this description, 'hue' means color, saturation, and lightness.

In general, grades are assigned according to color, saturation, and brightness, and similarity is determined based on the grades. Therefore, if the color, saturation, and brightness of adjacent pixels have a class difference of a designated interval, the pixels are regarded as displaying the same object image. Also, if two pixels whose color, saturation, and brightness are within the class are positioned within a designated interval, the pixels are also regarded as displaying the same object image. According to this algorithm, the identity of pixels is judged and clustered locally, and the image shape formed by the clustered pixels is confirmed.

The shape of an object expressed in a target image can be visually recognized and confirmed by a person, but it is not possible to set an area as if depicting an object only by analyzing the color of each pixel of the target image. Therefore, the occluded area detection module 180 forms a rough outline of the corresponding shape by connecting the pixels located outside among the pixels clustered to form the shape of the object. As a result, the shape formed by the contour line constitutes the image shape.

A plurality of the image shapes may be configured in the target image.

S19; Blocked area detection step

From the image shape identified in the target image, the learning module 160 detects the image shape corresponding to the object image of the dataset corresponding to the identification code of the target image in the target image, and detects the occluded area using the detected image shape. The module 180 designates the occluded area, and updates the dataset through deep learning using the target image in which the image shape is detected as a training image of the dataset.

More specifically, the image shape corresponding to the object image of the dataset among the image shapes identified by the occluded area detection module 180 by the learning module 160 based on the dataset retrieved by the image processing module 150. detected using deep learning technology. That is, if the dataset searched by the image processing module 150 has an identification code of a tree image set, the learning module 160 deep-learns the image shape based on the corresponding dataset to detect the tree image constructed in the target image. will be.

As a result, the dataset generation system of the present invention detects the image shape, which is an object image that causes the occlusion area, by analyzing the target image by itself without a separate operation by the operator. For reference, since the tree image is designated as the cause of the occlusion area in the video image and the operator designates it in the training image of the dataset, the image shape detected in the target image according to this method is the cause of the occlusion area.

Continuing, when the object image detected by the learning module 160 is finally confirmed as the cause of the occluded area, the learning module 160 uses the target image in which the image shape was detected as a learning image of the dataset and uses the corresponding data through deep learning. update the set

When an image shape corresponding to the object image is detected in the target image, the occluded area detection module 180 designates the area detected as the object image as the occluded area as shown in FIG. 5(b). Furthermore, the occluded area detection module 180 filters the occluded area by extracting only pixels of a color specified in the object image. More specifically, in order to set a tree image as an identification code, a color mainly applied to the tree image is designated. That is, green, yellow green, gray, brown, red clay, ocher, etc., which are general colors of trees, are designated as colors of tree images and used for object image detection.

The occluded area detection module 180 extracts only pixels of the color specified as the color of the tree image from the constituent pixels of the object image detected through the deep learning of the learning module 160, and pixels outputting other colors are extracted from the occluded area. Exclude. Eventually, the background seen through the tree image is exposed as it is and excluded from the occluded area.

A more detailed description of this is provided below.

6 is a diagram showing another example of a occluded area detected in a target image by the dataset generation system according to the present invention, and FIG. 7 is an actual occluded area among the occluded areas primarily detected by the dataset generation system according to the present invention. 8 is a diagram showing the occluded area filtered from the target image by the dataset generation system according to the present invention.

1 and 6 to 8, the labeling module 140 includes the component colors of the corresponding object image in the labeling image, and the learning module 160 includes the shape, position, and size of the object image Z along with Based on the color, the pattern of the training image and the labeling image is deep-learned. In addition, the occluded area detection module 180 filters the occluded area by extracting only pixels having a color specified in the corresponding object image from the occluded area.

More specifically, the occluded area is matched to the area of the image shape (Z) identified in the target image, and the image shape (Z) is formed wider than the tree image. Also, in the image shape (Z), the background image (A), which is a part of the structure (S), is reflected through various places of the tree image.

Therefore, the occluded area detection module 180 detects the actual occluded area through the following process.

In order to designate the occluded area in the target image, the shape, location, and size of the object image specified in the labeling process and the color are required. The designated color is a general color of the object, and if necessary, a color corresponding to a grade within a designated range based on the designated color may also be designated as the color of the object image.

Since a tree is composed of a plurality of branches and leaves, a rear view, which is a part of the structure S, is visible in the space of each part of the tree, as shown in (a) of FIG. 7 . Therefore, in the labeling process, the background image (A) can be revealed at various places in the object image whose range is set as a lump shape, and the background image (A) can be revealed at various places in the image shape (Z) of the target image. Therefore, it is not desirable to designate the entire image shape Z, which the occluded area detection module 180 has primarily identified as a occluded area, as the occluded area.

Considering this fact, the occluded area detection module 180 checks the designated color of the object image for the image shape Z, checks the output color of the pixels constituting the image shape Z, and determines the color specified in the object image. Extract only pixels of color. That is, as shown in (b) of FIG. 7, since the color output by the pixels in the sections 'B' and 'C' is the color designated for the object image, the section of corresponding pixels constitutes an actual occluded area. For reference, the pixels located in section 'C' of this embodiment output yellow, which may not be a general color of trees. However, the color specified during labeling of the object image is not limited to the general color of the object, and if application conditions are set for each color and the pixels of the image shape satisfy the above conditions, the occlusion image detection module 180 determines the corresponding specific color It can be used for image shape extraction. Since the yellow pixels of this embodiment are located close to pixels of gray, brown, red clay, ocher, etc., and are repeatedly configured at various positions, the occluded area detection module 180 uses yellow as the component color of the object image designated as a tree. Admit it.

Since the color output by the pixels in section 'A' is not the color specified for the object image, the section of corresponding pixels is regarded as the background image (A) and filtered in the occluded area that was first detected.

As a result, the finally detected occluded area is expressed very limitedly in the target image as shown in FIG. 8, and the image of the structure S partially hidden by the occluded area can also be accurately restored.

Although the detailed description of the present invention described above has been described with reference to preferred embodiments of the present invention, those skilled in the art or those having ordinary knowledge in the art will find the spirit of the present invention described in the claims to be described later. And it will be understood that the present invention can be variously modified and changed within a range that does not deviate from the technical scope.

Claims (4)

A learning image labeling step in which an image processing module retrieves learning image data designated with an identification code based on a specific object image configured in the learning image from the learning image storage module, and the labeling module divides and labels the object image from the learning image;
a machine learning step in which a learning module deep-learns object image patterns of the learning image and the labeling image based on the shape, position, and size of the labeled object image in the learning image;
A dataset generation step of classifying the dataset generated by the deep learning of the object image pattern by the learning module according to the identification code of the designated object image and storing the dataset in a dataset storage module;
a dataset checking step of an image processing module retrieving a corresponding dataset from the dataset storage module based on the identification code input to the target image;
an image analysis step in which an occluded area detection module analyzes constituent pixels of the target image and confirms an image shape formed by pixels clustered in colors and intervals having a similarity within a designated range in the target image; and
From the image shape identified in the target image, the learning module detects an image shape corresponding to an object image of a dataset corresponding to an identification code of the target image in the target image, and the occluded area detection module detects the detected image shape an occluded area detection step of designating a occluded area and updating the dataset through deep learning using the target image whose image shape is detected as a training image of the dataset;
A method for generating a dataset for machine learning for detecting a occluded region, comprising: According to claim 1,
The labeling module includes component colors of the corresponding object image in the labeling image,
The learning module deep-learns the pattern of the learning image and the labeling image based on the color as well as the shape, position, and size of the object image;
A method for generating a dataset for machine learning for detecting occluded regions, characterized by According to claim 2,
The occluded area detection module filters the occluded area by extracting only pixels of a color designated for a corresponding object image from the occluded area;
delete