KR102414153B1 - Method and apparatus and method for augmenting learning data for object recognition - Google Patents

Abstract

A method for augmenting learning data for object recognition according to the present invention comprises the steps of: collecting an object mask, which is a binary image including an object image, a label, and an object region; calculating the pairwise similarity of the collected object masks; Selecting a specific original object to be replaced with an augmented object from among the original objects in the object image based on a calculation result of the similarity; selecting an augmented object based on the similarity with the selected specific original object; deleting the selected region of the specific original object, overwriting the deleted region with the corresponding image region of the selected augmented object, and restoring (inpaint) the erased residual background region within the region of the specific original object may include.

Description

Method and device for augmenting learning data for object recognition

The present invention relates to a technique for augmenting learning data for object recognition, and more particularly, it can be augmented by substituting an area of a specific object in a specific image (binary image) with another object (augmented object) similar to the corresponding object. It relates to a method and apparatus for augmenting learning data for object recognition.

In general, a large amount of data is required to learn an AI-based algorithm, and there is a problem that it takes a lot of money and time to secure such a large amount of learning data.

A common approach is to increase the generalization performance of AI algorithms through transformation of the training data.

The existing method deletes a specific object area (Cutout), mixes it with other object areas (MixUp, CutMix), or adjusts the size of the object area and overlays it on another image (Copy-Paste). to reinforce

This conventional augmentation method is known to be effective when there are many different types of objects, but there is a problem in that the effectiveness is inferior in the identification problem (eg, military vehicle identification, etc.) of similar but different types of objects.

Korean Patent Registration No. 10-1779782 (Announcement Date: 2017. 10. 11.) Korean Patent Publication No. 10-2018-0069588 (published date: 2018. 06. 25.)

An object of the present invention is to provide a method and apparatus for augmenting learning data for object recognition that can be augmented by substituting a region of a specific object in a specific image (binary image) with another object (augmented object) similar to the corresponding object.

An object of the present invention is to provide a method and apparatus for augmenting learning data for object recognition capable of augmenting new object recognition learning data.

The problem to be solved by the present invention is not limited to those mentioned above, and another problem to be solved that is not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

According to one aspect, the present invention includes the steps of collecting an object mask, which is a binary image including an object image, a label, and an object region, calculating the pairwise similarity of the collected object masks, and adding based on the steps of selecting a specific original object to be replaced with an augmented object among original objects in the object image, selecting an augmented object based on similarity with the selected specific original object, Object recognition comprising the steps of: deleting a region of an object, overwriting the region of the deleted object with a corresponding image region of the selected augmented object, and inpainting the erased residual background region within the region of the specific original object It is possible to provide a learning data augmentation method for

The similarity of the present invention may be calculated based on an index of a distance between a pair of object masks.

The similarity of the present invention may be calculated based on a similarity index between the pair of object masks.

The similarity of the present invention may be calculated by measuring the similarity of a feature vector of an object mask pair through an artificial neural network model.

In the selecting of the specific original object of the present invention, an original object whose horizontal/vertical size of the original object area is greater than or equal to a preset specific size may be selected as the specific original object.

In the selecting of the augmented object of the present invention, the augmented object may be selected only from an original object having a different class except for the same class.

In the selecting of the augmented object of the present invention, the augmented object may be selected based on a predetermined criterion for the calculated similarity.

According to another aspect, the present invention is a computer readable recording medium storing a computer program, wherein the computer program, when executed by a processor, collects an object image and an object mask, which is a binary image including a label and an object region calculating the pairwise similarity of the collected object masks, and selecting a specific original object to be substituted with the augmented object from among the original objects in the object image based on the calculation result of the similarity; selecting an augmented object based on the degree of similarity with the specified original object, deleting the selected region of the specific original object, and overwriting the deleted region with a corresponding image region of the selected augmented object; It is possible to provide a computer-readable recording medium comprising the step of inpainting the erased residual background area within the area of the specific original object.

According to another aspect, the present invention is a computer program stored in a computer-readable recording medium, wherein the computer program, when executed by a processor, generates an object mask, which is a binary image including an object image, a label, and an object region. collecting, calculating the pairwise similarity of the collected object masks, and selecting a specific original object to be substituted with the augmented object from among the original objects in the object image based on the similarity calculation result; selecting an augmented object based on the degree of similarity to the selected specific original object; deleting the selected region of the specific original object; and overwriting the deleted region with a corresponding image region of the selected augmented object; . , a computer program including instructions for causing the processor to perform an operation including the step of inpainting the erased residual background area within the area of the specific original object.

According to another aspect, the present invention provides a mask collecting unit for collecting an object mask, which is a binary image including an object image, a label, and an object region, and a similarity calculating unit for calculating the pairwise similarity of the collected object masks; Based on the calculation result of the degree of similarity, the target object selection unit for selecting a specific original object to be replaced with the augmented object among the original objects in the object image, and selecting the augmented object based on the degree of similarity with the selected specific original object an augmented object selector, an object replacement unit that deletes the selected region of the specific original object, and overwrites the selected region of the corresponding image of the augmented object in the deleted region; It is possible to provide an apparatus for augmenting learning data for object recognition including a region restoration unit for inpainting a region.

The similarity calculator of the present invention may calculate the similarity based on a distance index between the pair of object masks.

The similarity calculator of the present invention may calculate the similarity based on the similarity index between the object mask pairs.

The similarity calculator of the present invention may calculate the similarity by measuring the similarity of a feature vector of an object mask pair through an artificial neural network model.

The target object selection unit of the present invention may select, as the specific original object, an original object having a horizontal/vertical size greater than or less than a predetermined specific size of the original object area.

The augmented object selector of the present invention may select the augmented object only from original objects having different classes except for the same class.

The augmented object selector of the present invention may select the augmented object based on a predetermined criterion for the calculated similarity.

According to an embodiment of the present invention, the area of a specific object in a specific image (binary image) can be augmented by substituting another object (augmented object) similar to the corresponding object, thereby improving the learning effect of the artificial intelligence object recognition algorithm. can be increased

According to an embodiment of the present invention, it is possible to additionally generate augmented data in addition to the existing dataset by substituting regions of many similar objects in the learning data, thereby further increasing the learning effect of the artificial intelligence object recognition algorithm. can

1 is a block diagram of an apparatus for augmenting learning data for object recognition according to an embodiment of the present invention.
2 is a flowchart illustrating a main process of augmenting learning data for object recognition according to an embodiment of the present invention.
3 is an exemplary diagram schematically showing a process of creating a two-dimensional matrix M for calculating the similarity, and the sequence is as follows.
4 is an exemplary diagram schematically illustrating a process of obtaining an intersection matrix from a two-dimensional matrix M. Referring to FIG.
5 is an exemplary diagram schematically illustrating a process of obtaining a union matrix from an intersection matrix.
6 is an exemplary diagram schematically illustrating a process of creating an IoU matrix using an intersection matrix and a union matrix.
7 is a picture and an example diagram schematically showing similarity calculation using encoder networks of two different object areas.
8 is a photograph exemplarily illustrating an augmentation process of an object recognition dataset using a selected augmented object (augmented image) after deleting an original object region.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully inform the person of the scope of the invention, and the scope of the invention is only defined by the claims.

In describing the embodiments of the present invention, detailed descriptions of well-known functions or configurations will be omitted except when it is actually necessary to describe the embodiments of the present invention. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

This embodiment proposes a method of augmenting a region of a specific object in a specific image (binary image) by substituting another object similar to the corresponding object.

After the object area of the image is covered with another object and augmented with a new image, the new image may be augmented by inpainting the existing object area similarly to the background.

At this time, the similarity between the object areas of the learning data is measured through a specific index so that the erased object area and the area of the newly covered image are similar, and substitution is possible only when this index is greater than or equal to a specific value (eg, mIoU 0.8, etc.) can be set.

As a method of obtaining the similarity, a method of obtaining a distance index between masks (e.g., Manhattan distance), a method of obtaining a similarity index between masks (e.g., Intersection over Union), or a feature vector of two object images through an artificial neural network model ( It can also be obtained by measuring the similarity of feature vectors).

1 is a block diagram of an apparatus for augmenting learning data for object recognition according to an embodiment of the present invention.

Referring to FIG. 1 , the learning data augmentation apparatus for object recognition according to the present embodiment includes a mask collecting unit 102 , a similarity calculating unit 104 , a target object selecting unit 106 , an augmented object selecting unit 108 , It may include an object replacement unit 110 , a region restoration unit 112 , and an augmented object DB 114 .

The mask collection unit 102 may perform a function such as collecting an object mask, which is a binary image including an object image, a label (class of an object, horizontal and vertical regions, etc.) and an object region.

The similarity calculating unit 104 may perform a function such as calculating the similarity of the object masks collected by the mask collecting unit 102 .

The similarity calculator 104 calculates the similarity based on the distance index between the pair of object masks, calculates the similarity based on the similarity index between the pair of object masks, or a feature vector of the pair of object masks through an artificial neural network model. The similarity can be calculated by measuring the similarity of

It is necessary to calculate the similarity of all collected object mask pairs, and IoU can be used as a method of similarity calculation for each object mask pair. Here, IoU is an index for quantitatively determining the degree of overlap between two masks. Calculation of IoU can be done as shown in the following figure.

In this embodiment, a method for efficiently performing IoU comparison is presented. When the number of object regions is N, since IoU values of all object pairs must be calculated, the number of IoU calculations has a complexity of O(N^2). Since this is a fairly large amount of computation, it is efficiently calculated through the following method.

3 is an exemplary diagram schematically showing a process of creating a two-dimensional matrix M for calculating the similarity, and the sequence is as follows.

1. For all objects in the dataset, load the masks.

2. After resizing all the masks to the same size, flatten them to make them one-dimensional.

개의 마스크들을 스택(stack)하여

모양의 2차원 행렬

을 만든다.3. Flattened

by stacking the masks

2D matrix of shapes

makes

4 is an exemplary diagram schematically illustrating a process of obtaining an intersection matrix from a two-dimensional matrix M. Referring to FIG.

를 하여

크기의 행렬을

를 구한다. 행렬

는

번째 마스크와

번째 마스크의 인터섹션 영역의 크기라고 할 수 있다.Referring to Figure 4,

by doing

matrix of size

save procession

Is

the second mask

It can be said to be the size of the intersection area of the th mask.

5 is an exemplary diagram schematically illustrating a process of obtaining a union matrix from an intersection matrix.

의 대각 성분

는 i번째 마스크 영역의 크기를 의미하므로, 대각성분 벡터

를 뽑아 가로 세로 방향으로 반복(repeat)하면 sum matrix A 와 sum matrix B를 구할 수 있다. A + B ?? M을 계산함으로써 Union matrix U를 얻을 수 있다.

는 i번째 마스크와 j 번째 마스크의 Union 영역의 크기를 의미하게 된다.5, the matrix

diagonal component of

is the size of the i-th mask region, so the diagonal component vector

By extracting and repeating horizontally and vertically, sum matrix A and sum matrix B can be obtained. A + B ?? By calculating M, we can get the union matrix U.

denotes the size of the union area of the i-th mask and the j-th mask.

6 is an exemplary diagram schematically illustrating a process of creating an IoU matrix using an intersection matrix and a union matrix.

을

로 나누면 IoU matrix R이 만들어지는데,

는 i와 j 번째 마스크의 IoU 값을 의미한다.Referring to Figure 6,

second

Dividing by , the IoU matrix R is created,

denotes the IoU value of the i and j-th masks.

Algorithms created through a series of processes as described above can operate more than hundreds of thousands of times faster as shown in the table below, compared to IoU operation by a general double for statement.

GPU (Bone Algorithm) CPU (double for statement) 0.00213 seconds 506.11142 seconds

(GPU: Based on one Titan XP, IoU matrix operation time comparison for 1,920 object masks of 224x224 size. 237,394 times difference)

7 is a picture and an example diagram schematically showing similarity calculation using encoder networks of two different object areas.

Referring to FIG. 7 , similarity may be calculated using an embedding vector through an artificial neural network encoder.

That is, an image region may be expressed as a vector of an embedding space through an artificial neural network encoder, and a distance (eg, Euclidean distance) of these vectors may be calculated and utilized as a similarity.

The target object selection unit 106 may perform a function such as selecting a specific original object (original image of the augmentation target) to be replaced with the augmented object among original objects in the object image based on the similarity calculation result.

The target object selector 106 may select an original object having a horizontal/vertical size of a predetermined size (eg, 200 pixels, etc.) or smaller than or equal to a specific original object area of the original object area.

The augmented object selector 108 may perform a function such as selecting an augmented object (augmented image) based on the similarity with a specific original object selected by the target object selector 106 .

The augmented object selection unit 108 selects the augmented object only from the original object having a different class except for the same class, or selects the augmented object by using a preset specific criterion (eg, IoU 0.8 or higher) for the calculated similarity. To this end, data (image data) of numerous augmented objects may be stored in the augmented object DB 114 .

The object replacement unit 110 deletes the area of the specific original object selected through the target object selection unit 106 and covers the deleted area with the corresponding image area of the augmented object selected through the augmented object selection unit 108 It can perform functions such as covering (replacing). That is, the object replacement unit 110 overwrites the augmented object image area after resizing the augmented object to a size similar to that of the original object.

8 is a photograph illustrating an augmentation process of an object recognition dataset using a selected augmented object (augmented image) after deleting an original object region.

Referring to FIG. 8 , the region restoration unit 112 may perform a function such as inpainting the erased residual background region in the region of the specific original object.

That is, the region restoration unit 112 may use an inpaint algorithm to naturally paint the remaining region similar to the background after deleting the original object (original image).

For the restoration algorithm, general fast marching, Navier-stokes, etc. methods can be used, and an in-painting algorithm based on a neural network can also be used.

And, the finally completed augmented image can be used as a training dataset for the deep learning model.

2 is a flowchart illustrating a main process of augmenting learning data for object recognition according to an embodiment of the present invention.

Referring to FIG. 2 , the mask collecting unit 102 collects an object mask, which is a binary image including an object image, a label (object class, horizontal and vertical regions, etc.), and an object region (step 202).

The similarity calculating unit 104 calculates the similarity of each object pair of the object masks collected through the mask collecting unit 102 (step 204).

To this end, the similarity calculator 104 calculates the similarity based on the distance index between the object mask pairs, calculates the similarity based on the similarity index between the object mask pairs, or a feature vector ( The similarity can be calculated by measuring the similarity of feature vectors).

The target object selection unit 106 selects a specific original object (original image of the augmentation target) to be replaced with the augmented object from among original objects in the object image based on the calculation result of the degree of similarity (step 206).

For example, the target object selection unit 106 may select an original object having a horizontal/vertical size of a preset specific size (eg, 200 pixels, etc.) or less as the specific original object.

The augmented object selector 108 selects an augmented object (augmented image) based on the similarity with the specific original object selected by the target object selector 106 (step 208), for example, a class other than the same class. The augmented object may be selected only from the original object having .

The object replacement unit 110 deletes the area of the specific original object selected through the target object selection unit 106 and covers the deleted area with the corresponding image area of the augmented object selected through the augmented object selection unit 108 Cover (substitute object) (step 210). At this time, the object replacement unit 110 overwrites the augmented object image area after resizing the augmented object to a size similar to that of the original object.

The area restoration unit 112 restores the erased residual background area within the area of the specific original object (step 212). An inpaint algorithm can be used.

Here, in the case of the restoration algorithm, general fast marching, Navier-stokes, etc. methods may be used, and an in-painting algorithm based on a neural network may also be used.

Meanwhile, combinations of each block in the accompanying block diagram and each step in the flowchart may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general-purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions executed by the processor of the computer or other programmable data processing equipment are not identical to each block in the block diagram or in the flowchart. Each step creates a means for performing the described functions.

These computer program instructions may also be stored in a computer-usable or computer-readable memory, etc., which may be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable memory. The instructions stored in the block diagram may also produce an item of manufacture containing instruction means for performing a function described in each block of the block diagram or each step of the flowchart.

And, since the computer program instructions may be mounted on a computer or other programmable data processing equipment, a series of operating steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other program It is also possible that instructions for performing the possible data processing equipment provide steps for carrying out the functions described in each block of the block diagram and in each step of the flowchart.

In addition, each block or each step may represent a module, segment, or part of code including at least one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative embodiments it is also possible for the functions recited in blocks or steps to occur out of order. For example, it is possible that two blocks or steps shown one after another may in fact be performed substantially simultaneously, or that the blocks or steps may sometimes be performed in the reverse order according to the corresponding function.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains may make various substitutions, modifications, and changes within the scope not departing from the essential characteristics of the present invention. It will be easy to see that this is possible. That is, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments.

Accordingly, the protection scope of the present invention should be interpreted by the claims described below, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

102: mask collection unit
104: similarity calculator
106: target object selection unit
108: augmented object selection unit
110: object replacement part
112: area restoration unit
114: augmented object DB

Claims (16)

acquiring an object mask, which is a binary image comprising an object image and a label and object region;
Computing the pairwise similarity of the collected object masks in a way that measures the similarity using the IoU (Intersection over Union) index or the similarity of the feature vector of the object mask pair through an artificial neural network model;
selecting a specific original object to be replaced with an augmented object from among the original objects in the object image based on the calculation result of the degree of similarity;
selecting an augmented object based on the degree of similarity to the selected specific original object;
deleting the selected region of the specific original object, and overwriting the deleted region with the region of the corresponding image of the selected augmented object;
and inpainting the erased residual background area within the area of the specific original object.
A method of augmenting training data for object recognition. delete delete delete The method of claim 1,
The step of selecting the specific original object comprises:
Selecting an original object whose horizontal and vertical size of the original object area is greater than or equal to a preset specific size as the specific original object
A method of augmenting training data for object recognition. The method of claim 1,
The step of selecting the augmented object is,
Selecting the augmented object only from the original object having a different class except for the same class
A method of augmenting training data for object recognition. The method of claim 1,
The step of selecting the augmented object comprises:
Selecting the augmented object based on a specific criterion set for the calculated similarity
A method of augmenting training data for object recognition. As a computer-readable recording medium storing a computer program,
The computer program, when executed by a processor,
acquiring an object mask, which is a binary image comprising an object image and a label and object region;
Computing the pairwise similarity of the collected object masks in a way that measures the similarity using the IoU (Intersection over Union) index or the similarity of the feature vector of the object mask pair through an artificial neural network model;
selecting a specific original object to be replaced with an augmented object from among the original objects in the object image based on the calculation result of the degree of similarity;
selecting an augmented object based on the degree of similarity to the selected specific original object;
deleting the selected region of the specific original object, and overwriting the deleted region with the region of the corresponding image of the selected augmented object;
Inpainting the erased residual background area within the area of the specific original object
A computer-readable recording medium comprising a. As a computer program stored in a computer-readable recording medium,
The computer program, when executed by a processor,
acquiring an object mask, which is a binary image comprising an object image and a label and object region;
Computing the pairwise similarity of the collected object masks in a way that measures the similarity using the IoU (Intersection over Union) index or the similarity of the feature vector of the object mask pair through an artificial neural network model;
selecting a specific original object to be replaced with an augmented object from among the original objects in the object image based on the calculation result of the degree of similarity;
selecting an augmented object based on the degree of similarity to the selected specific original object;
deleting the selected region of the specific original object, and overwriting the deleted region with the region of the corresponding image of the selected augmented object;
restoring (inpaint) the erased residual background area within the area of the specific original object;
A computer program comprising instructions for causing the processor to perform an operation comprising: a mask collecting unit for collecting an object mask, which is a binary image including an object image, a label, and an object region;
A similarity calculator that calculates the pairwise similarity of the collected object masks by using the IoU (Intersection over Union) index or by measuring the similarity of the feature vectors of the object mask pairs through an artificial neural network model;
a target object selection unit for selecting a specific original object to be replaced with an augmented object from among the original objects in the object image based on the calculation result of the degree of similarity;
an augmented object selection unit for selecting an augmented object based on the degree of similarity to the selected specific original object;
an object replacement unit that deletes the selected region of the specific original object and overwrites the selected region of the augmented object with the corresponding image region;
and a region restoration unit for inpainting the erased residual background region within the region of the specific original object.
Learning data augmentation device for object recognition. delete delete delete 11. The method of claim 10,
The target object selection unit,
Selecting an original object whose horizontal and vertical size of the original object area is greater than or equal to a preset specific size as the specific original object
Learning data augmentation device for object recognition. 11. The method of claim 10,
The augmented object selection unit,
Selecting the augmented object only from the original object having a different class except for the same class
Learning data augmentation device for object recognition. 11. The method of claim 10,
The augmented object selection unit,
Selecting the augmented object based on a specific criterion set for the calculated similarity
Learning data augmentation device for object recognition.

Images