KR20240060163A - Method of acquiring object segmentation information through trained neural network and server system performing the same method - Google Patents

Abstract

The present invention relates to a method for object segmentation. The method of obtaining object segmentation information through a learned neural network performed in a computing device according to the present invention inputs a target image into an encoder and reduces the characteristics of at least one object in the target image. outputting the feature map; Inputting the output feature map into a decoder to generate a segmentation map with expanded features of the object; And obtaining the object segmentation information by refining the uncertain mask in the image based on the edge of the object detected in the segmentation map, wherein the encoder uses the label of the object to determine the object in the feature map. It is preferable that the difference between the active map that activates the area and the reconstruction map reconstructed in pixel units from the active map is learned as a loss. According to the present invention, it is possible to generate more accurate object segmentation results, create labels using the segmentation results, and use them for learning neural networks.

Description

Method of acquiring object segmentation information through trained neural network and server system performing the same method}

The present invention relates to a method for object segmentation.

With the development of artificial intelligence technology, artificial intelligence is being applied to various technical fields.

In particular, various algorithms based on deep learning are being developed to track objects through mathematical operations using the pixel values of the input image and classify the tracked objects, and perform convolution operations on feature values defined as matrices. CNN (Convolution Neural Network) neural network models, which are combined with multiple layers that perform, are optimized and used according to the domain to which they are applied.

Specifically, it is being developed into a detection technology that classifies objects and provides the location of the object in the image in the form of a bounding box, and a segmentation technology that extracts the area by segmenting the object in pixel units.

For example, in a 2015 MIT paper (Learning Deep Features for Discriminative Localization, [Submitted on 14 Dec 2015], https://arxiv.org/abs/1512.04150), an algorithm called CAM (Class Activation Maps) is used for convolutional networks. Using the feature map, which is the final output, it displays in the form of a heat map which features the neural network model used as important factors when making a decision, and presents rough decision elements for the entire image so that it can be used as labeling information for learning.

However, in order to generate and utilize the pixel-level segmentation results of an object using CAM as label data for weakly supervised learning, a more precise classification process may be necessary, and therefore a specific method needs to be presented.

The purpose of the present invention is to propose a neural network structure for object segmentation.

The purpose of the present invention is to propose a specific learning method of a neural network for segmentation.

Additionally, the purpose of the present invention is to propose a method for generating segmentation data for learning using the output of a neural network.

Additionally, the purpose of the present invention is to propose a method for defining the optimal loss according to the purpose of the neural network.

Additionally, the purpose of the present invention is to propose a method for precise synthesis of objects in multiple images by utilizing segmentation data of a neural network.

Additionally, the purpose of the present invention is to propose a method for augmenting high-quality data for learning neural networks.

A method of obtaining object segmentation information through a learned neural network performed in a computing device according to an embodiment of the present invention for solving the above technical problem involves inputting a target image into an encoder and abbreviating the characteristics of at least one object in the target image. outputting the feature map; Inputting the output feature map into a decoder to generate a segmentation map with expanded features of the object; And obtaining the object segmentation information by refining the uncertain mask in the image based on the edge of the object detected in the segmentation map, wherein the encoder uses the label of the object to determine the object in the feature map. It is preferable that the difference between the active map that activates the area and the reconstruction map reconstructed in pixel units from the active map is learned as a loss.

The reconstructed map includes generating a first reconstructed map by detecting a false negative area according to correlation between pixels in the active map; And it is preferable that the reconstruction be performed by detecting a false positive area according to the affinity between pixels in the activation map and generating a second reconstruction map.

It is preferable that the decoder learns the difference between the segmentation map and the reconstructed segmentation map filtered according to the per-object probability of the pixel unit of the second reconstruction map as a loss.

Obtaining the object segmentation information further includes generating a second segmentation map by detecting a false positive area according to similarity between pixels in the segmentation map, and generating the object segmentation information from the generated second segmentation map. It is desirable to obtain.

The step of obtaining the object segmentation information further includes generating a third segmentation map filtered according to a probability for each object in pixel units of the second segmentation map, and dividing the uncertain mask in the third segmentation map into the third segmentation map. 2 It is desirable to refine using superpixels generated based on the edges of objects detected in the segmentation map.

It further includes generating a composite image in which objects are synthesized using segmentation information for each object in the first and second target images.

The step of outputting the feature map includes outputting a synthesized feature map in which the features of a plurality of objects in the synthesized image are abbreviated to the encoder, wherein the encoder uses the labels of the first and second objects in the synthesized image to Between a first activation map that activates the regions of the first and second objects in the composite feature map and a second composite map that synthesizes the reconstruction maps reconstructed from the activation maps of each of the first target image and the second target image. It is desirable to learn the difference as a loss.

In the step of generating the segmentation map, the decoder generates a synthetic segmentation map in which features of the object in the synthetic feature map are expanded, and the decoder generates a synthetic segmentation map for the synthetic segmentation map and the first and second target images, respectively. It is preferable that the third reconstruction map filtered according to the probability of each object is learned as a loss based on the difference between the synthesized synthetic reconstruction maps.

An image augmentation method using a learned neural network performed in a computing device according to an embodiment of the present invention to solve the above technical problem includes receiving a plurality of target images as input; Inputting each target image into an encoder and outputting a feature map in which features of at least one object in the target image are abbreviated; Inputting the output feature map into a decoder to generate a segmentation map with expanded features of the object; Obtaining the object segmentation information by refining an uncertain mask in the image based on the edge of the object detected in the segmentation map; and generating a composite image in which an object is synthesized using segmentation information for each object in the target image, wherein the encoder generates an activity map that activates an area of the object in the feature map using the label of the object and the activity. It is preferable that the difference between the reconstructed maps reconstructed in pixel units from the map is learned as a loss.

According to the present invention, more accurate object segmentation results can be generated.

In addition, the present invention can shorten the labeling time for learning a segmentation model using segmentation results compared to the existing method, and can learn a segmentation model using labels generated accurately and quickly.

In addition, the present invention can achieve higher performance improvement by individually training the encoder and decoder using the intermediate outputs of the network of the neural network model composed of the encoder and decoder, and can generate more accurate segmentation results without additional resources. .

Additionally, the present invention can generate various synthetic images by combining objects using a mask obtained as a result of segmentation.

1 is a diagram illustrating a process for obtaining object segmentation information according to an embodiment of the present invention.
Figure 2 is a diagram showing the operation pipeline of a neural network that acquires object segmentation information according to an embodiment of the present invention.
Figure 3 is a diagram showing a detailed operation pipeline of a neural network that acquires object segmentation information according to an embodiment of the present invention.
Figure 4 is a diagram showing a learning pipeline of a neural network that acquires object segmentation information according to an embodiment of the present invention.
Figure 5 is a diagram illustrating the process of generating a number label for first learning of a neural network that acquires object segmentation information according to an embodiment of the present invention.
Figure 6 is a diagram illustrating the process of generating a capital label for second learning of a neural network acquiring object segmentation information according to an embodiment of the present invention.
Figure 7 is a diagram illustrating the operation pipeline of a neural network for obtaining object segmentation information for a plurality of target images according to an embodiment of the present invention.
Figures 8 and 9 are diagrams showing a post-processing process for obtaining object segmentation information according to an embodiment of the present invention.
Figure 10 is a diagram illustrating a process for acquiring a composite image through object segmentation information according to an embodiment of the present invention.
Figure 11 is a diagram showing an additional learning pipeline of a neural network that acquires object segmentation information according to an embodiment of the present invention.
FIG. 12 is a diagram illustrating the implementation of a computing device that performs a method of obtaining object segmentation information according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, a person skilled in the art can invent various devices that embody the principles of the invention and are included in the concept and scope of the invention, although not clearly described or shown herein. In addition, all conditional terms and embodiments listed in this specification are, in principle, clearly intended only for the purpose of ensuring that the inventive concept is understood, and should be understood as not limiting to the specifically listed embodiments and states. .

The above-mentioned purpose, features and advantages will become clearer through the following detailed description in relation to the attached drawings, and accordingly, those skilled in the art in the technical field to which the invention pertains will be able to easily implement the technical idea of the invention. .

Additionally, when describing the invention, if it is determined that a detailed description of the known technology related to the invention may unnecessarily obscure the gist of the invention, the detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

Hereinafter, the segmentation process of the neural network according to this embodiment will be described in more detail with reference to FIG. 1.

1 is a flowchart showing an object segmentation method of a computing device according to an embodiment of the present invention.

In this embodiment, the computing device may be implemented as a single computer or server including a processor, or a server system composed of a plurality of servers.

Therefore, in addition to performing segmentation from images received in a single process, the computing device can be configured to enable network communication, and can be implemented in a cloud form to receive and process images captured by various photographing devices.

Additionally, in addition to the processor, the computing device may be configured to include a cloud-based memory device for using the collected images as learning data or for augmentation of the learning data.

For example, the computing device according to this embodiment can be implemented as an edge computer that directly obtains segmentation information from images received from cameras such as CCTV installed at the edge and can perform a segmentation process. Alternatively, it is possible to implement batch processing on the collected images for learning or inference of a neural network that performs segmentation by implementing it as a central server, and provide the inference results in the form of a response to an API call.

First, the computing device according to this embodiment receives the segmentation target image (I) for segmentation (S10).

In this embodiment, segmentation may mean classifying the object included in the image and classifying the area where the object is located on a pixel-by-pixel basis, in addition to detecting the location of the object in the form of a bounding box.

Detection of an object in an image can be performed by providing the area where the object is located in the form of a bounding box determined by coordinates, height, and size, and the image can be used for learning by using the detection result as a label.

However, false detection problems may occur due to unnecessary parts for learning being included in the area extracted as a bounding box, which may cause a decrease in the learning performance of the neural network. Therefore, more precise detection is needed to increase the learning effect of the neural network. Additionally, in order to create various composite images by combining the background and the object, it is necessary to more precisely distinguish the object area on a pixel basis.

In this embodiment, the segmentation of the neural network aims at semantic segmentation by dividing the image by pixel and classifying it based on a defined class, but also classifies each object by unit even for the same class (instance ( It is also possible to obtain segmentation information.

In this embodiment, the input segmentation target image (I) is captured using a camera device, and may be a large area captured at once for faster tracking depending on the purpose, or may be a large screen image.

Large screen images can be acquired, for example, in the form of aerial photography, for example, by acquiring ground images taken by a drone during flight, and segmenting objects corresponding to an area of very small size compared to the entire acquired area more precisely. can do.

Next, the computing device according to this embodiment inputs the target image (I) into the encoder 110 and outputs a feature map in which the features of at least one object in the target image (I) are abbreviated (S20).

The encoder 110 may be configured as a convolutional network that performs a general convolution operation. A convolutional network consists of a plurality of convolutional layers. The convolutional layers perform a convolution operation between the internal filter and the pixel values of the image for each channel, so that the features related to the object are emphasized and a condensed feature map is output. You can.

Next, the computing device according to this embodiment inputs the output feature map into the decoder 120 to generate a segmentation map with expanded features of the object (S30).

Specifically, the decoder 120 may be implemented in a shape symmetrical to the encoder 110 and may therefore be configured as a deconvolution network that performs a deconvolution operation.

That is, each layer in the deconvolution network uses the feature map, which is the output of the encoder 110, as input, maintains the location information of the feature information in the feature map, and expands it so that the features can be expressed well in the size of the original image. .

According to the symmetrical structure, the size and channel of the final output feature map of the encoder 110 may have a size and channel according to the input of the decoder 120, and the encoder 110 passes through a deconvolution layer inside the decoder 120. Through the output of (110), a segmentation map in which the location information of features is restored from the original image size can be output.

In order to properly learn the neural network consisting of the encoder 110 and the decoder 120 according to this embodiment, accurate labels for the feature information appearing in the abbreviated feature map and the segmentation map expanded by the decoder 120 are required. This may be necessary.

However, in order to secure an accurate label for each feature map, a handwritten label may be necessary, so for more efficient learning, in this embodiment, the image generated through the output or processing of an additional network is used as a pseudo label. Thus, learning can be performed through semantic-based weakly-supervised semantic segmentation.

The specific learning method will be described with reference to the learning pipeline including the neural network shown in FIG. 2.

Figure 2 is an exemplary diagram showing the structure of a learning pipeline according to an embodiment of the present invention.

Referring to FIG. 2, the encoder 110 generates and outputs a feature map using the input target image (I) as described above, and the feature map is converted into a segmentation map ( ) can be output in the form.

At this time, the pipeline for learning may be divided into a first learning pipeline for learning of the encoder 110 and a second learning pipeline for learning of the decoder 120.

First, to describe the first learning pipeline, the output of the encoder 110 is an active map ( ) can be created in the form.

Referring to Figure 3, the active map ( ) To describe in more detail the network structure that performs output in the form, as described above, the final feature map as the output of the encoder 110 may be configured in a form according to the channel and size predetermined in the classification unit 130. . Preferably, the channel of the final feature map can be determined according to the class to be classified, and the feature map is structured in the form of a matrix in which major features are highlighted according to location to identify objects within the class for each channel.

The feature map output from the above convolution layer can be flattened through a global average pooling (GAP: Global Average Pooling) layer.

Specifically, the average of the values of the feature map for each channel can be calculated as one vector value through a global average pooling layer, and thus a one-dimensional array consisting of one vector for each channel can be obtained.

Next, the output of the global average pooling layer is calculated with weights (W ₁ , W ₂ , … W _n ) defined for each class of the object to be classified. Each weight may indicate the importance of each channel in the corresponding class. The calculation of weights and vector values is given as an input to Softmax as an activation function, and the object classification results can be finally output in the form of probability values. In this embodiment, an activity map is created using the class with the highest probability as a label according to the above classification results.

For example, in Figure 3, if the object in the image is classified as an airplane and the result (Y) is output, the active map can be created using the internal computation network using the result.

Specifically, when the classification result is output as an airplane, the internal computation network multiplies the weights (W ₁ , W ₂ , … W _n ) used in the judgment of the airplane with the feature maps for each channel before pooling corresponding to each weight to determine the characteristics. You can check which pixel value corresponding to a location on the map influenced the airplane's decision. Therefore, by combining the feature maps for each channel with weights, an activation map in the form of a two-dimensional image can be output and intuitively indicate the location of the object and the importance of the object's features.

Finally, the weighted values of the feature maps for each channel are weighted and added, and relatively important pixel areas are emphasized, thereby creating an activation map in the form of a heat map that indirectly represents information about the location of the object.

In other words, pixels that have a major influence on classification have a larger value than pixels that do not, so they can have emphasized colors accordingly, and areas of objects can be distinguished based on pixel values.

However, as described above, it is difficult for the activation map output from the feature map of the encoder 110 to include detailed location information of the object, so additional processing is necessary. Therefore, in this embodiment, it is obtained during an iterative learning process. The encoder 110 is recursively trained using the reconstructed map by refining the area of the object in the active map in more detail.

Specifically, referring to FIG. 4, the reconstruction map ( ) can be used as a pseudo label to perform learning. That is, the active map generated in that cycle ( ) and active map ( ) Reconstruction map created by reconstruction from ( By defining the difference between ) as loss, the layers inside the encoder 110 can be updated by performing iterative learning in the direction of reducing the loss.

This learning process can be repeated as an update process through the reconstruction map generated using the output activation map, and therefore, the activation map ( ) is the activity map of the previous cycle ( ) can produce more detailed segmentation results than .

At this time, the number of repetitions, etc. may be determined according to predetermined hyperparameters.

Next, the process of reconstructing the active map according to this embodiment will be described in more detail.

Referring to FIG. 5, the reconstruction process can be divided into two stages.

First, it can be performed separately into a first reconstruction process that expands the area information of the object expressed in the active map and a second reconstruction process that performs refinement into a more detailed area from the expanded first reconstruction map.

First, the first reconstruction process will be described. In this embodiment, the first reconstruction process calculates the similarity between pixels in the feature map, and uses the output of the activation function of the calculated similarity to create a reconstruction map whose area is expanded according to the similarity. can be created. For example, in the activation map output from the feature map of the encoder 110, pixels that are determined to be the object area and pixels that have a similarity higher than a reference value can be expanded to include the object area.

Specifically, the first reconstruction (SCG, self-correlation map generating) module outputs the similarity between pixels in the feature map and the average similarity between non-adjacent pixels, takes the maximum value as the similarity through pixel-level similarity comparison, and determines the similarity. 1 If it exceeds the threshold, the first reconstruction map ( ) can be created. Specifically, the first reconstruction map is created by detecting and expanding the false negative area.

Accordingly, the first reconstruction map includes an area of the object expanded compared to the active map according to the similarity between pixels.

Next, a second reconstruction is performed to refine the expanded area of the first reconstruction map in more detail.

The second reconstruction process uses the first reconstruction map as input through an affinity matrix separately defined in the second reconstruction (PAMR, pixel-adaptive mask refinement) module. ) is generated, but the calculated values are corrected to have the same probability value as surrounding color values are similar, and a reconstruction map (false positive area) is removed. ) can be created. At this time, it is also possible to perform a region removal process for each channel of the first reconstruction map.

The first learning pipeline can perform learning by updating the layers inside the encoder 110 in the direction of reducing the error between the outputs of the encoder 110 using the reconstructed map generated from the output active map as a capital label. there is.

Next, the second learning pipeline will be described.

The second learning pipeline is a learning process of the decoder 120 through the output of the decoder 120, and learning can be performed using the capital label for the segmentation map as the output of the decoder 120.

That is, the feature map refined in the learned encoder 110 can be output as detailed segmentation information for the region through the decoder 120.

At this time, segmentation can use the reconstruction map generated in the first learning pipeline as a capital label for more accurate classification of objects. Specifically, the reconstruction map performs an additional process to convert the form generated from the active map into segmentation information.

Referring to Figure 6, in this embodiment, the reconstruction map ( ) can be reconstructed into segmentation information according to a specific threshold probability through a filter (CF, CertainFilter). Reconstructed segmentation map generated through the above process ( ) is the segmentation map output from the decoder ( ) is also used as a label.

Therefore, the decoder 120 uses a segmentation map ( ) and reconstructed segmentation map ( ) By defining the difference between them as loss, iterative learning can be performed in the direction of reducing the difference.

The computing device according to this embodiment acquires segmentation information through the neural network learned through the above process (S40).

In this embodiment, object segmentation information is obtained based on the edges of objects detected in the segmentation map.

Referring to FIG. 7, specifically, the step of acquiring object segmentation information is a segmentation map ( ) by detecting the edges within the third segmentation map ( ) and generate a third segmentation map ( ), object segmentation information that distinguishes the object and the background can be obtained.

At this time, in order to obtain object segmentation information, the segmentation map ( ) is possible to further process, and preferably, the segmentation map ( ) can be precisely purified.

Referring to FIG. 8, a second segmentation map ( ) and create a second segmentation map ( ) A third segmentation map filtered according to the probability of each pixel object ( ) can be created.

Specifically, the second segmentation map ( ) My uncertain mask is set to the second segmentation map ( ) can be performed through a third reconstruction process (EP, EdgePrediction) that obtains segmentation information by refining using superpixels generated based on the edges of the detected object.

The third reconstruction process according to this embodiment will be described with reference to FIG. 9.

Referring to FIG. 9, first, the third reconstruction process is a second segmentation map (which refines the segmentation map that is the output of the decoder 120 for the input image). ) Edges can be detected from . Edge detection can be detected based on the amount of change in pixel value according to conventional technology (A Computational Approach to Edge Detection, Canny 1986) and is composed of continuous line segments.

Next, pixels belonging to the same connected component are grouped together based on edges to extract a superpixel, which is defined as one pixel (Connected-component labeling (CCL)).

In addition, the second segmentation map ( ) to filter( ) is divided into a first mask that defines a certain area based on a threshold probability and a second mask that defines an uncertain area, and a process of removing the uncertain area is performed based on super pixels.

Through the above process, uncertain areas are removed from the second mask, and the remaining areas and the first mask are added to create a third segmentation map containing object segmentation information ( ) can be created.

Next, the computing device generates a third segmentation map ( ) using the object segmentation information extracted from ) to create a composite image in which the object is synthesized.

Referring again to FIG. 7, the above-described object segmentation information extraction process can be performed individually for a plurality of target images (I), and thus each object segmentation information can be extracted.

Afterwards, a new composite image can be created by compositing the image value corresponding to the extracted object segmentation information to a specific position of the composite image (I) or to a third background.

In this embodiment, the shape of the composite image can be preserved as much as possible and the sense of heterogeneity can be reduced by extracting more refined object segmentation information through the individually learned encoder 110 and decoder 120 and the output of the decoder 120.

Referring to FIG. 10, in this embodiment, object segmentation information is generated from target images corresponding to the classes of airplane and horse, and the generated object segmentation information is synthesized to create a synthetic segmentation map ( ) is created.

Next, the region within each input target image (I) corresponding to the synthetic segmentation information is extracted and synthesized to create a composite image of an airplane and a horse ( ) can be created.

Furthermore, in this embodiment, the performance of the neural network in the synthesis process can be further improved by re-performing the above-described learning pipeline of the encoder 110 and decoder 120 through the synthesized image.

Referring to Figure 11, the first and second learning pipelines described above are synthesized images ( ), but the class utilizes each target image (I) to map the active maps for horses and airplanes ( ) and the second reconstruction map ( 1-1 learning pipeline of the encoder 110 to reduce the error of ), and a segmentation map ( ) and segmentation map ( Reconstructed segmentation map (with uncertain regions removed from superpixels through the edges of ) ) The performance of the neural network can be further improved by performing the 2-1 learning pipeline of the decoder 120 to reduce the error.

Hereinafter, with reference to FIG. 12, a specific hardware implementation of a server as a computing device that performs the method of obtaining object segmentation information according to this embodiment will be described.

Referring to FIG. 12, in some embodiments of the present invention, the server 300 may be implemented in the form of a computing device. One or more of each module constituting the server 300 is implemented on a general-purpose computing processor and thus includes a processor 308, input/output I/O 302, memory 340, and interface ( 306) and a bus 314. The processor 308, input/output device 302, memory 340, and/or interface 306 may be coupled to each other through a bus 314. The bus 314 corresponds to a path through which data moves.

Specifically, the processor 308 includes a Central Processing Unit (CPU), Micro Processor Unit (MPU), Micro Controller Unit (MCU), Graphic Processing Unit (GPU), microprocessor, digital signal processor, microcontroller, and application processor (AP). , application processor) and logic elements capable of performing similar functions.

The input/output device 302 may include at least one of a keypad, a keyboard, a touch screen, and a display device. The memory device 340 may store data and/or programs.

The interface 306 may perform the function of transmitting data to or receiving data from a communication network. Interface 306 may be wired or wireless. For example, the interface 306 may include an antenna or a wired or wireless transceiver. The memory 340 is a volatile operating memory that improves the operation of the processor 308 and protects personal information, and may further include high-speed DRAM and/or SRAM.

Additionally, memory 340 stores programming and data configurations that provide the functionality of some or all of the modules described herein. For example, it may include logic to perform selected aspects of the learning method described above.

A program or application is loaded with a set of instructions including each step of performing the above-described learning method stored in the memory 340 and allows the processor to perform each step. For example, an operation of inputting a target image (I) to the encoder 110 and outputting a feature map in which the features of at least one object in the target image (I) are abbreviated, and sending the output feature map to the decoder 120. An operation of inputting and generating a segmentation map with expanded features of the object, and an operation of obtaining the object segmentation information by refining an uncertain mask in the image based on the edge of the object detected in the segmentation map. The included computer program may be executed by a processor.

According to the present invention described above, more accurate object segmentation results can be generated.

Additionally, the present invention can generate a label using the segmentation result and use it for learning of a neural network.

In addition, the present invention can achieve higher performance improvement by individually training the encoder 110 and the decoder 120 using the intermediate outputs of the neural network consisting of the encoder 110 and the decoder 120, and additional resources. More accurate segmentation results can be generated without.

Additionally, the present invention can generate various synthetic images by combining objects using a mask obtained as a result of segmentation.

Furthermore, various embodiments described herein may be implemented in a recording medium readable by a computer or similar device, for example, using software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). In some cases, it may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. The described embodiments may be implemented as a control module itself.

According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code can be implemented as a software application written in an appropriate programming language. The software code may be stored in a memory module and executed by a control module.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications, changes, and substitutions can be made by those skilled in the art without departing from the essential characteristics of the present invention. will be.

Accordingly, the embodiments disclosed in the present invention and the attached drawings are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments and the attached drawings. . The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

Claims (10)

In a method of obtaining object segmentation information through a learned neural network performed on a computing device,
Inputting a target image into an encoder and outputting a feature map in which features of at least one object in the target image are abbreviated;
Inputting the output feature map into a decoder to generate a segmentation map with expanded features of the object; and
Obtaining the object segmentation information by refining the uncertain mask in the image based on the edge of the object detected in the segmentation map,
The encoder is a method of obtaining object segmentation information, characterized in that the difference between an active map that activates the area of the object in the feature map using the label of the object and a reconstruction map reconstructed in pixel units from the active map is learned as a loss. . According to claim 1,
The reconstruction map is,
generating a first reconstruction map by detecting a false negative area according to correlation between pixels in the active map; and
A method of obtaining object segmentation information, characterized in that it is reconstructed by detecting a false positive area according to the affinity between pixels in the active map and generating a second reconstruction map. According to claim 2,
The decoder is a method of acquiring object segmentation information, characterized in that the difference between the segmentation map and the reconstructed segmentation map filtered according to the pixel-unit object probability of the second reconstruction map is learned as a loss. According to claim 1,
The step of acquiring the object segmentation information is,
Further comprising generating a second segmentation map by detecting a false positive area according to the similarity between pixels in the segmentation map,
A method of obtaining object segmentation information, characterized in that obtaining the object segmentation information from the generated second segmentation map. According to claim 4,
The step of obtaining the object segmentation information is,
Further comprising generating a third segmentation map filtered according to the pixel-level object-specific probability of the second segmentation map,
A method of obtaining object segmentation information, characterized in that the uncertain mask in the third segmentation map is refined using superpixels generated based on the edges of the object detected in the second segmentation map. According to claim 2,
A method of obtaining object segmentation information, further comprising generating a composite image in which an object is synthesized using each object segmentation information in the first and second target images. According to claim 6,
The step of outputting the feature map is,
The encoder outputs a composite feature map in which the features of a plurality of objects in the composite image are abbreviated,
The encoder is,
A first activation map that activates areas of the first and second objects in the composite feature map using labels of the first and second objects in the composite image, and each of the first target image and the second target image A method of acquiring object segmentation information, characterized in that the difference between the second composite map synthesized by the reconstructed map reconstructed from the active map of is learned as a loss. According to claim 7,
The step of generating the segmentation map is,
The decoder generates a synthetic segmentation map in which the features of the object in the synthetic feature map are expanded,
The decoder is,
A method of obtaining object segmentation information, characterized in that the difference between the synthetic segmentation map and the synthetic reconstruction map filtered according to the probability of each object for each of the first and second target images is learned as a loss. . In an image augmentation method using a learned neural network performed on a computing device,
Receiving a plurality of target images as input;
Inputting each target image into an encoder and outputting a feature map in which features of at least one object in the target image are abbreviated;
Inputting the output feature map into a decoder to generate a segmentation map with expanded features of the object;
Obtaining the object segmentation information by refining an uncertain mask in the image based on the edge of the object detected in the segmentation map; and
Generating a composite image in which an object is synthesized using segmentation information for each object in the target image,
The encoder is an image augmentation method characterized in that the difference between an active map that activates the area of the object in the feature map using the object's label and a reconstruction map reconstructed in pixel units from the active map is learned as a loss. A computer-readable recording medium storing a program for performing a method of acquiring object segmentation information through a learned neural network performed in the computing device according to any one of claims 1 to 8.

Images