KR102338372B1 - Device and method to segment object from image - Google Patents

Abstract

A method and apparatus for segmenting an object from an image are provided. The apparatus for segmenting an object from an image according to an embodiment may segment an object image from an input image by using a pre-learned image model.

Description

Method and apparatus for segmenting an object from an image {DEVICE AND METHOD TO SEGMENT OBJECT FROM IMAGE}

Hereinafter, a technique for segmenting an object from an image is provided.

In the field of image-related technologies, a technology for recognizing an object such as a human face using an image has recently been developed. In order to recognize an object such as a face, it is necessary to extract a portion excluding the background from the image.

For example, in order to extract a portion excluding a background from an image, a technique for segmenting an object based on depth information may be used. However, the object segmentation technology based on depth information divides an object (eg, a human body object) by combining color information and depth information, and a module for obtaining depth information is required in addition to a camera that obtains color information and an excessive amount of computation may be required in the process of processing depth information.

Accordingly, a technique for segmenting an object using color information is required.

According to an embodiment, a method of segmenting an object from an image includes receiving an input image including an object, and using an image model to divide the object from the input image. It may include generating a corresponding output image, and extracting an object image from the output image.

The extracting of the object image may include classifying each pixel of the output image according to a property, and extracting the object image by using the classified pixel.

The classifying may include comparing a pixel value of each pixel with a threshold value, and determining a property of each pixel based on a result of the comparison.

Extracting the object image may include generating a mask image by binarizing the output image based on a result of comparing a pixel value and a threshold value of each pixel of the output image. have.

The extracting of the object image may further include generating a foreground image based on the mask image and the input image.

The extracting of the object image may include generating a foreground image from the output image based on a result of comparing a pixel value of each pixel of the output image with a threshold value.

The image model may be configured such that a resolution of the object image generated from the input image is the same as a resolution of the input image.

The image model may include a neural network, and an activation function of the neural network may include at least one nonlinear function.

An apparatus for segmenting an object from an image according to an embodiment receives an input image including a memory for storing an image model and an object, and generates an output image corresponding to the object from the input image by using the image model and a processor for extracting an object image from the output image.

The processor may classify each pixel of the output image according to a property, and extract an object image using the classified pixel.

The processor may compare a pixel value of each pixel with a threshold value, and determine the property of each pixel based on a result of the comparison.

The processor may generate a mask image by binarizing the output image based on a result of comparing a pixel value of each pixel of the output image with a threshold value.

The processor may generate a foreground image based on the mask image and the input image.

The processor may generate a foreground image from the output image based on a result of comparing a pixel value of each pixel of the output image with a threshold value.

The image model may be configured such that a resolution of the object image generated from the input image is the same as a resolution of the input image.

The image model may include a neural network, and an activation function of the neural network may include at least one nonlinear function.

According to an embodiment, a method for learning object division from an image includes: receiving a reference training image and a reference object image; and learning parameters of the image model so that a processor divides the reference object image from the reference training image by using an image model that generates an output image corresponding to the object from an input image including the object. can do.

The image model may include a neural network including at least one nonlinear function as an activation function, and the neural network may be configured such that a resolution of the output image generated from the input image is the same as that of the input image. .

The image model is based on an image on which at least one of rotation, resize, shift, flip, and noise adding is performed on the reference training image. can be learned by

According to another exemplary embodiment, a method of segmenting an object from an image includes receiving an input image including the object, and using a first image model, an intermediate corresponding to the object from the input image. generating an intermediate image, generating an output image corresponding to the object from the intermediate image using a second image model, and extracting an object image from the output image can do.

According to another embodiment, a method for learning object division from an image includes receiving a reference training image and a reference object image, and a first image model generating an intermediate image corresponding to the object from an input image including the object. learning the parameters of the first image model so that the processor divides the reference object image from the reference training image by using a reference intermediate image from the reference training image using the first image model image), and using a second image model for generating an output image corresponding to the object from the intermediate image, the processor divides the reference object image from the reference intermediate image, the second image It may include training the parameters of the model.

1 is a diagram illustrating an object image divided from an image according to an exemplary embodiment.
2 is a flowchart illustrating a method of segmenting an object from an image according to an exemplary embodiment.
3 to 5 are diagrams illustrating examples of generating an output image from an input image by using an image model according to an exemplary embodiment.
6 is a flowchart illustrating a method of extracting an object image from an output image according to an exemplary embodiment.
7 is a flowchart illustrating a method of classifying each pixel of an output image according to an attribute, according to an exemplary embodiment.
8 and 9 are flowcharts illustrating a method of extracting an object image using classified pixels according to an exemplary embodiment.
10 is a flowchart illustrating a method of extracting an object image using pixels of an output image according to an embodiment.
11 is a block diagram illustrating a configuration of an apparatus for segmenting an object from an image according to an embodiment.
12 is a diagram illustrating a flowchart of a method of learning an image model used to segment an object from an image according to an embodiment.
13 is a diagram illustrating a configuration of an apparatus for learning an image model used to segment an object from an image according to an embodiment.
14 is a diagram illustrating an object image generated from an input image by using the image model learned in FIG. 13 according to an exemplary embodiment.
15 is a flowchart illustrating a method of segmenting an image according to another exemplary embodiment.
16 is a diagram illustrating an example of generating an output image from an input image by using an image model according to another embodiment.
17 is a flowchart illustrating a method of learning an image model used to segment an object from an image according to another embodiment;
18 is a diagram illustrating a process of learning an image model used to segment an object from an image according to another embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference numerals in each figure indicate like elements.

Various modifications may be made to the embodiments described below. It should be understood that the embodiments described below are not intended to limit the embodiments, and include all modifications, equivalents, and substitutes thereto.

Terms used in the examples are used only to describe specific examples, and are not intended to limit the examples. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 is a diagram illustrating an object image divided from an image according to an exemplary embodiment.

The apparatus for dividing an object from an image according to an embodiment may divide an object image from an input image 110 including an object.

In the present specification, an object may include a subject excluding a background, such as a human, an animal, a thing, etc., and a part of a person's face, arms, legs, and body, etc. It may contain parts of the same object.

The input image 110 may include an object as a received image. The input image 110 is a two-dimensional image, and may be, for example, a color image and a grayscale image. The input image 110 may include a plurality of pixels, and each pixel may have a pixel value. When the input image 110 is a color image, the pixel value may be a color value (eg, an RGB value, but other color spaces may be used), and the input image 110 may be a color space. ) is a grayscale image, the pixel value may be a brightness value or an intensity value. However, the input image 110 is not limited thereto, and may be a three-dimensional image, and in this case, each pixel may further include a depth value.

The object image may be an image representing the object. For example, the object image is an image in which a background part is excluded from the input image 110 , and may be a foreground image 120 including only the foreground part or a mask image 130 including only a mask part. The foreground image 120 is an image in which each pixel of a portion corresponding to the foreground has a corresponding pixel value, and the mask image 130 is a pixel of a portion corresponding to the foreground and pixels of a portion other than the foreground of the image. An image divided by this specific value (eg, a pixel value of a foreground portion is 1 and a pixel value of a rear view portion is 0) may be displayed. For example, as shown in FIG. 1 , in the foreground image 120 , the pixel value of a portion corresponding to the foreground may be maintained to be the same as that of the input image 110 , and the mask image 130 may be configured to correspond to the foreground. Pixel values of the portion and the portion corresponding to the rear view may be binarized.

Hereinafter, extracting an object image from the input image 110 will be described.

2 is a flowchart illustrating a method of segmenting an object from an image according to an exemplary embodiment.

First, the processor of the apparatus for segmenting an object from an image in operation 210 may receive an input image including the object. For example, the processor may receive the input image from the outside by wire or wirelessly, or may acquire the input image from photographing through a camera inside the device.

In step 220 , the processor may generate an output image corresponding to the object from the input image by using the image model trained to output the reference object image from the reference training image. _{A pixel value p i} of each pixel of the output image may correspond to a probability that the corresponding pixel represents an object. For example, when the output image relates to a mask, the pixel value of the output image may have a minimum value of 0 and a maximum value of 1. The pixel value may have a greater probability of representing the mask as the pixel value is closer to 1.

The image model is a model trained to output a specific output with respect to a specific input, and may represent, for example, a parameter of a machine learning structure. Machine learning structures can be expressed as black box functions in which arbitrary outputs are generated based on pre-trained parameters for specific inputs. According to an embodiment, the image model may be configured to output an output image representing an object from an input image. For example, the image model may include a connection weight as a parameter of the neural network, and may be trained to output a reference object image from a reference training image. Learning of the image model will be described in detail with reference to FIGS. 12 and 13 below.

Subsequently, in step 230 , the processor may extract an object image from the output image. For example, the processor may label the pixels of the output image by classifying them into pixels corresponding to the foreground and pixels not corresponding to the foreground. The extraction of the object image by the classification of pixels will be described in detail with reference to FIGS. 6 to 10 below.

3 to 5 are diagrams illustrating examples of generating an output image from an input image by using an image model according to an exemplary embodiment.

A case in which an image model includes a connection weight as a parameter of a neural network will be described as an example in FIGS. 3 to 5 . The neural network corresponding to the image model shown in FIGS. 3 to 5 has been trained, and the i-th pixel (here, i is an integer greater than or equal to 1) has _{a pixel value of x i} with respect to the input image. An output image having a pixel value of _{p i may be output.}

In the neural network of the present specification, artificial neurons that simplify the function of biological neurons are used, and the artificial neurons may be interconnected through a connection line having a connection weight. A connection weight, which is a parameter of a neural network, is a specific value of a connection line and can also be expressed as connection strength. Neural networks can perform human cognitive actions or learning processes through artificial neurons. An artificial neuron may also be referred to as a node.

A neural network may include a plurality of layers. For example, the neural network may include an input layer, a hidden layer, and an output layer. The input layer may receive an input for performing learning and transmit it to the hidden layer, and the output layer may generate an output of the neural network based on signals received from nodes of the hidden layer. The hidden layer is located between the input layer and the output layer, and can change the training data transmitted through the input layer into a value that is easy to predict. Nodes included in the input layer and the hidden layer may be connected to each other through a connection line having a connection weight, and nodes included in the hidden layer and the output layer may also be connected to each other through a connection line having a connection weight. The input layer, the hidden layer, and the output layer may include a plurality of nodes.

A neural network may include a plurality of hidden layers. A neural network including a plurality of hidden layers is called a deep neural network, and training a deep neural network is called deep learning. A node included in the hidden layer is called a hidden node. The output of the hidden node in the previous time interval may be connected to the hidden nodes in the current time interval. And, the output of the hidden node in the current time interval may be connected to the hidden nodes in the next time interval. A neural network with recurrent connections between hidden nodes in different time intervals is called a recurrent neural network.

In addition, the hidden layer may include, for example, a convolution layer, a pooling layer, a normalization layer, and a fully connected layer. The convolutional layer may be used to perform convolutional filtering of filtering information extracted from the previous layer using a filter of a predetermined size, and may be shown as “C” in FIGS. 3 to 5 . The pooling layer, through pooling, creates a window of a predetermined size with respect to the representative value (eg, the information of the previous layer in the pooling layer (eg, pixel values of the image) from the information of the previous layer) While sliding, extract the maximum value within the window), and may be shown as “P” in FIGS. 3 to 5 . The normalization layer may indicate a layer in which values of pixels of an image are normalized, and may be denoted as “N” in FIGS. 3 to 5 . The fully connected layer may be connected to all nodes of the previous layer, and may be shown as “F” in FIGS. 3 to 5 .

According to an embodiment, the neural network shown in FIG. 3 may include an input layer, an output layer, and six hidden layers. The input layer may receive the input image 301 . In the first layer 310 shown in FIG. 3 , C1 (64@5*5+S1) has a convolutional layer with, for example, 64 filters, each filter having a size of 5*5, and the filter is It indicates that it is moved by 1 space, P(3*3+S2) indicates that, for example, the size of the window is 3*3 and is moved by 2 spaces, P(3*3+S2) may indicate the normalization layer. The second layer 320 may include a convolution layer having 64 filters with a size of 5*5 moved by 1 space, a pooling layer and a normalization layer having a window of size 3*3 moved by 2 spaces. The third layer 330 may include a convolutional layer having 64 filters with a size of 3*3 that is moved by 1 space. Each of the fourth layer 340 and the fifth layer 350 may include a fully connected layer having 100 nodes. The sixth layer 360 may include a fully connected layer having 48*48 nodes. Here, the sixth layer 360 immediately before the output layer may be configured such that an image having the same resolution as the input layer (48*48 in FIG. 3) is output from the output layer. The neural network shown in FIG. 3 may be trained to output an output image 309 corresponding to a mask from an input image 301 .

According to another embodiment, the neural network shown in FIG. 4 may include an input layer, an output layer, and eight hidden layers. The first layer 410 shown in FIG. 4 includes a convolution layer having 48 filters with a size of 5*5 moved by 1 space, a pooling layer having a window of size 3*3 moved by 2 spaces, and a normalization layer. can do. The second layer 420 may include a convolution layer having 128 filters with a size of 5*5 moved by 1 space, a pooling layer having a window of size 3*3 moved by 2 spaces, and a normalization layer. The third layer 430 and the fourth layer 440 may include a convolutional layer having 192 filters in a 3*3 size that is moved by 1 space, respectively. The fifth layer 450 may include a convolution layer having 64 filters with a size of 3*3 that is moved by 1 space and a pooling layer having a window of size of 3*3 that is moved by 2 spaces. The sixth layer 460 and the seventh layer 470 may include fully connected layers having 1024 nodes. The eighth layer 480 may include a fully connected layer having 112*112 nodes. The eighth layer 480 may be configured such that the resolution of the input image 401 (112*112 in FIG. 4) and the resolution of the output image 409 (112*112 in FIG. 4) are the same.

According to another embodiment, the first to eighth layers 510 , 520 , 530 , 540 , 550 , 560 , 570 and 580 of the neural network shown in FIG. 5 are composed of layers having the same structure as in FIG. 4 . can be However, the neural network shown in FIG. 5 may be trained to output the output image 509 corresponding to the foreground from the input image 501 . As such, even for a neural network having the same structure, an output image may be different for the same input image according to training data.

6 is a flowchart illustrating a method of extracting an object image from an output image according to an exemplary embodiment.

According to an embodiment, FIG. 6 may show a flowchart for explaining step 230 of FIG. 2 described above in more detail.

First, in operation 610, the processor may classify each pixel of the output image according to an attribute. The property of a pixel may indicate whether a corresponding pixel corresponds to an object, a part of an object, a foreground, or a background in the output image. For example, the property of a pixel may indicate whether a corresponding pixel is a foreground in an output image. The classification of pixels will be described in detail with reference to FIG. 7 below.

And in step 620, the processor may extract an object image by using the classified pixels. For example, the processor may generate an object image by collecting pixels classified as objects. The extraction of the object image using pixels will be described in detail with reference to FIGS. 8 to 10 below.

7 is a flowchart illustrating a method of classifying each pixel of an output image according to an attribute, according to an exemplary embodiment.

According to an embodiment, FIG. 7 may show a flowchart for explaining step 610 of FIG. 6 in more detail.

First, in step 710, the processor may compare the pixel value of each pixel with a threshold value. For example, the processor may determine whether a pixel value of each pixel of the output image generated in step 220 of FIG. 2 is greater than a threshold value. The threshold value may be set as a value for classifying pixels. For example, when dividing a mask image, a pixel value corresponding to a background in the mask image is 0 and a pixel value corresponding to the mask is 1, and the threshold value may be set to 0.5. For another example, in the case of dividing the foreground image, the pixel value corresponding to the background in the foreground image may be 0 and the maximum value of the pixel may be 255, and the threshold value may be set to 127. However, the minimum value, the maximum value, and the threshold value of the pixel are not limited as described above, and may be changed according to design.

And in step 720, the processor may determine the property of each pixel based on the comparison result. The processor may determine that a pixel greater than the threshold has a foreground attribute or a mask attribute, and may determine that a pixel below the threshold has a background attribute. However, the present invention is not limited thereto, and when the value corresponding to the background attribute is greater than the value corresponding to the foreground attribute according to design, the processor may determine that a pixel greater than the threshold has the background attribute.

For example, in the segmentation of the mask image, the closer the pixel value of the output image generated in step 220 of FIG. 2 to 1 is, the higher the probability that the pixel is a mask, and the closer it is to 0, the higher the probability that the pixel is a background. This can be high. Accordingly, in segmenting the mask image, the processor may determine a property of a pixel having a pixel value greater than 0.5, which is a threshold value, as a mask property in the output image, and may determine a property of a pixel having a pixel value less than or equal to 0.5, which is a threshold value, as a background property. For another example, even in the division of the foreground image, the closer the pixel value is to 0, the higher the probability that the corresponding pixel is the background, and the closer the pixel value is 255, the higher the probability of the foreground image. It is possible to determine whether the property of each pixel is foreground or background. However, the present invention is not limited thereto, and the processor may determine whether the property of the pixel is an object or a part of the object.

8 and 9 are flowcharts illustrating a method of extracting an object image using classified pixels according to an exemplary embodiment.

According to one embodiment, FIG. 8 is a flowchart illustrating an example of a method of performing step 620 of FIG. 6 . The image model stored in the memory of the apparatus for dividing an object from an image for performing the method of FIG. 8 may be a model trained to output a reference mask image from a reference training image. The reference mask image may indicate a mask image set to be output from the reference training image.

First, in operation 810 , the processor may generate a mask image by binarizing the output image based on the determined property of the pixel. For example, the processor sets the pixel value of the pixel determined as the mask attribute to 1 in step 610 of FIG. 6 and sets the pixel value of the pixel determined as the background attribute to 0, so that each pixel has a binary value ) may be generated. However, the present invention is not limited thereto, and the processor may set the pixel value corresponding to the background to 1 and the pixel value corresponding to the mask to 1 . Also, the binary values are not limited to 0 and 1, and real numbers of two different values may be used. Hereinafter, a value of 1 for the mask attribute and 0 for the background attribute will be described as a reference.

In operation 820 , the processor may generate a foreground image based on the mask image and the input image generated in operation 810 . For example, the processor may generate a foreground image by multiplying a pixel value of a pixel of the mask image by a pixel value corresponding to a corresponding pixel in the input image. Since the mask image has a pixel value of 1 for the mask-in portion, when the pixel value of each pixel of the mask image is multiplied by the input image, the non-mask portion of the input image is removed and the pixels of the mask-in portion Only values can be maintained.

According to an embodiment, when dividing the mask image, the processor may not perform the above-described operation 820 . The above-described operation 820 may be performed in the case of segmenting the foreground image.

According to another embodiment, FIG. 9 is a flowchart illustrating another example of a method of performing step 620 of FIG. 6 . The image model stored in the memory of the apparatus for segmenting an object from an image for performing the method of FIG. 9 may be a model trained to output a reference foreground image from a reference training image. The reference foreground image may indicate a foreground image set to be output from the reference training image.

In operation 910 , the processor may generate a foreground image from the output image based on the determined attribute of the pixel. For example, the processor may generate a foreground image by maintaining the pixel value of the portion corresponding to the foreground in the output image and initializing the pixel value of the non-foreground portion (eg, changing the pixel value to 0). have.

10 is a flowchart illustrating a method of extracting an object image using pixels of an output image according to an embodiment.

According to an embodiment, FIG. 10 shows an example of a process 810 of generating a mask image and an example of processes 910 and 820 of generating a foreground image in step 230 of FIG. 2 for convenience of explanation. It is a flowchart shown in In FIG. 10 , only one of steps 810 , 910 , and 820 may be performed according to an embodiment. However, the present invention is not limited thereto, and the above-described steps 810 , 910 , and 820 may be selectively performed according to design.

First, from step 220 , the processor may generate the output image 1010 as described above. The i-th pixel of the output image 1010 may have a pixel value _{of p i .}

And in step 1020, the processor may extract a pixel having a larger _{p i than the threshold τ.} Here, the detailed description of the threshold τ as described above with reference to FIG. 7 will be omitted. For example, the processor may _{label a pixel with p i} greater than the threshold τ as having a foreground attribute or a mask attribute.

Subsequently, in operation 810 , the processor may generate a mask image 1030 by combining pixels determined as a mask property as described above with reference to FIG. 8 .

In operation 910 , the processor may generate a foreground image 1040 by collecting pixels determined as a foreground attribute as described above with reference to FIG. 9 .

Subsequently, in operation 820 , the processor generates a mask image 1030 by collecting pixels determined as mask properties as described above with reference to FIG. 8 , and a foreground image 1050 using the mask image 1030 and the input image 1001 . ) can be created.

11 is a block diagram illustrating a configuration of an apparatus for segmenting an object from an image according to an embodiment.

The apparatus 1100 for segmenting an object from an image includes a processor 1110 and a memory 1120 .

The processor 1110 may receive an input image including an object, generate an output image from the input image using an image model, and extract an object image from the output image. A detailed operation of the processor 1110 is omitted since it has been described above with reference to FIGS. 1 to 10 .

The memory 1120 may store an image model trained to output a reference object image from the reference training image. Also, the memory 1120 may temporarily or permanently store input, intermediate results, and final results of image processing, such as an input image, an output image, and an object image.

Also, the apparatus 1100 for segmenting an object may further include a camera (not shown). A camera (not shown) may acquire an input image by photographing the outside of the device 1100 . The apparatus 1100 for dividing an object may further include a communication unit (not shown). The communication unit (not shown) may receive an input image from the outside by wire or wirelessly.

The device 1100 according to an embodiment may determine in units of images instead of in units of pixels in order to separate an object from an image using an image model (eg, a neural network). For example, the device 1100 does not determine whether a patch corresponding to each pixel is a foreground or a background, but divides the object by collectively determining the properties of each pixel with respect to the entire input image. , it takes less time for segmentation, so the speed is fast and the accuracy can be high. The device 1100 may be implemented as a mobile device such as a smart phone or a stationary device such as a PC, or may be implemented in the form of a chip and mounted on a mobile phone or TV.

12 is a diagram illustrating a flowchart of a method of learning an image model used to segment an object from an image according to an embodiment.

First, the model learning unit of the apparatus for learning an image model in operation 1210 may receive training data including a reference training image and a reference object image. The reference training image is an image used as an input in training, and the reference object image may indicate an image preset to be output with respect to a specific reference training image. The training data may include a training pair including a reference training image and a reference object image mapped to the reference training image.

And in step 1220, the model learning unit performs at least one processing of rotation, resizing, shift, flip, and noise adding on the reference training image. Training data may be augmented. The model learning unit may augment the reference training image mapped to the same reference object image through processing such as rotating, resizing, moving, inverting, and adding noise to the reference training image with respect to a pair of a reference training image and a reference object image. can

The rotation processing may refer to image processing for rotating the reference training image by a predetermined angle. For example, the model learning unit may rotate the reference training image at a randomly selected angle between ±8 degrees. Resizing may refer to image processing to increase or decrease the size of the reference training image. For example, the model learner may adjust the size of the reference training image at a ratio randomly selected between 0.9 times and 1.1 times. The movement may represent image processing for cropping the reference training image. For example, the model learner may crop a random position of a random size within the reference training image. Inversion may refer to image processing that flips the reference training image upside down or left and right. For example, the model learning unit may invert the movement-processed reference training image with a probability of 50%. The noise addition may represent image processing in which Gaussian noise is added to the reference training image. For example, the model learner may add Gaussian noise having an average value of 0 and a deviation of 0.9 to each pixel of the reference training image.

Subsequently, in operation 1230 , the model learner may train the image model based on the augmented training data. A process in which the model learning unit trains the image model will be described in detail with reference to FIG. 13 below.

13 is a diagram illustrating a configuration of an apparatus for learning an image model used to segment an object from an image according to an embodiment.

The apparatus 1300 for learning an image model includes a model learning unit 1310 and a training data storage 1320 . The model learning unit 1310 may include at least one processor and may train an image model. For example, the model learning unit 1310 may receive the reference training image 1301 and the reference object image 1309 from the reference training data storage 1320, and the received reference training image and the reference object image are paired ( pair) can be configured. The training data storage 1320 may include at least one memory, and may store training data 1325 used for learning an image model. The training data 1325 may include at least one training pair in which the reference training image 1301 and the reference object image 1309 are mapped to each other. A specific learning process will be described in detail below. Hereinafter, a case in which an image model includes parameters of a neural network will be described as an example.

According to an embodiment, the apparatus 1300 for learning an image model may perform a method of learning division of an object from an image, for example, from an input image (1401 of FIG. 14 ) including an object to an object. Using an image model (eg, the neural network 1311 ) that generates a corresponding output image ( 1405 of FIG. 14 ), the processor ( 1110 of FIG. 14 ) is configured to generate the reference object image 1309 from the reference training image 1301 . ), a parameter of the image model (eg, a parameter of the neural network 1311) may be trained. For example, the apparatus 1300 for learning an image model may train the neural network 1311 through supervised learning. Supervised learning refers to inputting a reference training image 1301 and a reference object image 1309 corresponding thereto to the neural network 1311, and connecting lines such that a reference object image 1309 corresponding to the reference training image 1301 is output. How to update connection weights. For example, the apparatus 1300 for learning an image model may update a connection weight between artificial neurons through a delta rule and error backpropagation learning.

Error backpropagation learning, after estimating an error by forward computation for a given reference training image 1301, starts from the output layer and propagates the estimated error by moving backward in the direction of the hidden layer and the input layer, , is a method of updating the connection weights in a way that reduces errors. Although the processing of the neural network 1311 proceeds in the direction of the input layer hidden layer output layer, the update direction of the connection weights in error backpropagation learning may proceed in the direction of the output layer hidden layer input layer. For example, stochastic gradient descent may be used as error backpropagation learning. The initial connection weight in each layer may be determined by a Gaussian distribution having an average value of 0 and a standard deviation of 0.01. Also, the bias of the convolutional layers and the fully connected layers may be initialized to zero. The learning rate may start at 0.001 and decrease to 0.0001.

The apparatus 1300 for learning an image model defines an objective function for measuring how close to optimal the currently set connection weights are, continues to change the connection weights based on the result of the objective function, and repeats learning. can be done with For example, the objective function may be an error function for calculating an error between an output value actually output by the neural network 1311 based on the reference training image 1301 and an expected value desired to be output. The apparatus 1300 for learning the image model may update the connection weights in a direction to reduce the value of the error function. The error function is a squared L2 norm, and the error L _i of the i-th pixel of the output image may be expressed as in Equation 1 below.

In Equation 1, m _i may represent the binary value of the i-th pixel of the reference object image 1309 mapped to the reference training image 1301 . p _i may represent the pixel value of the i-th pixel of the output image generated with respect to the reference training image 1301 of the neural network 1311 , and may be expressed as Equation 2 below.

In Equation 2 above, f(x _i ) may represent a value in which the reference training image 1301 is projected into a feature space through one or more convolutional filtering, and g( ) is fully connected. A function for deriving the final result of the neural network 1311 processed through the layers may be represented.

According to an embodiment, the image model may be configured such that the resolution of the output image or the object image generated from the input image is the same as the resolution of the input image. In addition, the image model may include a neural network 1311 , and an activation function of the neural network 1311 may include at least one nonlinear function (eg, a sigmoid neuron function). . Furthermore, the image model may include at least one of rotation, resizing, shift, flip, and noise adding with respect to the reference training image 1301. It may be learned based on the performed image.

14 is a diagram illustrating an object image generated from an input image by using the image model learned in FIG. 13 according to an exemplary embodiment.

According to an embodiment, the image model 1121 stored in the memory 1120 of the apparatus 1100 for segmenting an object from an image may be the one learned in FIG. 13 described above.

The processor 1110 may generate an output image 1405 from the input image 1401 by using the image model 1121 learned in FIG. 13 . A pixel value of each pixel of the output image 1405 may indicate, for example, a probability that the corresponding pixel corresponds to a mask. For example, as shown in the output image 1405 of FIG. 14 , even for a portion that is not actually an object, it may appear that an arbitrary pixel has a probability of corresponding to the mask. As described above with reference to FIG. 10 , a pixel having a low probability of corresponding to the mask may be removed through comparison with a threshold value.

15 is a flowchart illustrating a method of segmenting an image according to another exemplary embodiment.

First, the processor of the apparatus for segmenting an object from an image in operation 1510 may receive an input image including the object. The received input image may be as described above with reference to FIG. 1 .

In operation 1520 , the processor may generate an intermediate image corresponding to the object from the input image by using a first image model. The first image model may be configured similarly to the image model described above with reference to FIG. 2 , and the first image model may be used to primarily segment an object image from an input image. For example, the first image model may be trained to output a reference object image from a reference training image. Learning of the first image model will be described in detail with reference to FIGS. 17 and 18 below. The intermediate image may represent an intermediate result generated from the input image by the processor using the first image model.

Subsequently, in operation 1530 , the processor may generate an output image corresponding to the object from the intermediate image using the second image model. The second image model may be configured similarly to the image model described above with reference to FIG. 2 , and the second image model may be used to secondarily segment an object image from the above-described intermediate image. For example, the second image model may be trained to output a reference object image from a reference intermediate image that is a result of applying the first image model to the reference training image. Learning of the second image model will be described in detail with reference to FIGS. 17 and 18 below. The output image may represent a final result generated from the intermediate image by using the second image model by the processor.

As described above, the apparatus for segmenting an object from an image according to another embodiment generates an intermediate image as a result of primarily roughing an input image using a first image model, and secondarily, an intermediate image An output image may be generated as a fine result by using the second image model from

And in step 1540, the processor may extract an object image from the output image. The process of extracting the object from the output image is the same as described above with reference to FIGS. 1 to 10 .

According to an embodiment, the apparatus 1100 described above with reference to FIG. 11 may perform the method of FIG. 15 . For example, the processor 1110 of FIG. 11 may perform the operations of steps 1510 to 1540 described above, and the memory 1120 of FIG. 11 stores the above-described first image model and second image model. can be saved

16 is a diagram illustrating an example of generating an output image from an input image by using an image model according to another embodiment.

16 , a case in which the first image model and the second image model include a connection weight as a parameter of a neural network will be described as an example. The first neural network corresponding to the first image model and the second neural network corresponding to the second image model shown in FIG. 16 may have been trained.

According to an embodiment, the first neural network and the second neural network shown in FIG. 16 may include an input layer, an output layer, and six hidden layers, respectively. For example, the input layer of the first neural network may receive the input image 1601 . The first to eighth layers 1611 , 1612 , 1613 , 1614 , 1615 , 1616 , 1617 , and 1618 of the first neural network may include layers having the same structure as in FIG. 4 . However, the first neural network shown in FIG. 16 may be trained to output an object image from the input image 1601 . Here, the output layer of the first neural network may output an intermediate image 1605 .

Also, the input layer of the second neural network shown in FIG. 16 may receive the above-described intermediate image 1605 . The first to eighth layers 1621 , 1622 , 1623 , 1624 , 1625 , 1626 , 1627 , and 1628 of the second neural network may include layers having the same structure as in FIG. 4 . However, the second neural network shown in FIG. 16 may be trained to output an object image from the intermediate image 1605 . Here, the output layer of the second neural network may output an output image 1609 corresponding to the object.

However, although the structure of the first image model and the second image model is illustrated as the same in FIG. 16 , the present invention is not limited thereto, and the first image model and the second image model may be configured to have different structures. In addition, although the first image model and the second image model have been described as models of a neural network type in FIG. 16 , the present invention is not limited thereto, and the first image model and the second image model may be composed of different types of image models. . In addition, the first image model and the second image model may have different parameters depending on the learning result, but are not limited thereto, and may have the same parameters.

17 is a diagram illustrating a flowchart of a method of learning an image model used to segment an object from an image according to another embodiment.

First, the model learning unit of the apparatus for learning an image model in operation 1710 may receive a reference training image and a reference object image. The reference training image is an image used as an input in training and may be used for learning the first image model. The reference object image may indicate an image preset to be output with respect to a specific reference training image. The same reference object image may be used in learning the first image model and the second image model. The training data may include a training pair including a reference training image and a reference object image mapped to the reference training image.

And in step 1720, the apparatus for learning the image model uses the first image model to generate an intermediate image corresponding to the object from the input image including the object, so that the processor divides the reference object image from the reference training image, A parameter of the first image model may be learned. For example, learning of the first image model may be performed through a process similar to that described above with reference to FIG. 13 .

Subsequently, in operation 1730 , the apparatus for learning the image model may generate a reference intermediate image from the reference training image by using the first image model. The reference intermediate image is an image generated to train the second image model and may be mapped to the reference object image.

And in step 1740, the apparatus for learning the image model uses a second image model that generates an output image corresponding to the object from the intermediate image, so that the processor divides the reference object image from the reference intermediate image, the second image model parameters can be learned. For example, learning of the second image model may be performed through a process similar to that described above with reference to FIG. 13 , except that the second image model divides the reference object image from the reference intermediate image generated in the above-described step 1730 . can be learned to do.

According to one embodiment, the apparatus for learning the image model completes the learning of the first image model in step 1720, and thereafter, it is applied to the reference intermediate image and the reference object image generated using the first image model on which the learning is completed. Based on this, the second image model may be trained in operation 1740 . However, the present invention is not limited thereto, and the device for learning the image model generates a reference intermediate image by using the first image model being trained in step 1730, even if the first image model is being learned in step 1720, In operation 1740, the second image model may be trained using the generated reference intermediate image. As described above, the learning of the first image model for primary object segmentation and the second image model for secondary object segmentation is not limited to being separately performed, but may be performed simultaneously.

The apparatus 1300 shown in FIG. 13 may perform the learning method shown in FIG. 17 .

18 is a diagram illustrating a process of learning an image model used to segment an object from an image according to another embodiment.

The first to eighth layers 1811 , 1812 , 1813 , 1814 , 1815 , 1816 , 1817 , and 1818 of the first neural network shown in FIG. 18 are the first to eighth layers of the first neural network of FIG. 16 . It can be assumed that the layer has the same structure as (1611, 1612, 1613, 1614, 1615, 1616, 1617, 1618). In addition, the first to eighth layers 1821, 1822, 1823, 1824, 1825, 1826, 1827, and 1828 of the second neural network shown in FIG. 18 are the first to second layers of the second neural network shown in FIG. 16 . It may be assumed that the 8 layers 1621 , 1622 , 1623 , 1624 , 1625 , 1626 , 1627 , and 1628 have the same structure as the layer.

As shown in FIG. 18 , the apparatus for learning the image model is configured to minimize the error between the reference intermediate image 1804 output from the reference input image 1801 and the reference object image 1809 , the first image model (eg, For example, the first neural network) may be trained. However, the result by the first image model may be rough. In addition, the apparatus for learning the image model generates a second image model (eg, a second neural network) such that an error between the output image 1805 output from the reference intermediate image 1804 and the reference object image 1809 is minimized. can learn The output image 1805 that is a result of the second image model may be finer than a result of the first image model.

According to an embodiment, an apparatus for segmenting an object from an image may more accurately segment an object image from an input image by using the first image model and the second image model learned in two steps.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (22)

receiving an input image including an object and a background portion;
generating an output image corresponding to the object from the input image by using an image model; and
extracting the object image from the output image such that the object is included in the object image and the background part is excluded
including,
The step of extracting the object image,
determining that a pixel having a pixel value greater than a threshold value among pixels of the output image has a mask property and determining that a pixel having a pixel value less than or equal to the threshold value has a background property;
By binarizing the output image by setting a pixel value of a pixel corresponding to a pixel determined by the mask property in the output image to 1 and setting a pixel value of a pixel corresponding to a pixel determined by the background property in the output image to 0 , generating a mask image in which each pixel has a binary value;
generating a foreground image by multiplying a pixel value of a pixel of the mask image and a pixel value corresponding to a corresponding pixel in the input image;
includes,
The video model is
a neural network, wherein a pixel value of each output image pixel among a plurality of output image pixels included in the output image is included in the input image so that a pixel corresponding to the corresponding output image pixel is included in the object set to generate the output image to indicate the probability that it will not be included in the part,
How to segment an object from an image. delete delete delete delete delete According to claim 1,
The video model is
and a resolution of the object image generated from the input image is configured to be the same as a resolution of the input image,
How to segment an object from an image. According to claim 1,
The activation function of the neural network comprises at least one non-linear function,
How to segment an object from an image. A computer program stored on a medium in combination with hardware to execute the method of any one of claims 1, 7, and 8. a memory for storing the image model; and
Receive an input image including an object and a background part, generate an output image corresponding to the object from the input image using the image model, and output the object so that the object is included in the object image and the background part is excluded A processor for extracting the object image from the image
including,
The processor is
It is determined that a pixel having a pixel value greater than a threshold value among pixels of the output image has a mask property, and a pixel having a pixel value less than the threshold value is determined as having a background property, and in the output image, the mask By binarizing the output image by setting a pixel value of a pixel corresponding to a pixel determined as a property to 1 and setting a pixel value of a pixel corresponding to a pixel determined as a background property in the output image to 0, each pixel is a binary value generating a mask image having a (binary value), multiplying a pixel value of a pixel of the mask image by a pixel value corresponding to a corresponding pixel in the input image to generate a foreground image;
The video model is
a neural network, wherein a pixel value of each output image pixel among a plurality of output image pixels included in the output image is included in the input image so that a pixel corresponding to the corresponding output image pixel is included in the object and the background set to generate the output image to indicate the probability that it will not be included in the part,
A device for segmenting an object from an image. 11. The method of claim 10,
The processor is
Classifying each pixel of the output image according to a property, and extracting an object image using the classified pixel,
A device for segmenting an object from an image. delete delete delete delete 11. The method of claim 10,
The video model is
and a resolution of the object image generated from the input image is configured to be the same as a resolution of the input image,
A device for segmenting an object from an image. 11. The method of claim 10,
The video model is
The activation function of the neural network comprises at least one non-linear function,
A device for segmenting an object from an image. delete delete delete receiving an input image including an object and a background part;
generating an intermediate image corresponding to the object from the input image by using a first image model;
generating an output image corresponding to the object from the intermediate image using a second image model; and
extracting the object image from the output image such that the object is included in the object image and the background part is excluded
including,
The step of extracting the object image,
determining that a pixel having a pixel value greater than a threshold value among pixels of the output image has a mask property and determining that a pixel having a pixel value less than or equal to the threshold value has a background property;
By binarizing the output image by setting a pixel value of a pixel corresponding to a pixel determined by the mask property in the output image to 1 and setting a pixel value of a pixel corresponding to a pixel determined by the background property in the output image to 0 , generating a mask image in which each pixel has a binary value;
generating a foreground image by multiplying a pixel value of a pixel of the mask image and a pixel value corresponding to a corresponding pixel in the input image;
includes,
The first image model is
a neural network, wherein a pixel value of each intermediate image pixel among a plurality of intermediate image pixels included in the intermediate image is included in the input image so that a pixel corresponding to the corresponding intermediate image pixel is included in the object and the background set to generate the intermediate image to indicate the probability that it will not be included in the part,
How to segment an object from an image. delete

Images