KR102458243B1 - Method and apparatus for analysis of facial image - Google Patents

Abstract

A face image analysis method and apparatus are disclosed. The face image analysis apparatus inputs a face image to a residual network including a plurality of sequentially combined residual blocks arranged in an input to output direction, processes the face image using the residual network, and performs the residual inverse An analysis map is obtained from the output of an Nth residual block among a plurality of residual blocks by using a residual deconvolution network. Here, the residual network passes the output of the Nth residual block to the residual deconvolution network, where N is a natural number and is less than the number of total residual blocks of the residual network, and the residual deconvolution network is sequentially combined with a plurality of residual inverses. a convolution block, wherein the plurality of residual deconvolution blocks respectively correspond to first to Nth residual blocks among the plurality of residual blocks.

Description

Facial image analysis method and apparatus {METHOD AND APPARATUS FOR ANALYSIS OF FACIAL IMAGE}

The following description relates to an apparatus and method for analyzing a face image, and to a technique for analyzing a face image using a convolution or deconvolution operation of a neural network.

Computer vision technology is a technology that automates operations such as identification, tracking, and surveying of a target by replacing the human eye with a camera and computer, and a signal processed by image processing is suitable for human eye or computer input. Analyzing meaning using deep learning refers to analyzing each pixel point in an image when given an image. Deep neural networks are able to abstract the low-dimensional features of the input image nicely. However, it is difficult to extract features by the neural network in the stage of analyzing the meaning, lower the resolution of features using a pooling layer, and convert a low-dimensional and high-efficiency feature map into a high-dimensional pixel-level analysis result. to be.

The above-mentioned background art is possessed or acquired by the inventor in the process of derivation of the present invention, and it cannot be said that it is necessarily known technology disclosed to the general public prior to the filing of the present invention.

A face image analysis method according to an embodiment includes: inputting a face image into a residual network including a plurality of sequentially combined residual blocks arranged in an input to an output direction; processing an image and obtaining an analysis map from an output of an Nth residual block among the plurality of residual blocks by using a residual deconvolution network, wherein the residual network comprises the Nth residual block pass the output of the block to the residual deconvolution network, where N is a natural number and less than the total number of residual blocks in the residual network, the residual deconvolution network comprising a plurality of sequentially combined residual deconvolution blocks and the plurality of residual deconvolution blocks may respectively correspond to first to N-th residual blocks among the plurality of residual blocks.

The face image analysis method includes the steps of: obtaining prior information of a face image using a prior information acquisition network; and obtaining an analysis result by combining the prior information and an output of the residual deconvolution network. may include more.

The step of obtaining the prior information of the face image may include comparing a face included in the face image with faces included in a plurality of face image analysis training data, and a face including a face most similar to the face included in the face image. The method may include selecting training data for image analysis and obtaining an average value of calibration information obtained from the most similar face as prior information of the face image.

The obtaining of the analysis result includes obtaining a combination map by combining the analysis map output from the residual deconvolution network and prior information, and performing convolution on the combination map using a convolution kernel. A contribution plot of the prior information may be obtained, and the analysis result may be obtained by adding the contribution plot and the analysis map to an element level.

The facial image analysis method may further include improving the analysis result by using a dense condition random field method.

In the step of improving the analysis result, the analysis result may be improved by setting the analysis result output by the prior information acquisition network as a unary term of the dense condition random field.

An apparatus for analyzing a face image according to an embodiment includes a processor for inputting a face image into a residual network including a plurality of sequentially combined residual blocks arranged in an input to an output direction, a residual network for processing the face image, and the plurality of a residual deconvolution network for obtaining an analysis map from an output of an Nth residual block among residual blocks, wherein the residual network passes the output of the Nth residual block to the residual deconvolution network, where N is a natural number smaller than the total number of residual blocks of the residual network, wherein the residual deconvolution network includes a plurality of sequentially combined residual deconvolution blocks, wherein the plurality of residual deconvolution blocks is a first of the plurality of residual blocks. Each of the residual blocks from th to N th may correspond to each other.

A training method of an apparatus for analyzing a face image according to an embodiment includes a pre-training step of training the residual network by adjusting a weight parameter of the residual network using training data for face identification and training data for analyzing a face image. a joint training step of training the residual network and the residual deconvolution network by adjusting a weight parameter of the residual deconvolution network using the a residual network and the residual deconvolution network, wherein the residual network comprises a plurality of sequentially combined residual blocks arranged in an input to output direction, processing a face image, and outputting the Nth residual block to the residual deconvolution network, where N is a natural number and less than the total number of residual blocks of the residual network, and the residual deconvolution network obtains an analysis map from the output of an Nth residual block among the plurality of residual blocks. a plurality of residual deconvolution blocks obtained and sequentially combined, wherein the plurality of residual deconvolution blocks may respectively correspond to first to Nth residual blocks among the plurality of residual blocks.

The pre-training step includes inputting the training data for face identification into the residual network, performing a face identification operation, and performing average pooling on the output of the last residual block of the residual network, and adding to the residual network. The residual network may be pre-trained by performing a full join operation on the data and adjusting a weight parameter of the residual network.

The joint training step initializes the weight parameter of the residual network with the weight parameter obtained from the pre-training step, randomly initializes the weight parameter of the residual deconvolution network, and converts the output of the Nth residual block to the residual inverse The weight parameter of the residual deconvolution network is input to a convolution network, the training data for facial image analysis is input to the residual network, and a face image analysis operation is performed using the residual network and the residual deconvolution network. and a weight parameter of the residual network can be adjusted.

The step of training the prior information acquisition network includes a first training step of training the prior information acquisition network and a second training step of training the residual network, the residual deconvolution network, and the prior information acquisition network, In the first training stage, all parameters other than the weight parameters in the prior information acquisition network are fixed, the weight parameters in the dictionary information acquisition network are adjusted, and in the second training stage, the adjusted weights in the dictionary information acquisition network The prior information acquisition network is initialized using the parameters, all parameters other than the weight parameters in the prior information acquisition network are variable, and face images are applied to the residual network, the residual deconvolutional network and the prior information acquisition network as a whole. As the facial image analysis operation is performed using the training data for analysis, the residual network and the residual deconvolution network may be adjusted, and each weight parameter in the prior information acquisition network may be additionally adjusted.

1 is a diagram illustrating an overall configuration of an apparatus for analyzing a face image according to an exemplary embodiment.
2 is a flowchart illustrating an operation of performing pre-training on a residual network according to an embodiment.
3 is a flowchart illustrating an operation of performing joint training on a residual network and a residual deconvolution network according to an embodiment.
4 is a flowchart illustrating a face image analysis method according to an exemplary embodiment.
5 is a diagram illustrating an overall configuration of an apparatus for analyzing a face image according to another exemplary embodiment.
6 is a flowchart illustrating an operation of training a face image analyzing apparatus according to another exemplary embodiment.
7 is a flowchart illustrating a face image analysis method according to another exemplary embodiment.
8A is a diagram illustrating a configuration of a residual network according to an embodiment.
8B is a diagram illustrating a residual block in a residual network according to an embodiment.
9A is a diagram illustrating a configuration of a residual deconvolution network according to an embodiment.
9B is a diagram illustrating a residual deconvolution block in a residual deconvolution network according to an embodiment.
10 is a diagram illustrating a configuration in which a result of a residual deconvolution network and a result of a prior information acquisition network are combined according to another embodiment.
11A is an example in which a result obtained by the apparatus for analyzing a face image according to an exemplary embodiment is compared with a result of VGG-deconvolution.
11B is an example of comparing a result obtained by the apparatus for analyzing a face image according to an exemplary embodiment and a result of VGG-deconvolution.
11C is an example in which a result obtained by the apparatus for analyzing a face image according to an exemplary embodiment is compared with a result of VGG-deconvolution.
11D is an example of comparing a result obtained by the apparatus for analyzing a face image according to an embodiment with a result of VGG-deconvolution.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various components, these terms should only be analyzed for the purpose of distinguishing one component from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be analyzed to have meanings consistent with the meanings in the context of the related art, and should not be analyzed in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

On the other hand, when a certain embodiment can be implemented differently, a function or operation specified in a specific block may be performed differently from the flowchart. For example, two consecutive blocks may be performed substantially simultaneously, or the order of the blocks may be reversed depending on a related function or operation.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted.

1 is a diagram illustrating an overall configuration of an apparatus for analyzing a face image according to an exemplary embodiment.

According to an embodiment, the facial image analysis apparatus 100 includes a processor 110 , a residual network 120 , and a residual deconvolution network 130 .

The processor 110 may input an input image to be analyzed to the residual network 120 . The input may include various types of images including the face image 101 . For example, the input may include a face image, a virtual face generated by computer graphics, or an animal face.

The residual represents the difference between the input and the predicted or fitted value. The output of the residual network may be obtained by adding outputs and inputs of a plurality of convolution cascades and activating a correction linear unit (ReLU). Here, the phase of the output and the input of the convolution layer may be the same.

The residual network 120 may process a face image. The residual network 120 may include a plurality of sequentially combined residual blocks arranged in an input-to-output direction. The residual network passes the output of the Nth residual block to the residual deconvolution network, where N is a natural number and may be less than the total number of residual blocks in the residual network.

The residual deconvolution network 130 may obtain the analysis map 103 from an output of an Nth residual block among a plurality of residual blocks. The residual deconvolution network 130 includes a plurality of sequentially combined residual deconvolution blocks, and the plurality of residual deconvolution blocks may respectively correspond to first to Nth residual blocks among the plurality of residual blocks. have.

A good analysis result can be obtained when the predetermined Nth residual block is the last second residual block among the sequentially combined residual blocks arranged in the input to output direction. Also, the predetermined N-th residual block may be the last third residual block among the sequentially combined residual blocks arranged in the input to output direction.

In the facial image analysis apparatus 100 according to an embodiment, the residual network 120 may effectively extract facial features by performing pre-training as a face identification operation. By performing joint training as a facial image analysis operation on the residual deconvolutional network 130 of the facial image analysis apparatus 100 , a high-resolution feature map may be generated and an accurate analysis result may be derived. The facial image analysis apparatus 100 may finely adjust the entire network by using the prior information. The facial image analysis apparatus 100 may further improve the analysis result by using the dense condition random field method.

2 is a flowchart illustrating an operation of performing pre-training on a residual network according to an embodiment.

According to an embodiment, the residual network 120 may be pre-trained through face identification operations during the pre-training phase. In the pre-training stage, the residual network may be trained by adjusting a weight parameter of the residual network using training data for face identification.

In step 210 , random initialization is performed on the weight parameters of the convolution kernel in the residual network 120 , and training data for face identification stored in the training database may be input to the residual network 120 . . The training data for face identification may include a plurality of face images.

In operation 220 , a face identification operation may be performed using training data for face identification. Pre-training may be performed on the residual network through the face identification operation. The convolution block and the residual block may perform a face identification operation on the input training data for face identification. Average pooling may be performed on the output of the last residual block of the residual network 120 , and a full join operation may be performed on the residual network 120 . By performing the pull-join operation on the residual network 120 , the weight of the convolution kernel of the residual network 120 may be adjusted and the softmax function may be minimized. Here, the softmax function is a kind of loss function.

The weight of the convolution kernel can be adjusted by pulling the number of persons with the same identity in the training database by the pull-join operation. When the training data for each face identification is input, the value of the sptmax function is minimized, so that the residual network 120 can be trained to accurately identify the identity from the face of each training data. For example, when data is trained by a face identification operation, if there are a plurality of persons with identities in the training database, a number corresponding to the number of persons may be output by the pull-join operation.

In step 230, an optimized weight parameter may be obtained. As such, the residual network may be pre-trained by adjusting the weight parameters of the residual network 120 .

3 is a flowchart illustrating an operation of performing joint training on a residual network and a residual deconvolution network according to an embodiment.

In the joint training step, the last block of the residual network 120 is removed, the residual deconvolution network 130 is cumulatively coupled to the residual network 120 , and the entire network may be adjusted by a facial image analysis operation.

In step 310, the weight parameters in the residual network 120 may be initialized to the weight parameters obtained in the pre-training step.

In operation 320 , training data for facial image analysis may be input to the residual network 120 on which pre-training has been performed.

In step 330 , a feature map may be obtained. The feature map may be an output of the Nth residual block of the residual network 120 . For example, the feature map may be the output of the second and last residual block. Here, the resolution of the feature map may be low.

In step 340 the residual deconvolution network 130 may be initialized. The weight parameters of the residual deconvolution network 130 may be randomly initialized, and the output of the Nth residual block may be input to the residual deconvolution network 130 . The residual deconvolution network 130 may be further processed by a feature map.

In operation 350 , a facial image analysis operation is performed using the training data for facial image analysis, so that the residual network 120 and the residual deconvolution network 130 may be jointly trained. By adjusting the weight parameter of the residual deconvolution network using training data for facial image analysis and further adjusting the weight parameter of the residual network, the residual network and the residual deconvolution network can be jointly trained.

The weight parameter of the convolution kernel in the residual deconvolution network 130 may be optimized, and the weight parameter of the convolution kernel in the residual network 120 may be further optimized. The softmax function can be minimized by adjusting the weight parameter of the residual deconvolution network 130 and the weight parameter of the residual network 120 .

The training data used in the pre-training step and the combined training step may be face images that have not been pre-processed, but the training data with random mirroring or random cut-out may be used. Random mirroring may mean left-right reversal or vertical reversal, and random cut-out may mean a pre-processing operation of editing a face image to an arbitrary size. The amount of training data may be increased by performing a preprocessing step on the training data. The training data may include training data for face identification and training data for face image analysis.

For example, the face image may be edited into an image of any size such as 224*224 size by random cutout or random mirroring may be performed. The input to the convolution block may be a raw image of training data, may be a result of random mirroring, may be a result of random cutout, or may be a result of applying both random mirroring and random cutout. According to the pre-processing step, the overfitting phenomenon that occurs when training is performed multiple times using the same image can be alleviated.

4 is a flowchart illustrating a face image analysis method according to an exemplary embodiment.

In operation 410 , the face image may be input to the residual network on which pre-training has been performed. In step 420, the residual network may process the face image. In step 430 , the residual network may output a feature map as a result of processing the face image. The feature map may be the output of the second and last residual block of the residual network. In step 440, the feature map may be input to the residual deconvolution network. In step 450, the residual deconvolution network may output an analysis map. The analysis map may be referred to as an analysis result.

5 is a diagram illustrating an overall configuration of an apparatus for analyzing a face image according to another exemplary embodiment.

According to another embodiment, the facial image analysis apparatus 100 may include a processor 110 , a residual network 120 , a residual deconvolution network 130 , and a prior information acquisition network 510 .

The face image 101 may be input to the residual network 120 . The processor 110 may obtain an analysis result 503 by combining the output of the prior information acquisition network 510 and the output of the residual deconvolution network 130 . The processor 110 may improve the analysis result 503 by using the dense condition random field method. The processor 110 may obtain a more accurate analysis result by setting the analysis result 503 as a unary term of the cluster condition random field.

6 is a flowchart illustrating an operation of training a face image analyzing apparatus according to another exemplary embodiment.

The residual network 120 of the facial image analysis apparatus 100 may be trained by a pre-training step, and the residual network 120 and the residual deconvolutional network 130 may be trained by a joint training step. The facial image analysis apparatus 100 trains the prior information acquisition network 510 using the residual network 120 and the residual deconvolution network 130 trained by the pre-training step and the joint training step, and the residual network 120 ) and the residual deconvolution network 130 may be additionally trained.

In step 610 the training data may be input to the residual network 120 . Here, the training data may include training data for face image analysis. In step 620, the residual network 120 may process the training data. In step 630 , the residual network 120 may output a feature map as a processing result. Here, the feature map may be an output of the second and last residual block of the residual network 120 . In step 640 , the feature map may be input to the residual deconvolution network 130 . In step 650 , the residual deconvolution network 130 may output an analysis map.

In step 660 , the prior information acquisition network 510 may obtain prior information from the training data. In step 670, the prior information and the analysis map may be combined. In step 680 , a first training step may be performed on the prior information acquisition network 510 . In step 690 , a second training step may be performed for the residual network 120 , the residual deconvolution network 130 , and the prior information acquisition network 510 .

In the first training step, all parameters other than the weight parameters in the dictionary information acquisition network 510 may be fixed, and the weight parameters in the dictionary information acquisition network 510 may be adjusted. The weight parameters optimized by the first training step are obtained and the softmax function can be minimized.

In the second training step, the dictionary information acquisition network 510 is initialized using the adjusted weight parameters, and all parameters other than the weight parameters in the dictionary information acquisition network 510 may be released. All parameters other than the weight parameter in the prior information acquisition network 510 may be adjusted.

In the second training step, a facial image analysis operation is performed using training data for facial image analysis on all of the residual network, the residual deconvolution network, and the prior information acquisition network, so that the residual network and the residual deconvolution network are adjusted and each weight parameter in the prior information acquisition network may be further adjusted.

As training data for training the entire network including the prior information acquisition network 510, a face image that has not undergone a preprocessing step may be used, but training data on which random mirroring is performed or training data on which random cutout is performed may be used. have. The amount of training data may be increased by performing a preprocessing step on the training data. The training data may include training data for face identification and training data for face image analysis.

For example, the face image may be edited into an image of any size such as 224*224 size by random cutout or random mirroring may be performed. The input to the convolution block may be a raw image of training data, may be a result of random mirroring, may be a result of random cutout, or may be a result of applying both random mirroring and random cutout. According to the pre-processing step, the overfitting phenomenon that occurs when training is performed multiple times using the same image can be alleviated.

7 is a flowchart illustrating a face image analysis method according to another exemplary embodiment.

In operation 710 , the face image to be analyzed may be input to the entire network. In operation 720 , the input face image may be processed by the residual network 120 on which pre-training and joint training have been performed. In step 730 , a feature map may be obtained from the residual network 120 as a result of the processing in step 720 . In step 740 , the feature map may be input to the residual deconvolution network 130 . Here, association training may be performed on the residual deconvolution network 130 in advance. An analysis map may be obtained from the residual deconvolution network 130 in step 750 .

In operation 760, the dictionary information may be acquired by the dictionary information acquisition network 510 from the input face image. The dictionary information is information related to an input image. The face image analysis apparatus 100 may use different face images stored in the training database. For example, type information of each face image may be displayed through calibration. For example, each part in the face image may be displayed as a background, skin, hair, or five tubes representing sensory organs.

The face image analysis apparatus 100 may select training data for analyzing a face image including a face most similar to a face included in the face image. The facial image analysis apparatus 100 may select the most similar face or group of faces by comparing the face in the face image to be analyzed with the face in the face image stored in the training database. The face image analysis apparatus 100 may obtain an average value of calibration information obtained from the most similar face as prior information of the face image or may obtain the prior information by calculating an average value of a selected group.

In operation 770 , the facial image analysis apparatus 100 may combine the prior information and the analysis map. As a result of the combination in operation 780 , the facial image analysis apparatus 100 may obtain an analysis result. In operation 790, the analysis result may be processed by a dense condition random field method. As a processing result in operation 800 , the facial image analysis apparatus 100 may obtain an improved analysis result.

8A is a diagram illustrating a configuration of a residual network according to an embodiment.

The residual network 120 may include sequentially combined residual blocks arranged in an input-to-output direction. For example, the residual network 120 may include a convolution block 11 and a residual block 12 through a residual block 16 in an input-to-output direction. The residual network 120 may include a convolutional block positioned before the first residual block. The residual network 120 may transmit a predetermined N-th residual block among the sequentially combined residual blocks to the residual deconvolution network 130 . Here, N is a natural number and is smaller than the number of all residual blocks included in the residual network 120 . For example, the output of the convolution block 15 may be passed to the residual deconvolution network 130 .

For example, the convolution block 11 may include two accumulated convolution layers. The size of the convolution kernel in the convolution layer may be 3*3, and the number of convolution kernels in each convolution layer may be 64. The convolution block 11 may convert the input data into a format suitable for reception of the residual block. The size and number of the above-described convolution kernels are merely examples and are not limited thereto.

For example, when three RGB channels and a face image with a size of 224*224 are input, the face image is displayed as 224*224*3. The convolution operation is performed by 64 3*3 convolution kernels on the face image of 224*224*3 size input to the first convolution layer of the convolution block 11, resulting in 224*224*64 An intermediate image 801 may be generated. A convolution operation may be performed again on the intermediate image 801 by 64 3*3 convolution kernels to generate an intermediate image 802 of 224*224*64.

Each residual block may be connected through max pooling. Maximum pooling is an operation that sets the maximum value within a range of a certain size as a representative of the range. For example, when 2*2 maximum value pooling is performed on the input image, the maximum value in each 2*2 range of the input image is set to represent the 2*2 range, and other values may be omitted. Average pooling is an operation that sets the average value within a range of a certain size as a representative of the range. For example, when 7*7 average pooling is performed on the input image, the average value in each 7*7 range of the input image is set to represent the 7*7 range, and other values may be omitted. .

A maximum pooling process may be performed on an output of a first residual block among the plurality of residual blocks, and a result of the maximum pooling process may be input to a residual block of a next level of the first residual block.

When maximum pooling is performed on the intermediate image 802 , an intermediate image 803 of 112*112*64 may be generated. By maximal pooling, the size of the image can be reduced to 1/2. The intermediate image 803 is input to the residual block 12 , and the residual block 12 increases the dimension of the intermediate image 803 to output an intermediate image 804 of 112*112*128. Through this, the number of channels of the intermediate image 803 may increase from 64 to 128.

Maximum pooling may be performed on the intermediate image 804 and the intermediate image 805 of 56*56*128 may be output. The intermediate image 805 may be converted into an intermediate image 806 whose dimension is increased to 56*56*256 by the block 13 in the future. Maximum pooling is performed on the intermediate image 806 , and an intermediate image 807 of 28*28*256 may be output. The intermediate image 807 may be transformed into an intermediate image 808 whose dimension is increased to 28*28*512 by the residual block 14 . Maximum pooling is performed on the intermediate image 808 , and an intermediate image 809 of 14*14*512 may be output. The intermediate image 809 may be transformed into an intermediate image 810 whose dimension is increased to 14*14*1024 by the residual block 15 . Maximum pooling is performed on the intermediate image 810 and an intermediate image 811 of 7*7*1024 may be output. The intermediate image 811 may be transformed into an intermediate image 812 of 7*7*1024 by the residual block 16 . Average pooling is performed on the intermediate image 812 and an intermediate image 813 of 1*1*1024 may be output.

The number of convolution kernels in each convolutional layer of the convolution block 11 (eg, 64) and the number of channels of the intermediate image as a result of which the dimension is increased by the residual block (eg, 128, 256, 512, 1024), the accuracy of the analysis result may be improved when the weight parameter is set.

8B is a diagram illustrating a residual block in a residual network according to an embodiment.

FIG. 8B exemplarily shows the structure of the residual block 13 of FIG. 8A . Residual block 12 - Residual block 16 of FIG. 8A includes a residual structure. The structures of the residual block 12 to the residual block 16 are similar to each other, but there may be differences in the number of convolution kernels. The number of residual blocks may be 4 or 5.

For example, referring to FIG. 8B , an input image 821 of 56*56*128 (length*width*number of channels) may be input. A convolution operation may be performed on the input image 821 by 128 1*1 convolution kernels. The convolution operation may be performed again on the result of the operation by 128 3*3 convolution kernels. As a result of the operation, a convolution operation may be performed again by 256 convolution kernels having a size of 1*1, and an intermediate image 825 of 56*56*256 may be obtained.

A convolution operation is performed on the input image 821 by 256 convolution kernels having a size of 1*1, and an intermediate image 823 of 56*56*256 may be obtained. The intermediate image 823 and the intermediate image 825 may be summed by an adder.

A convolution operation may be performed on the summed result by 128 1*1 convolution kernels. Convolution operation may be performed again on the corresponding operation result by 128 3*3 convolution kernels. A convolution operation may be performed again on the result of the operation by 256 convolution kernels having a size of 1*1, and an intermediate image 827 of 56*56*256 may be obtained.

A result of adding the intermediate image 823 and the intermediate image 825 and the intermediate image 827 may be summed again by the adder.

A convolution operation may be performed on the summed result by 128 1*1 convolution kernels. Convolution operation may be performed again on the corresponding operation result by 128 3*3 convolution kernels. A convolution operation may be performed again by 256 convolution kernels having a size of 1*1 on the corresponding operation result, and an intermediate image 829 of 56*56*256 may be obtained.

The previous summing result and the intermediate image 829 are summed again by the adder, and an output image 831 of 56*56*256 may be output.

9A is a diagram illustrating a configuration of a residual deconvolution network according to an embodiment.

Referring to FIG. 9A , the trained residual deconvolution network 130 receives an analysis map 913 from an Nth residual block of the residual network 120 , for example, a feature map 901 that is an output of the residual block 15 . ) can be printed.

The residual deconvolution network 130 includes sequentially combined residual deconvolution blocks. The residual deconvolution network 130 may further include a deconvolution block located after the last residual deconvolution block. For example, the residual deconvolution network 130 includes a residual deconvolution block 22 through a residual deconvolution block 25 , and a residual deconvolution block 22 through a residual deconvolution block 25 . ) may respectively correspond to the Nth residual block 15 in the first residual block 12 of the plurality of residual blocks of the residual network 120 .

For example, the residual deconvolution network 130 may include a residual deconvolution block 25 to a residual deconvolution network 22 and one deconvolution block 21 coupled from an input to an output direction. can Similar to the residual network 120 , the residual deconvolution network 130 may include, but is not limited to, three or four residual deconvolution blocks.

A maximum anti-pooling process is performed on the output of the first residual deconvolution block among the plurality of residual deconvolution blocks, and the result of the maximum anti-pooling process is the next of the first residual deconvolution block. It can be input to the residual deconvolution block of the level.

The output of the residual block 15 may be input to the residual deconvolution network 130 after maximal pooling is performed, and may be transformed into an intermediate image 902 of 14*14*1024 by maximal anti-pooling. The intermediate image 902 may be transformed into an intermediate image 903 of 14*14*512 by the residual deconvolution block 25 . The intermediate image 903 may be converted into an intermediate image 904 of 28*28*512 by maximum anti-pooling. The intermediate image 904 may be transformed into an intermediate image 905 of 28*28*256 by the residual deconvolution block 24 . The intermediate image 905 may be converted into the intermediate image 906 of 56*56*256 by maximum anti-pooling. The intermediate image 906 may be transformed into the intermediate image 907 of 56*56*128 by the residual deconvolution block 23 . The intermediate image 907 may be converted into the intermediate image 908 of 112*112*128 by maximum anti-pooling. The intermediate image 908 may be transformed into an intermediate image 909 of 112*112*64 by the residual deconvolution block 22 . The intermediate image 909 may be converted into the intermediate image 910 of 224*224*64 by maximum anti-pooling.

An analysis map 913 may be obtained by performing a 1*1 convolution operation on the intermediate image 910 by the deconvolution block 21 . For example, the deconvolution block 21 may include two deconvolution layers, and each deconvolution layer may include 64 3*3 deconvolution kernels.

9B is a diagram illustrating a residual deconvolution block in a residual deconvolution network according to an embodiment.

9B exemplarily shows the structure of the residual deconvolution block 23 of FIG. 9A. The structures of the residual deconvolution block 22 to the residual deconvolution block 25 are similar to each other, but there may be differences in the number of deconvolution kernels.

Each of the residual deconvolution blocks may include a compactor 910 , a detail learner 920 , and a dimension reducer 930 . The clusterer 910 performs three deconvolution operations, the detail learner 920 performs three deconvolution operations and summation, and the dimension reducer 930 performs four deconvolution operations and summation. can be done

For example, an input image 901 of 56*56*256 (length*width*number of channels) is input to the compactor 910 , and 128 1*1 size deconvolutional kernels in the compactor 910 . may perform a deconvolution operation on the input image 901 . The deconvolution operation is again performed by 128 3*3 size deconvolution kernels on the result of the operation, and the deconvolution operation is again performed on the corresponding operation result by 256 1*1 size deconvolution kernels. This can be done.

The detail learner 920 may include a residual branch 922 and a deconvolution branch 923 . The deconvolution branch 923 includes three different deconvolution operations, and the operation result of the deconvolution branch 923 is summed with the operation result of the clusterer 910 transferred to the residual branch 922 . can be

For example, the operation result of the clusterer 910 is transmitted to the detail learner 920 , and 128 1*1 size deconvolution kernels in the detail learner 920 are calculated for the operation result of the clusterer 910 . Deconvolution operation can be performed. The deconvolution operation is again performed by the corresponding operation result and 128 3*3 size deconvolution kernels, and the deconvolution operation is again performed on the corresponding operation result by 256 1*1 size deconvolution kernels. can be performed. The corresponding operation result may be summed with the operation result of the clusterer 910 transmitted to the residual branch 922 . The summed result may be transmitted to the dimension reducer 930 . The residual branch 922 can more easily optimize the residual deconvolution network 130 by ensuring that the gradient is not dispersed.

The output of the detail learner 920 may be transmitted to the dimension reducer 930 . A deconvolution operation may be performed on the output of the detail learner 920 by 128 1*1 size deconvolution kernels. The convolution operation may be performed again on the result of the operation by 128 3*3 convolution kernels. As a result of the operation, a convolution operation may be performed again by 128 convolution kernels having a size of 1*1, and an intermediate image 931 of 56*56*128 may be obtained.

A convolution operation is performed on the output of the detail learner 920 by 128 1*1 convolution kernels, and an intermediate image 932 of 56*56*128 may be obtained. The intermediate image 931 and the intermediate image 932 are summed by the adder, and an output image of 56*56*128 may be obtained.

10 is a diagram illustrating a configuration in which a result of a residual deconvolution network and a result of a prior information acquisition network are combined according to another embodiment.

Referring to FIG. 10 , a combination map may be obtained by combining the analysis map output from the residual deconvolution network and prior information. By performing convolution on the combination map using a convolution kernel, a contribution plot of prior information may be obtained, and an analysis result may be obtained by adding the contribution plot and the analysis map to an element level.

The analysis map 31 output from the residual deconvolution network 130 and the prior information may be combined. A convolution operation is performed on the dictionary information output from the dictionary information acquisition network 510 by a convolution kernel of 224*224*(3+3), and a contribution plot 34 may be output as a result of the operation. . For example, if the channel is N and the size of the image is W*H, the size of the analysis map 31 and the dictionary information may be W*H*N, and the size of the combination map may be W*H*2N. The contribution plot 34 is a contribution plot for each channel, and the size may be W*H*N. The contribution plot 34 and the analysis map 31 output from the residual deconvolution network 130 are summed to the element level, the softmax function value 1010 is output by the softmax function, and the analysis result can be obtained. .

11A to 11D are examples of comparing a result obtained by the apparatus for analyzing a face image according to an exemplary embodiment and a result of VGG-deconvolution.

11A to 11D are results of using the LFW face database. The LFW face database is an internationally authoritative database. The raw image of FIGS. 11A to 11D means an unprocessed face image, the VGG-deconvolution result is a result of the conventional VGG-deconvolution method, and the result of the facial image analysis apparatus is different from one embodiment. It is a result by the face image analysis apparatus 100 . For example, in FIG. 9C , the result of the VGG-deconvolution method analyzes a part of the hair and the background, but the result of the facial image analysis apparatus accurately distinguishes the hair and the background.

Results by face image analysis device Results of VGG-Deconvolution Method Consequences of Liu's Method pixel accuracy 97.53% 97.06% 95.12% size of the model 103M 960M 127M

Table 1 shows the pixel accuracy of the various methods and the size of the model. Compared with the conventional VGG-deconvolution method or Liu's work, the pixel accuracy by the facial image analysis apparatus 100 reaches 97.53% and the size of the model is only 103M. As such, the facial image analysis apparatus 100 may reduce the size of a model while having high accuracy.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device for analysis by or providing instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Claims (27)

inputting a face image to a residual network including a plurality of sequentially combined residual blocks arranged in an input to an output direction;
processing a face image using the residual network; and
obtaining an analysis map from the output of an Nth residual block among the plurality of residual blocks using a residual deconvolution network,
the residual network passes the output of the Nth residual block to the residual deconvolution network, where N is a natural number and is less than the total number of residual blocks of the residual network;
The residual deconvolution network includes a plurality of sequentially combined residual deconvolution blocks, wherein the plurality of residual deconvolution blocks respectively correspond to first to Nth residual blocks among the plurality of residual blocks.
Face image analysis method. According to claim 1,
obtaining prior information of the face image using a prior information acquisition network; and
obtaining an analysis result by combining the prior information and an output of the residual deconvolution network
Further comprising, a face image analysis method. 3. The method of claim 2,
The step of obtaining the prior information of the face image comprises:
A face included in the face image and faces included in a plurality of training data for analyzing facial images are compared to select training data for face image analysis including a face most similar to the face included in the face image, and the most similar Comprising the step of obtaining an average value of the calibration information obtained from the face as prior information of the face image,
Face image analysis method. 3. The method of claim 2,
The step of obtaining the analysis result is,
A combination map is obtained by combining the analysis map output from the residual deconvolution network and the prior information, and the contribution plot of the prior information is obtained by performing convolution on the combination map using a convolution kernel. to obtain the analysis result by adding the contribution plot and the analysis map to an element level
Face image analysis method. 3. The method of claim 2,
improving the analysis result by using a dense condition random field method;
Further comprising, a face image analysis method. 6. The method of claim 5,
The step of improving the analysis result is,
The analysis result is improved by setting the analysis result output by the prior information acquisition network as a unary term of the random field of the dense condition.
Face image analysis method. The method of claim 1,
The residual network further comprises a convolution block positioned before a first residual block, and the residual deconvolution network further comprises a deconvolution block positioned after the last residual deconvolution block.
Face image analysis method. According to claim 1,
A maximum pooling process is performed on an output of a first residual block among the plurality of residual blocks, and a result of the maximum pooling process is input to a residual block of a next level of the first residual block,
A maximum anti-pooling process is performed on an output of a first residual deconvolution block among the plurality of residual deconvolution blocks, and the result of the maximum anti-pooling process is the first residual deconvolution block. input to the residual deconvolution block of the next level of the block
Face image analysis method. According to claim 1,
Each of the plurality of residual deconvolution blocks includes a compactor, a detail learner, and a dimensionality reducer.
Face image analysis method. 10. The method of claim 9,
The detail learner includes a residual branch and a deconvolution branch.
Face image analysis method. According to claim 1,
The residual network includes 4 or 5 residual blocks, and the number of residual deconvolution blocks of the residual deconvolution network is one less than the number of residual blocks of the residual network.
Face image analysis method. According to claim 1,
each convolutional layer of a convolution block in the residual network comprises 64 convolution kernels, and each deconvolution layer of a deconvolution block in the residual deconvolution network comprises 64 deconvolution kernels.
Face image analysis method. According to claim 1,
The residual block doubles the number of channels of the input data, and the residual deconvolution block reduces the number of channels of the input data by half.
Face image analysis method. According to claim 1,
wherein the Nth residual block comprises a second to last residual block or a third to last residual block among a plurality of sequentially combined residual blocks arranged in an input-to-output direction.
Face image analysis method. a processor for inputting a face image into a residual network including a plurality of sequentially combined residual blocks arranged in an input-to-output direction;
a residual network for processing face images; and
A residual deconvolution network for obtaining an analysis map from an output of an Nth residual block among the plurality of residual blocks
including,
the residual network passes the output of the Nth residual block to the residual deconvolution network, where N is a natural number and is less than the total number of residual blocks of the residual network;
The residual deconvolution network includes a plurality of sequentially combined residual deconvolution blocks, wherein the plurality of residual deconvolution blocks respectively correspond to first to Nth residual blocks among the plurality of residual blocks.
face image analysis device 16. The method of claim 15,
The apparatus of claim 1, further comprising: a prior information acquisition network configured to acquire prior information of a face image, and obtain an analysis result by combining the prior information and an output of the residual deconvolution network. 17. The method of claim 16,
The processor is configured to improve the analysis result by using a dense condition random field method. a pre-training step of training the residual network by adjusting a weight parameter of the residual network using training data for face identification; and
A joint training step of training the residual network and the residual deconvolution network by adjusting a weight parameter of the residual deconvolution network using training data for face image analysis and further adjusting a weight parameter of the residual network
including,
The facial image analysis apparatus includes the residual network and the residual deconvolution network,
The residual network includes a plurality of sequentially combined residual blocks arranged in an input to output direction, processes a face image, and passes an output of an Nth residual block to the residual deconvolution network, where N is a natural number and is less than the total number of residual blocks of the residual network,
The residual deconvolution network obtains an analysis map from an output of an Nth residual block among the plurality of residual blocks, and includes a plurality of sequentially combined residual deconvolution blocks, wherein the plurality of residual deconvolution blocks include: corresponding to the first to Nth residual blocks among the plurality of residual blocks, respectively,
A training method for a facial image analysis device. 19. The method of claim 18,
The pre-training step is
Input the training data for face identification into the residual network, perform a face identification operation, perform average pooling on the output of the last residual block of the residual network, and perform a full join on the residual network ) and pre-training the residual network by adjusting the weight parameters of the residual network,
A training method for a facial image analysis device. 20. The method of claim 19,
The pre-training step is
adjusting a weight parameter of the residual network to minimize a softmax function,
A training method for a facial image analysis device. 19. The method of claim 18,
The pre-training step is
randomly initializing a weight parameter in the residual network,
A training method for a facial image analysis device. 19. The method of claim 18,
The combined training phase is
Initialize the weight parameter of the residual network with the weight parameter obtained from the pre-training step, randomly initialize the weight parameter of the residual deconvolution network, and input the output of the Nth residual block to the residual deconvolution network, , input the training data for facial image analysis into the residual network, and perform a facial image analysis operation using the residual network and the residual deconvolution network to obtain a weight parameter of the residual deconvolution network and a weight of the residual network to adjust the parameters,
A training method for a facial image analysis device. 19. The method of claim 18,
The combined training phase is
Adjusting a weight parameter of the residual deconvolution network and a weight parameter of the residual network to minimize a softmax function,
A training method for a facial image analysis device. 19. The method of claim 18,
The facial image analysis apparatus further comprises a prior information acquisition network for acquiring prior information of the face image, combining the prior information and an output of the residual deconvolution network to obtain an analysis result,
Further comprising a prior information acquisition network training step of training the prior information acquisition network by adjusting a weight parameter in the prior information acquisition network by a facial image analysis operation,
A training method for a facial image analysis device. 25. The method of claim 24,
The pre-information acquisition network training step includes:
a first training step of training the prior information acquisition network; and
a second training step of training the residual network, the residual deconvolution network, and the prior information acquisition network;
In the first training step, all parameters other than the weight parameters in the prior information acquisition network are fixed, and the weight parameters in the dictionary information acquisition network are adjusted;
In the second training step, the prior information acquisition network is initialized by using the weight parameter in the adjusted prior information acquisition network, all parameters other than the weight parameter in the dictionary information acquisition network are variable, the residual network, the residual A facial image analysis operation is performed on the entire deconvolution network and the prior information acquisition network using training data for facial image analysis, so that the residual network and the residual deconvolution network are adjusted and within the prior information acquisition network. Each weight parameter is further adjusted,
A training method for a facial image analysis device. 26. The method of claim 25,
The pre-information acquisition network training step includes:
Adjusting a weight parameter in the prior information acquisition network to minimize a softmax function,
A training method for a facial image analysis device. 19. The method of claim 18,
Further comprising a pre-processing step for the training data for face identification or the training data for face image analysis,
The pre-processing step may include directly inputting the training data for face identification or the training data for face image analysis into the residual network without changing it, or performing random mirroring ( performing random mirroring, or performing a random cut out on the training data for face identification or the training data for analyzing the face image,
A training method for a facial image analysis device.

Images