KR20230061161A - Fashion simulation device and method - Google Patents

Abstract

A fashion simulation device is disclosed. A fashion simulation device according to the present invention includes an encoder that extracts a feature map including image features and spatial features using an original image, original pose information, and target pose information, and generates a pose conversion image using the feature map. A decoder, a style enhancement model that extracts a style code from the original image and generates a style enhancement image using the style code and the feature map, and generates a final image by synthesizing the pose conversion image and the style enhancement image. It includes a control unit that

Description

Fashion simulation device and method {FASHION SIMULATION DEVICE AND METHOD}

The present invention relates to a fashion simulation device and method capable of implementing fashion in detail and accurately even after a pose is transformed by separately processing and synthesizing style information in a technique of transforming a pose while maintaining a person's fashion. .

Recently, research on a Human Pose Transformer (HPT) that transforms a person's pose while maintaining a person's fashion (clothes, makeup, hair style, etc.) has been actively conducted. However, with the currently implemented HPT technology, there is a problem in that it is difficult to implement a natural image.

1 is a diagram illustrating a result of converting an original image according to currently implemented HPT technology.

Referring to FIG. 1 , when target images 1120 and 1130 obtained by transforming a pose from an original image 1110 are generated, detailed resolution of the fabric is lowered, and various style information such as color, texture, and pattern of the clothes is lost. There is a problem that is distorted or omitted.

Also, when a pose is changed from an original image, style information must be inferred for a region that was not expressed in the original image (for example, a part of a shirt covered by an arm when a person is wearing a shirt). However, in the conventional HPT technology, there is a problem in that style information in a corresponding area is expressed ambiguously.

The present invention is to solve the above-mentioned problems, and an object of the present invention is a technique for transforming a pose while maintaining a person's fashion, by separately processing and then synthesizing style information, detailing the fashion even after the pose is transformed. It is to provide a fashion simulation device and method that can be accurately implemented.

A fashion simulation device according to the present invention includes an encoder that extracts a feature map including image features and spatial features using an original image, original pose information, and target pose information, and generates a pose conversion image using the feature map. A decoder, a style enhancement model that extracts a style code from the original image and generates a style enhancement image using the style code and the feature map, and generates a final image by synthesizing the pose conversion image and the style enhancement image. It includes a control unit that

In this case, the style enhancement model includes a style encoder for extracting the style code including color, texture, and pattern features from the original image, and upsampling by mapping the style code to the feature map. It may include a style decoder that generates an enhanced image.

In this case, the style decoder may include a plurality of upsampling layers, and each of the plurality of upsampling layers may output a current enhancement map by mapping the style code to an enhancement map output from a previous upsampling layer.

Meanwhile, the encoder may generate an intermediate feature map for an intermediate pose between the original pose information and the target pose information.

In this case, the encoder includes a plurality of intermediate generation units, and each of the plurality of intermediate generation units receives a previous intermediate feature map, and uses the previous intermediate feature map to intermediate target poses corresponding to the corresponding intermediate generation units. It is possible to create a current intermediate feature map in which is reflected.

In this case, each of the plurality of intermediate generating units obtains a pose code by pose processing a previous intermediate spatial feature included in the previous intermediate feature map, and converts a previous intermediate image feature included in the previous intermediate feature map to the pose code. It may include a pose processing unit that reflects and outputs the current intermediate space feature.

In this case, each of the plurality of intermediate generating units obtains an image code by image processing a previous intermediate image feature included in the previous intermediate feature map, and outputs a current intermediate image feature by reflecting the pose code to the image code. An image processing unit may be further included.

Meanwhile, a face transformation model for generating a face image by detecting a face from the original image, and generating a face pose transformation image using the face image, original face pose information, and target face pose information, wherein the control unit, A second final image may be generated by additionally combining the face pose conversion image with the final image.

Meanwhile, in the fashion simulation method according to the present invention, an encoder extracts a feature map including image features and spatial features using an original image, original pose information, and target pose information, and a decoder uses the feature map. generating a pose transformation image, a style enhancement model extracting a style code from the original image, and generating a style enhancement image using the style code and the feature map, and, by a controller, the pose transformation and generating a final image by compositing the image and the style-enhanced image.

In this case, the step of extracting a style code from the original image and generating a style-enhanced image using the style code and the feature map includes the style code including color, texture, and pattern characteristics from the original image. The method may include extracting, and generating the style enhanced image by upsampling by mapping the style code to the feature map.

In this case, the step of generating the style enhanced image by performing upsampling by mapping the style code to the feature map, each of a plurality of upsampling layers maps the style code to an enhancement map output from a previous upsampling layer, and outputting the current enhancement map.

Meanwhile, the extracting of the feature map including the image feature and the spatial feature may include generating an intermediate feature map for an intermediate pose between the original pose information and the target pose information.

In this case, the step of generating the intermediate feature map may include receiving a previous intermediate feature map by each of a plurality of intermediate generating units, and using the previous intermediate feature map to reflect the current intermediate target pose corresponding to the corresponding intermediate generating unit. It may include generating a feature map.

In this case, the step of generating the current intermediate feature map by each of the plurality of intermediate generating units includes obtaining a pose code by pose-processing a previous intermediate spatial feature included in the previous intermediate feature map, and obtaining a pose code included in the previous intermediate feature map. and outputting a current intermediate space feature by reflecting a previous intermediate image feature to the pose code.

In this case, the step of generating a current intermediate feature map by each of the plurality of intermediate generating units may include obtaining an image code by image processing a previous intermediate image feature included in the previous intermediate feature map, and converting the pose code to the image code. The method may further include outputting a current intermediate image feature by reflection.

Meanwhile, a face conversion model detects a face from the original image to generate a face image, and generates a face pose conversion image using the face image, original face pose information, and target face pose information, and the control unit , generating a second final image by additionally combining the face pose conversion image with the final image.

According to the present invention, it is possible to generate an image in which a pose is transformed while minimizing damage to style information such as color, color tone, texture, pattern, etc., and in particular, problems such as lack of detailed color expression, ambiguity of pattern patterns, and style inconsistency can be solved. By doing so, there is an advantage of providing a natural pose conversion image.

1 is a diagram illustrating a result of converting an original image according to currently implemented HPT technology.
2 is a block diagram illustrating a fashion simulation device according to the present invention.
3 is a flowchart illustrating a fashion simulation method according to the present invention.
4 is a diagram for explaining an artificial intelligence model according to the present invention.
5 is a diagram for explaining in detail the operation of a pose transformation model according to the present invention.
6 is a diagram for explaining the operation of the style reinforcement model according to the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In implementing the present invention, components may be subdivided for convenience of description, but these components may be implemented in one device or module, or one component may be divided into multiple devices or modules may be implemented in

2 is a block diagram illustrating a fashion simulation device according to the present invention.

Components of the fashion simulation device 100 described in FIG. 1 are not essential to implement the operation according to the present invention, so some of the components may be omitted.

The fashion simulation device 100 may include an image capture unit 110, a controller 120, a memory 130, and an output unit 140.

The fashion simulation device 100 may be configured to receive, classify, store and output information to be used for data mining, data analysis, intelligent decision making and machine learning algorithms.

The image acquisition unit 110 may collect original images. Here, the original image means an image before pose transformation.

Here, the image acquisition unit 110 may include a camera for inputting an image signal and obtain an original image captured by the camera. In addition, the image acquisition unit 110 includes a communication unit for communicating with an external device or an input unit for receiving data from a user, and may receive an original image through the communication unit or input unit.

Also, the image acquiring unit 110 may obtain a target image. Here, the target image may include pose information to be transformed (target pose information), and in this case, the controller 120 may extract target pose information from the target image.

Also, the image acquisition unit 110 may obtain target pose information generated from the outside.

The output unit 140 may include a display unit that generates an output related to time. The display unit displays information processed by the fashion simulation device 100, for example, execution screen information of an application program driven by the fashion simulation device 100 or UI (User Interface) and GUI (Graphic User Interface) according to such execution screen information. information can be displayed.

The memory 130 may store the artificial intelligence model 131 . Specifically, the artificial intelligence model 131 may be implemented as hardware, software, or a combination of hardware and software. In this case, one or more instructions constituting the artificial intelligence model 131 may be stored in the memory 130 .

Also, the memory 130 may store programs or other commands for operating the fashion simulation device 100 .

Meanwhile, the controller 120 may control overall operations of the fashion simulation device 100 . Here, the control unit may be used interchangeably with terms such as controller, processor, and microprocessor.

Meanwhile, the controller 120 may read and drive the artificial intelligence model 131 stored in the memory 130 . Therefore, the operation of the artificial intelligence model 131 described below can also be seen as the operation of the control unit 120.

3 is a flowchart illustrating a fashion simulation method according to the present invention.

In the fashion simulation method according to the present invention, the encoder extracts a feature map including image features and spatial features using an original image, original pose information, and target pose information (S310), and a decoder, the feature map Generating a pose conversion image using (S330), extracting a style code from the original image by a style enhancement model, and generating a style enhancement image using the style code and the feature map (S350); And, it may include, by the controller, generating a final image by synthesizing the pose change image and the style enhancement image (S370).

4 is a diagram for explaining an artificial intelligence model according to the present invention.

The artificial intelligence model 131 according to the present invention may include a pose transformation model 410 and a style enhancement model 420 .

The pose conversion model 410 is an artificial neural network that receives an original image Io and aims to transform only a pose of a person while maintaining style information of the original image.

Accordingly, the pose conversion model 410 may include an encoder 413 that receives input data and extracts a feature map Ft. In this case, the encoder 413 may output a feature map Ft by encoding input data based on parameters (weight, bias, etc.) of elements constituting the encoder (neural network, node, etc.).

In addition, the pose transformation model 410 may include a decoder 416 that receives the feature map Ft and outputs a pose transformation image Ic. In this case, the decoder 416 may decode input data based on parameters (weights, biases, etc.) of elements constituting the decoder (neural networks, nodes, etc.) and output pose conversion images Ic.

Meanwhile, input data for the pose transformation model 410 may include an original image Io, original pose information Ps, and target pose information Pt.

Here, the original pose information Ps includes information on a pose of a person included in the original image Io, and may be skeleton information representing a skeleton point of the person included in the original image Io.

For example, the controller 120 may extract the original pose information Ps from the original image Io. In this case, the control unit 120 extracts 18 skeletal points from the original image Io through a pre-learned pose extractor, and the original pose information (Ps) representing the coordinates of the extracted skeletal points as an 18-channel pose heat map. ) can be created.

In addition, the target pose information Pt includes information on a pose to be transformed of a person included in the original image Io, and may be skeleton information representing a skeletal point of the person taking the transformed pose.

Meanwhile, the style enhancement model 420 is an artificial neural network that aims to enhance and express the style information of the original image Io rather than the pose conversion model 410 .

Here, the style information may include at least one of color, texture, and pattern. Here, the pattern may refer to a pattern in which colors or textures repeatedly appear, such as a sprite pattern or checkerboard pattern of a T-shirt.

Input data for the style enhancement model 420 may be the original image Io. That is, in order to enhance and express the style information of the original image Io rather than the pose conversion model 410, the pose conversion model 410 includes the original image Io, the original pose information Ps, and the target pose information Pt. In contrast to being used as input data, only the original image Io may be used as input data in the style enhancement model 420 .

Also, the style enhancement model 420 may extract a style code including features of color, texture, and pattern from the original image Io. Also, the style enhancement model 420 may output the style enhancement image Rt using the style code and the feature map Ft.

The following will be described with reference to FIG. 5 together with FIG. 4 with respect to S310.

5 is a diagram for explaining in detail the operation of a pose transformation model according to the present invention.

The controller 120 may input the original image Io, the original pose information Ps, and the target pose information Pt to the pose conversion model 410.

In this case, the encoder 413 of the pose conversion model 410 may extract and output the feature map Ft using the original image Io, the original pose information Ps, and the target pose information Pt. .

Here, the feature map (Ft) is obtained by downsampling the original image (Io), the original pose information (Ps), and the target pose information (Pt), and may include an image feature (Fin) and a spatial feature (Fpn) there is.

Here, the spatial feature (Fpn) may be a feature vector of a pose of a person, and may include a feature (spatial information) of a pose (target pose) after transformation. In addition, the space feature Fpn may include not only the characteristics of the character's posture, but also the characteristics of clothes and accessories worn by the character.

Also, the image feature (Fin) may be a feature vector for pose and style information of a person. That is, the image feature Fin may be information in which the style information of the original image Io is reflected in the pose after conversion (target pose).

On the other hand, the larger the range of conversion from the original pose information (Ps) to the target pose information (Pt), the more distortion occurs in the style information and pose. On the other hand, the range of conversion from the original pose information (Ps) to the target pose information (Pt) The smaller it is, the higher the accuracy.

Therefore, in the present invention, accuracy is increased by calculating intermediate feature maps for intermediate poses between the original pose information (Ps) and the target pose information (Pt).

Specifically, the encoder 413 of the pose conversion model 410 may generate an intermediate feature map for an intermediate pose between the original pose information Ps and the target pose information Pt.

More specifically, the original image Io may be initially input to the encoder 413 of the pose transformation model 410 . In this case, the encoder 413 may output an initial image feature (Fio) through a downsampling convolution layer.

Also, initially, original pose information (Ps) and target pose information (Pt) may be input to the encoder 413 . In this case, the encoder 413 may output the initial spatial feature Fpo through the downsampling convolution layer.

In addition, the initial spatial map including the initial image feature (Fio) and the initial spatial feature (Fpo) is converted while sequentially passing through a plurality of intermediate generating units (Unit 1, Unit 2, Unit n) in the encoder 413, Accordingly, the feature map Ft including the image feature Fin and the spatial feature Fpn may be finally output. Here, the number n of units may be changed according to the type of original image or other settings.

Further, each of the plurality of intermediate generating units (Unit 1, Unit 2, and Unit n) may generate a current intermediate feature map in which an intermediate target pose corresponding to the intermediate generating unit is reflected by using a previous intermediate feature map.

Hereinafter, the second intermediate generating unit (Unit 2) will be described as an example, and the operation of the second intermediate generating unit (Unit 2) can be applied to other intermediate generating units as well.

Referring to FIG. 5B , the second intermediate generation unit (Unit 2) may receive the previous intermediate feature map from the previous intermediate generation unit (Uint 1). Here, the previous intermediate feature map may include a previous intermediate image feature (Fi1) and a previous intermediate spatial feature (Fp1).

And, the second intermediate generating unit (Unit 2) includes a pose processing unit 520 composed of one or more pose processing layers, and pose-processes the previous intermediate spatial feature (Fp1) included in the previous intermediate feature map to generate a pose code. can be obtained

In this case, the intermediate target pose corresponding to the second intermediate generating unit Unit 2 may be provided as well as the previous intermediate spatial feature Fp1 . That is, the controller 120 uses the original pose information (Ps) and the target pose information (Pt) to generate an intermediate target pose for an intermediate posture in the process of proceeding from the original pose information (Ps) to the target pose information (Pt). can do. In addition, the control unit 120 generates the same number of intermediate target poses as the number of intermediate generation units, and sequentially assigns the intermediate target poses to the intermediate generation units (Unit 1, Unit 2, and Unit n) according to the order in which the poses are transformed. can provide Therefore, the intermediate target pose for the second intermediate posture in the process of proceeding from the original pose information (Ps) to the target pose information (Pt) may be provided to the second intermediate generation unit (Unit 2).

In this case, the second intermediate generating unit (Unit 2) may obtain a pose code by pose-processing the previous intermediate spatial feature (Fp1) using an intermediate target pose corresponding to the second intermediate generating unit (Unit 2). That is, the intermediate target pose corresponding to the second intermediate generating unit (Unit 2) may be reflected in the pose code output from the pose processing unit 520 of the second intermediate generating unit (Unit 2).

On the other hand, the second intermediate generating unit (Unit 2) includes an image processing unit 510 composed of one or more image processing layers, and image-processes the previous intermediate image feature (Fi1) included in the previous intermediate feature map to obtain an image code can do.

Meanwhile, the previous intermediate image feature (Fi1) includes style information to which an intermediate target pose corresponding to the first intermediate image generating unit (Unit 1) is applied, and the second intermediate image generating unit (Unit 2) uses the previous intermediate image feature (Fi1). The image code was generated using . That is, the image code generated by the second intermediate generating unit (Unit 2) is still reflecting the previous intermediate target pose information.

Accordingly, the image processing unit 510 may output the current intermediate image feature Fi2 by reflecting the pose code generated by the second intermediate generating unit Unit 2 to the calculated image code.

)를 적용하여, 제2 중간 생성 유닛(Unit 2)에 상응하는 중간 타겟 포즈에서의 마스크 정보(Mt)를 생성할 수 있다. 그리고 이미지 처리부(510)는 제2 중간 생성 유닛(Unit 2)에서 출력된 이미지 코드와 마스크 정보(Mt) 간의 요소 별 곱셈(multiply)을 통하여, 현재 중간 이미지 특징(Fi2)을 출력할 수 있다. 즉, 중간 이미지 특징(Fi2)은 제2 중간 생성 유닛(Unit 2)에 상응하는 중간 타겟 포즈가 적용된 스타일 정보를 포함할 수 있다.Specifically, the image processing unit 510 performs a sigmoid function (

) may be applied to generate mask information (Mt) in an intermediate target pose corresponding to the second intermediate generating unit (Unit 2). The image processing unit 510 may output the current intermediate image feature Fi2 through element-by-element multiplication between the image code output from the second intermediate generation unit Unit 2 and the mask information Mt. That is, the intermediate image feature Fi2 may include style information to which an intermediate target pose corresponding to the second intermediate generating unit Unit 2 is applied.

Meanwhile, the pose code output from the pose processing unit 520 includes information on the skeleton points of a person taking an intermediate target pose corresponding to the second intermediate generating unit (Unit 2).

Further, the pose processing unit 520 may output the current intermediate spatial feature Fp2 by reflecting the previous intermediate image feature Fio included in the previous intermediate feature map to the pose code. That is, the pose processing unit 520 performs a depth concatenation operation between the previous intermediate image feature (Fio) and the pose code, and the current information including information about the skeletal points of the person taking the intermediate target pose and the skeletal points connecting the skeletal points. The intermediate space feature (Fp2) can be output. That is, the pose processing unit 520 generates a current intermediate spatial feature Fp2 in which errors between skeletal points are corrected and connection information between skeletal points is reinforced using the previous intermediate image feature Fio and the pose code, and the generated current intermediate space feature Fp2 is generated. The intermediate spatial feature (Fp2) can be passed on to the next intermediate generating unit.

Meanwhile, in the same way, the last intermediate generating unit (Unit n) may output an intermediate feature map including the current intermediate spatial feature (Fpn) and the current intermediate image feature (Fin).

In this case, the intermediate feature map output from the last intermediate generating unit (Unit n) may be the feature map (Ft) output from the encoder 413 of the pose conversion model 410. That is, the target pose information (Pt) is reflected in the intermediate space feature (Fpn) output from the last intermediate generation unit (Unit n), and the intermediate image feature (Fin) output from the last intermediate generation unit (Unit n) is the target pose information. Style information to which (Pt) is applied may be reflected.

Next, referring to FIG. 4 , the decoder 416 of the pose transformation model 410 may generate a pose transformation image Ic using a feature map Ft including image features and spatial features.

Specifically, the decoder 416 of the pose conversion model 410 includes one or more upsampling layers, and the one or more upsampling layers upsamples the feature map Ft to generate a pose conversion image Ic reflecting the target pose Pt. ) can be created.

Meanwhile, the pose transformation process Gp of the pose transformation model 410 may be expressed as the following function.

(Ic: pose conversion image, Go: pose conversion process function of pose conversion model, Io: original image, Ps: original pose information, Pt: target pose information)

Meanwhile, the pose conversion image Ic includes a person taking a target pose, and style information about the color, texture, pattern, etc. of the person included in the original image Io is reflected in this person. However, as described in FIG. 1, this style information is distorted or partially missing. And such incomplete style information may be supplemented by the style enhancement model 420 . This will be described with reference to FIG. 6 together with FIG. 4 .

6 is a diagram for explaining the operation of the style reinforcement model according to the present invention.

The style enhancement model 420 according to the present invention may include a style encoder 610 and a style decoder 620 . The style enhancement model 420 may extract a style code from the original image Io and generate a style enhancement image Rt using the style code and the feature map Ft (S350).

First, the style encoder 610 may extract features related to style information from the original image Io.

In detail, the style encoder 610 may include one or more style encoding layers 611 . In this case, the one or more style encoding layers 611 may extract features of color, texture, and pattern by downsampling the original image Io.

In addition, the fully connected neural network (FC) in the style encoder 610 may convert the extracted feature into a style code through Global Average Pooling (GAP). Accordingly, the style code may include characteristics of color, texture, and pattern extracted from the original image Io.

Next, the style decoder 620 may generate the style enhanced image Rt using the style code and the feature map Ft. Here, the style enhanced image Rt may also be named Detail Residual Map.

In detail, the style decoder 620 may include a plurality of upsampling layers 621 .

In this case, the plurality of upsampling layers 621 may generate the style enhanced image Rt by performing upsampling by mapping a style code to the feature map Ft.

That is, the feature map Ft described above includes spatial features, and target pose information is reflected in the spatial features. The plurality of upsampling layers 621 may generate the style enhanced image Rt by mapping and upsampling spatial features and style codes included in the feature map Ft.

In addition, the plurality of upsampling layers 621 may individually reflect style codes and output enhancement maps.

In detail, each of the plurality of upsampling layers 621 may output a current enhancement map by mapping a style code to an enhancement map output from a previous upsampling layer.

For example, the first upsampling layer may output a first enhancement map by mapping and upsampling spatial features and style codes included in the feature map Ft. Also, the second upsampling layer may output a second enhancement map by mapping and upsampling the first enhancement map and the style code. In this way, an enhancement map output by a last upsampling layer among a plurality of upsampling layers may be used as the style enhancement image Rt.

Meanwhile, one or more style decoding layers 621 may be composed of an Adaptive Instance Normalization (AdaIN) layer.

Meanwhile, since the style enhanced image Rt is generated by repeating mapping and upsampling of the style code and feature map Ft, the target pose information is reflected in the style information. In addition, the image feature extracted from the pose conversion model 410 is a mixture of pose-related information and style-related information, whereas the style code extracted from the style enhancement model 420 is style-related from only the original image Io. information was extracted. Accordingly, the style enhancement image Rt may include enhanced style information compared to the pose change image Ic.

Meanwhile, the style reinforcement process GT of the style decoder 620 of the style reinforcement model 420 can be expressed as the following function.

(Rt: style enhanced image, GT: style enhanced function, style code: style code, Ft: feature map)

Next, the controller 120 may generate a final image It by combining the style enhancement image Rt with the pose change image Ic (S370).

In this case, the controller 120 may superimpose the style enhancement image Rt on the pose conversion image Ic, and superimpose the style enhancement image Rt on the pose conversion image Ic using an alpha blending technique. can

Meanwhile, referring to FIG. 4 again, the artificial intelligence model 131 may further include a face conversion model 430.

Here, the face transformation model 430 is an artificial neural network for the purpose of extracting a face image from the original image Io and transforming only the pose of the face while maintaining style information of the face image.

Accordingly, the face transformation model 430 may include a face detector 430 that detects a face from the original image Io and generates a face image Fo.

In addition, the operation of the pose transformation model 410 described above may also be applied to the operation of the face enhancement model 436 .

That is, the controller 120 may provide input data including the face image Fo, original face pose information, and target face pose information FS to the face enhancement model 436 . In this case, the face enhancement model 436 may generate a face pose conversion image Fg using the face image Fo, original face pose information, and target face pose information Fs.

Specifically, the encoder of the face enhancement model 436 may encode input data and output a facial feature map, and the decoder of the face enhancement model 436 may decode the facial feature map and output a face pose conversion image Fg. there is.

In this case, the controller 120 may generate the second final image IF by additionally combining the face pose conversion image Fg with the final image It. This can be expressed in the following equation.

는 2D 가우시간(Gaussian) 커널을 의미하고, *는 2D 컨볼루션 연산자를 의미할 수 있다. 또한 1_BF는 인디케이터(indicator) 함수로, 얼굴 영역에서는 1을 반환하고, 나머지 영역에서는 0을 반영할 수 있다.In Equation 3,

denotes a 2D Gaussian kernel, and * may denote a 2D convolution operator. Also, 1 _BF is an indicator function, and may return 1 in the face area and reflect 0 in the other areas.

Accordingly, according to Equation 4, a secondary final image I F in which the face pose conversion image Fg is reflected in the face area and the final image _It is reflected in the remaining areas may be generated.

Next, a loss function for training the artificial intelligence model 131 will be described.

First, a loss function for training the pose transformation model 410 may be defined as follows.

)와 포즈 변환 이미지(Ic)간의 차이, L_per: Perceptual loss,

: 상수,

: 상수)(L _PTP : loss function for pose conversion model 410, L _recon : correct image (

) and the difference between the pose conversion image (Ic), L _per : Perceptual loss,

: a constant,

: a constant)

)(타겟 포즈를 취하고 완전한 스타일 정보를 가지고 있는 이미지)와 포즈 변환 이미지(Ic)간의 차이(Lrecon)를 이용하여 포즈 변환 모델(410)을 트레이닝 할 수 있다. 더욱 구체적으로 정답 이미지(

)와 포즈 변환 이미지(Ic)간의 차이(Lrecon)는, 정답 이미지(

)와 포즈 변환 이미지(Ic)의 각 픽셀 값을 유클리드 거리(Euclidean Distance)로 정규화(Normalized)함으로써 산출될 수 있다.That is, the control unit 120 provides an answer image (

) (an image taking a target pose and having complete style information) and the pose conversion image Ic (Lrecon), the pose conversion model 410 can be trained. More specifically, the correct answer image (

) And the difference (Lrecon) between the pose conversion image (Ic) is the correct image (

) and each pixel value of the pose conversion image Ic by Euclidean distance.

In addition, L _per means perceptual loss, and can be expressed as follows.

: 미리 학습된 VGG19 네트워크, l: 레이어의 인덱스, C: 채널수, H: 높이, W: 넓이,

: 정답 이미지, Ic: 포즈 변환 이미지)(

: pretrained VGG19 network, l: layer index, C: number of channels, H: height, W: width,

: Answer image, Ic: Pose conversion image)

Next, the pose conversion model 410 and the style reinforcement model based on the loss function of Equation 5, the loss function for training the style reinforcement model 420, and the generative adversarial network (GAN) algorithm A loss function to train 420 may be added. This can be defined by the following equation.

(L _DEP : loss function for pose transformation model 410 and style enhancement model 420, L _sty : loss function for style enhancement model 420, L _GAN : loss function based on GAN algorithm)

Also, Lsty is a loss function based on the Gram Matrix and can be expressed as follows.

: 미리 학습된 VGG19 네트워크, l: 레이어의 인덱스, C: 채널수, H: 높이, W: 넓이,

: 정답 이미지, I_F: 최종 이미지)(L _sty : loss function for the style enhancement model 420,

: pretrained VGG19 network, l: layer index, C: number of channels, H: height, W: width,

: correct answer image, I _F : final image)

)와 최종 이미지(I_F)를 VGG19 네트워크에 입력하고, 레이어에서 출력한 Style Representation에 Gram Matrix 함수(G)를 활용하여 내적을 수행함으로써, 최소 손실 값을 산출할 수 있다. 즉 Gram Matrix 함수(G)를 활용하여 레이어의 Style Representation를 내적함으로써, 스타일 정보에 더 집중한 손실 함수가 설계될 수 있다.In this case, the control unit 120 is a correct image (

) and the final image (I _F ) into the VGG19 network, and performing the dot product using the Gram Matrix function (G) on the Style Representation output from the layer, the minimum loss value can be calculated. That is, by using the Gram Matrix function (G) to dot the style representation of the layer, a loss function more focused on style information can be designed.

Meanwhile, L _GAN can be expressed as a loss function based on the GAN algorithm as follows.

That is, the loss function (L _GAN ) based on the GAN algorithm may include two conditional descriptors. Specifically, the loss function (L _GAN ) based on the GAN algorithm may include a style descriptor (Ds) and a pose descriptor (Dp).

) 간의 제2 차이를 산출하고, 제1 차이와 제2 차이 간의 차를 산출할 수 있다. 이에 따라, 원본 이미지(Io)와 최종 이미지(It)의 스타일 정보에 대하여, 제1 차이와 제2 차이의 차가 0에 가까울수록 GAN 알고리즘에 기반한 손실 함수(L_GAN)는 작아질 수 있다.In this case, the style descriptor (Ds) is the first difference between the original image (Io) and the final image (It), the original image (Io) and the correct image (with respect to style information)

), and a difference between the first difference and the second difference may be calculated. Accordingly, the loss function (L _GAN ) based on the GAN algorithm may decrease as the difference between the first difference and the second difference between the original image Io and the final image It is closer to 0.

) 간의 제2 차이를 산출하고, 제1 차이와 제2 차이 간의 차를 산출할 수 있다. 이에 따라, 타겟 포즈 정보(Pt)와 최종 이미지(It)의 포즈 정보에 대하여, 제1 차이와 제2 차이의 차가 0에 가까울수록 GAN 알고리즘에 기반한 손실 함수(L_GAN)는 작아질 수 있다.In addition, the pose descriptor (Dp) is the first difference between the target pose information (Pt) and the final image (It), the target pose information (Pt) and the correct image (with respect to pose information)

), and a difference between the first difference and the second difference may be calculated. Accordingly, as the difference between the first difference and the second difference between the target pose information Pt and the pose information of the final image It is closer to 0, the loss function L _GAN based on the GAN algorithm may decrease.

Currently, in the case of virtual fitting service technology applied in commercial online fashion shopping malls, not only poses are fixed, but also there is a limit in that the texture, color, pattern, etc. of clothes cannot be realistically expressed.

However, according to the present invention, it is possible to generate an image in which a pose is transformed while minimizing damage to style information such as color, color, texture, pattern, etc. By solving the problem, there is an advantage in providing a natural pose change image.

In addition, according to the present invention, when a pose is changed from an original image, there is an advantage in that even style information of a region not expressed in the original image can be clearly expressed.

Accordingly, the present invention can be applied to various fields such as fashion, security, and video production. For example, the present invention may be used to analyze a suspect in various ways in the field of image processing security such as CCTV. That is, in the present invention, not only can images of various poses be created from one original image, but also clothes worn by a suspect can be reproduced without damage, so it can be effectively used to analyze a suspect in various crime scenes or security fields.

For another example, the present invention can be used to provide fashion information of various poses to consumers using online shopping malls, and has the advantage of being able to generate various poses of a person without additional photography in video production.

The above-described present invention can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is Also, the computer may include a control unit. Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

110: image acquisition unit 120: control unit
130: memory 140: output unit

Claims (16)

an encoder that extracts a feature map including image features and spatial features using the original image, original pose information, and target pose information;
a decoder generating a pose conversion image using the feature map;
a style enhancement model extracting a style code from the original image and generating a style enhanced image using the style code and the feature map; and
A control unit generating a final image by synthesizing the pose conversion image and the style enhancement image;
Fashion simulation device. According to claim 1,
The style reinforcement model,
a style encoder extracting the style code including color, texture, and pattern features from the original image; and
A style decoder configured to map the style code to the feature map and perform upsampling to generate the style enhanced image.
Fashion simulation device. According to claim 2,
The style decoder includes a plurality of upsampling layers,
Each of the plurality of upsampling layers,
Mapping the style code to the enhancement map output from the previous upsampling layer to output the current enhancement map
Fashion simulation device. According to claim 1,
The encoder,
Generating an intermediate feature map for an intermediate pose between the original pose information and the target pose information
Fashion simulation device. According to claim 4,
The encoder includes a plurality of intermediate generating units;
Each of the plurality of intermediate generating units,
Receiving a previous intermediate feature map, and generating a current intermediate feature map reflecting an intermediate target pose corresponding to a corresponding intermediate generation unit using the previous intermediate feature map
Fashion simulation device. According to claim 5,
Each of the plurality of intermediate generating units,
A pose processing unit that obtains a pose code by pose processing a previous intermediate spatial feature included in the previous intermediate feature map, and outputs a current intermediate spatial feature by reflecting a previous intermediate image feature included in the previous intermediate feature map to the pose code. including
Fashion simulation device. According to claim 6,
Each of the plurality of intermediate generating units,
An image processing unit that obtains an image code by image processing a previous intermediate image feature included in the previous intermediate feature map, and outputs a current intermediate image feature by reflecting the pose code to the image code.
Fashion simulation device. According to claim 1,
A face transformation model for generating a face image by detecting a face from the original image, and generating a face pose transformation image using the face image, original face pose information, and target face pose information;
The control unit,
Generating a second final image by additionally synthesizing the face pose conversion image to the final image
Fashion simulation device. extracting, by an encoder, a feature map including image features and spatial features using the original image, original pose information, and target pose information;
generating, by a decoder, a pose conversion image using the feature map;
extracting, by a style enhancement model, a style code from the original image, and generating a style enhancement image using the style code and the feature map; and
A controller generating a final image by synthesizing the pose conversion image and the style enhancement image;
Fashion simulation method. According to claim 9,
The step of extracting a style code from the original image and generating a style enhanced image using the style code and the feature map,
extracting the style code including color, texture, and pattern features from the original image;
generating the style-enhanced image by upsampling by mapping the style code to the feature map;
Fashion simulation method. According to claim 10,
The step of generating the style enhanced image by upsampling by mapping the style code to the feature map,
Each of the plurality of upsampling layers outputs a current enhancement map by mapping the style code to an enhancement map output from a previous upsampling layer.
Fashion simulation method. According to claim 9,
The step of extracting a feature map including the image feature and spatial feature,
Generating an intermediate feature map for an intermediate pose between the original pose information and the target pose information; comprising
Fashion simulation method. According to claim 12,
Generating the intermediate feature map,
Receiving, by each of a plurality of intermediate generating units, a previous intermediate feature map, and generating a current intermediate feature map in which an intermediate target pose corresponding to the corresponding intermediate generating unit is reflected by using the previous intermediate feature map.
Fashion simulation method. According to claim 13,
Generating, by each of the plurality of intermediate generating units, a current intermediate feature map,
Pose-processing a previous intermediate spatial feature included in the previous intermediate feature map to obtain a pose code, and outputting a current intermediate spatial feature by reflecting a previous intermediate image feature included in the previous intermediate feature map to the pose code; containing
Fashion simulation method. According to claim 14,
Generating, by each of the plurality of intermediate generating units, a current intermediate feature map,
Image processing of previous intermediate image features included in the previous intermediate feature map to obtain an image code, and outputting a current intermediate image feature by reflecting the pose code to the image code
Fashion simulation method. According to claim 9,
generating, by a face conversion model, a face image from the original image by detecting a face, and generating a face pose conversion image using the face image, original face pose information, and target face pose information; and
Further comprising, by the controller, generating a secondary final image by additionally combining the face pose conversion image with the final image
Fashion simulation method.

Images