KR20220124594A - Virtual fitting method supporting high-resolution image - Google Patents

Abstract

The present invention relates to a virtual fitting method for compositing a fitting garment to an original image using a computer device. In one embodiment, the disclosed virtual fitting method using a computer device may include the steps of: generating a segmentation from the original image; generating a modified fitting garment by modifying the fitting garment (c) based on the original image; generating a misaligned area in which a region of the modified fitting garment does not overlap among existing garment regions of the segmentation; and generating a synthesized image by inputting input data including the original image and fitting garment into an image synthesis algorithm.

Description

Virtual fitting method supporting high-resolution image

The present invention relates to a virtual fitting method using a computer, and more particularly, to a virtual fitting method and apparatus for generating a composite image obtained by synthesizing fitting clothes with an original image.

A virtual fitting technology that creates an image of wearing clothes virtually before actually wearing them and shows them to a user is currently widely used. However, the existing virtual fitting technology suggests a method of virtual fitting mainly on low-resolution datasets, so that at a high resolution of 1024x768 or higher, the virtual fitting result due to a small error remarkably leads to incorrect synthesis.

In addition, since the virtual fitting technology synthesizes and shows the clothes on the model on the image, the clothes do not naturally fit into the background and are awkward and often unnatural compared to the appearance of the actual consumer. It is often difficult to predict accurately.

Patent Document 1: Korean Patent Publication No. 2020-0038777 (published on April 14, 2020) Patent Document 2: Korean Patent Publication No. 2020-0139766 (published on December 14, 2020)

The present invention is to solve this problem, and by generating a robust image even at high resolution, it is possible to show a naturally synthesized appearance of clothes and to provide a virtual fitting method and apparatus that allow consumers to relatively accurately predict the appearance of actual clothes. intended to provide

According to an embodiment of the present invention, there is provided a virtual fitting method for synthesizing a fitting costume with an original image using a computer device, the method comprising: generating a segmentation from the original image; generating a deformed fitting garment by deforming the fitting garment (c) based on the original image; generating a misalignment region (Mmisalign) in which regions of the deformed fitting clothes do not overlap among the segmentation regions of the existing clothes; and generating a composite image by inputting input data including the original image and fitting clothes into an image synthesis algorithm.

According to an embodiment of the present invention, there is provided a computer-readable recording medium in which a computer program for executing the virtual fitting method is recorded.

According to an embodiment of the present invention, it is possible to generate a robust image even at high resolution and to generate a synthetic image in which clothes are naturally synthesized, which has a technical effect that enables consumers to relatively accurately predict the appearance of actual clothes. .

1 is a block diagram illustrating a system for implementing a virtual fitting method according to an embodiment;
2 is a block diagram of a virtual fitting system according to an embodiment;
3 is a view for explaining an operation of a preprocessor according to an embodiment;
4 is a flowchart illustrating an operation of a preprocessor according to an embodiment;
5 is a view for explaining a method of generating a clothing removal image (Ia) according to an embodiment;
6 is a view for explaining an operation of a segmentation unit according to an embodiment;
7 is a flowchart illustrating an operation of a segmentation unit according to an embodiment;
8 is a view for explaining the operation of the clothes transforming unit according to an embodiment;
9 is a flowchart for explaining the operation of the clothes transforming unit according to an embodiment;
Fig. 10 is a view for explaining a garment transforming unit according to an alternative embodiment;
11 is a view for explaining an operation of an image synthesizing unit according to an embodiment;
12 is a flowchart illustrating an operation of an image synthesizing unit according to an embodiment;
13 is a diagram for explaining input data of an image synthesis algorithm according to an embodiment;
14 to 16 are diagrams for explaining an image synthesis algorithm according to an embodiment;
Fig. 17 is a view for explaining an image synthesizing unit according to an alternative embodiment;
18 is a view for explaining the effects of the present invention according to an embodiment.

The above objects, other objects, features and advantages of the present invention will be easily understood through the following preferred embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed subject matter may be thorough and complete, and that the spirit of the present invention may be sufficiently conveyed to those skilled in the art.

In this specification, when terms such as first, second, etc. are used to describe components, these components should not be limited by these terms. These terms are only used to distinguish one component from another. The embodiments described and illustrated herein also include complementary embodiments thereof.

In this specification, the singular also includes the plural, unless specifically stated otherwise in the phrase. The expressions 'comprising', 'consisting of', and 'consisting of' as used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated elements.

As used herein, the term 'software' refers to a technology that moves hardware in a computer, and the term 'hardware' refers to a tangible device or device (CPU, memory, input device, output device, peripheral device, etc.) constituting the computer, and , the term 'step' means a series of processing or manipulations linked in time series to achieve a predetermined goal, and the term 'computer program', 'program', or 'algorithm' refers to a set of instructions that are summed up for processing by a computer. means, and the term 'program recording medium' refers to a computer-readable recording medium in which a program used for installing, executing, or distributing a program is recorded.

Terms such as '~ unit', '~ module', '~ unit', '~ block', and '~ board' used in this specification to refer to the elements of the present invention are used to refer to at least one function or operation. It may mean a physical, functional, or logical unit, which may be implemented as one or more hardware, software, or firmware, or may be implemented as a combination of one or more hardware, software, and/or firmware.

In this specification, 'processing device', 'computer', 'computing device', 'server device', and 'server' are operating systems such as Windows, Mac, or Linux, computer processors, memory, applications, storage devices (eg For example, HDD, SDD), and a device having a monitor may be implemented. The computer may be, for example, a device such as a desktop computer, a notebook computer, or a mobile terminal, but these are exemplary and not limited thereto. The mobile terminal may be one of a smart phone, a tablet PC, or a mobile wireless communication device such as a PDA.

A computer device according to an embodiment of the present invention receives an "original image" and a "fitting image" and synthesizes them to generate a "composite image" or a "result image". As used herein, the term “original image” refers to an image in which the whole body or a part of a person (model) who is going to wear a virtual costume is displayed. In addition, in this specification, "fitting clothes image" means an image including clothes (fitting clothes) to be virtually worn on the model of the original image. The fitting clothes may be tops and/or bottoms.

Also, in this specification, the term "existing clothes" refers to clothes replaced with fitting clothes, that is, clothes to be replaced with fitting clothes among clothes worn by the model of the original image. For example, in the case of a model wearing a top and a bottom in the original image, it will be understood that 'existing clothes' means the top of the original image.

In the present specification, a "composite image" or "result image" is an image obtained by receiving an original image and an image of fitting clothes and synthesizing them, and is an image in which a person of the original image is virtually dressed in fitting clothes.

Meanwhile, in this specification, when referring to a body or clothes of a person (model), the word 'image' may be omitted for convenience of explanation. For example, in this specification, 'person', 'top', 'bottom', 'fitting clothes', 'existing clothes', etc. may mean a real person, body part, or real clothes, but the object (body or clothes) is expressed It will be fully understood by those skilled in the art that it may mean an image (image file) or a portion of an image (pixels).

Hereinafter, the present invention will be described in detail with reference to the drawings. In describing the specific embodiments below, various specific contents have been prepared to more specifically describe the invention and help understanding. However, a reader having enough knowledge in this field to understand the present invention may recognize that the present invention may be used without these various specific details. In addition, it is mentioned in advance that parts that are well-known or commonly used techniques in describing the invention, but not largely related to the invention, are not described in order to avoid confusion in describing the present invention.

1 is a block diagram illustrating a system for executing a virtual fitting method according to an embodiment.

Referring to FIG. 1 , a virtual fitting method according to an embodiment may be executed in a computer device 10 . The computer device 10 may be implemented as a general general computer such as a desktop computer or a notebook computer, or may be implemented as a mobile terminal such as a smart phone. The computer device 10 may execute one or more application programs for executing a virtual fitting method to be described later with reference to FIGS. 2 to 17 .

In an embodiment, the computer device 10 may receive an image of a whole body or a part (eg, upper body) of a person (model) from the camera 20 and/or the user terminal 30 . The camera 20 may be a camera device independent of the computer device 10 or a camera connected to the computer device 10 by wire/wireless. When the computer device 10 is a smartphone, the camera 20 may be a camera embedded in the computer device 10 (ie, a smartphone). The user terminal 30 may be a portable device such as a smart phone, for example, may photograph a person with the camera of the user terminal 30 and transmit the captured image to the computer device 10 by wire or wirelessly.

In an embodiment, the computer device 10 receives an image file of a subject (eg, a person) from the camera 20 or the user terminal 30 . The image file is a computer-readable image file, and may be, for example, an image file having an extension such as JPG or TIF. Hereinafter, an image of the whole body or a part of a person inputted by the computer device 10 will be referred to as an “original image”.

Also, the computer device 10 may receive an image of a costume (hereinafter referred to as “fitting clothes”) to be virtually put on a person of the original image. An image of fitting clothes may also be received from an external device such as the camera 20 or the user terminal 30 .

The computer device 10 may output an image (hereinafter, referred to as a “result image” or “composite image”) of a person (model) wearing the fitting clothes by synthesizing the original image and the fitting clothes image.

In an embodiment, the computer device 10 may output the result image to the user through the display 40 . The display 40 may be a device independent of the computer device 10 , for example, if the computer device 10 is a smartphone, the display 40 is embedded in the computer device 10 (ie, a smartphone). It could also be a display.

2 is a block diagram of a virtual fitting system according to an embodiment.

Referring to FIG. 2 , in an embodiment, the computer device 10 may include a preprocessor 100 , a segmentation unit 200 , a clothing transforming unit 300 , and an image synthesizing unit 400 . Here, each of the pre-processing unit 100, the segmentation unit 200, the clothes transforming unit 300, and the image synthesizing unit 400 is one or more programs that process the original image and/or the fitting clothes image, deep learning algorithms, etc. It may be implemented as software, and may be stored in a storage device (eg, SDD or hard disk) of the computer device 10 and then loaded into a memory and executed.

The respective operations of the preprocessor 100 , the segmentation unit 200 , the garment transforming unit 300 , and the image synthesizing unit 400 will be described later with reference to the following drawings.

3 is a diagram for explaining the operation of the preprocessor 100 according to an embodiment, and FIG. 4 is a flowchart for explaining the operation of the preprocessor 100 .

The preprocessor 100 may perform an operation of removing the existing clothes from the original image. When the new clothes (ie, fitting clothes) are put on while the existing clothes are removed, the pre-processing unit 100 removes the existing clothes because the clothes can be freely worn.

3 and 4, in step S110, the preprocessor 100 extracts the pose information P from the original image I. The pose information P (or simply referred to as a “pose P”) may be extracted using a general pose extraction algorithm that finds feature points of a human pose in an image. In the drawing of FIG. 3 , the pose (P) image represents the feature points found by the pose extraction algorithm and lines connecting the feature points.

Thereafter, in step S120 , a first segmentation S is generated from the original image I. Segmentation (S) is also called "segmentation map", "semantic image", "semantic map", etc., and classifies all pixels in an image into a pre-specified class to create a channel image corresponding to each class. will create For example, a face, hair, top, bottom, arm, background, etc. are each a predefined class, and the first segmentation S is image data composed of as many channels as the number of this class. It is shown in color as one segmentation (S) image.

For the generation of the first segmentation S, the first segmentation S may be generated using a known segmentation generation algorithm 110 such as a deep learning algorithm, an artificial neural network algorithm, an image generation algorithm connected by an encoder and a decoder, etc. .

Thereafter, in step S130 , existing clothes (that is, clothes in this embodiment) are removed from the original image I based on the pose P and the first segmentation S. For example, first, a specific color (gray in the drawing) is painted with a brush along the shoulder-elbow-wrist path among the feature points of the pose (P). In other words, you can remove the existing clothes from the original image by painting gray with a circular brush. Then, as shown in Fig. 5 (a), based on the feature point information of the pose (P), a rectangular region is created based on both shoulders and the pelvis, and gray is painted in the region of the original image corresponding thereto, and Fig. 5 ( As shown in b), a rectangular area centered on the neck point is created and the corresponding area of the original image is painted gray to remove the existing clothes and the surrounding area from the original image.

After that, even in the final composite image (result image), if there is a class area (eg, face and bottom) that must be maintained, the area of the class to be maintained is set to the same as the original image again based on the first segmentation (S). Then, as shown in FIG. 5(c) , an image from which the existing clothes are removed (hereinafter also referred to as “clothes removed image Ia”) is obtained. In the costume removal image (Ia), only the existing clothes to be replaced with fitting clothes can be removed, but as shown in FIG. may be The costume removal image (Ia) can be used to provide information to the deep learning algorithm in the later image synthesis step, "You must keep this part".

FIG. 6 is a diagram illustrating an operation of the segmentation unit 200 according to an exemplary embodiment, and FIG. 7 is a flowchart illustrating an operation of the segmentation unit 200 . The segmentation unit 200 is a functional block for generating a new segmentation (hereinafter also referred to as “second segmentation”) suitable for clothes to be worn (ie, fitting clothes), and subsequent synthesis may be performed based on the generated second segmentation. have. That is, by completely removing the information of the existing clothes, creating a segmentation reflecting the shape of the fitting clothes, and using this for subsequent synthesis, the fitting clothes can be freely fitted to the body. In addition, it is possible to estimate the posture by providing pose information together.

In step 210 of FIG. 7, a pose P is extracted from the original image I, and a first segmentation S is generated in step S220. These steps (S210, S220) are the same as steps (S120 and S220) of FIG. 4, and the pose P and the first segmentation (S) generated in these steps (S120, S220) may be used.

Thereafter, in step S230 , the clothes are removed in the first segmentation S based on the pose P to generate the clothes removal segmentation Sa. For example, in the first segmentation (S), the clothes removal segmentation (Sa) is generated by painting black (meaning the background) in the area corresponding to the existing clothes with a circular brush to remove the existing clothes. For example, in the first segmentation S, black is painted on the upper part (orange in the original image I in FIG. 6) to remove the existing garment. Also, at this time, among the features of the pose (P), the arm area is also removed by painting black along the shoulder-elbow-wrist path with a brush, and then the class area (eg, face) that must be maintained in the composite image (result image). and bottom) is returned to the same as the original image again based on the first segmentation S, and accordingly, as shown in FIG. 6 , a clothing removal segmentation Sa from which the clothing and arm regions are removed is obtained.

)을 생성한다. 제2 세그먼테이션(

)의 생성은 딥러닝 알고리즘, 인공신경망 알고리즘 등 공지의 세그먼테이션 생성 알고리즘(220)을 사용할 수 있다. After that, in step S240, the second segmentation (

) is created. Second segmentation (

) may be generated using a well-known segmentation generation algorithm 220 such as a deep learning algorithm or an artificial neural network algorithm.

)은 피팅 의상(c)이 반영된 세그먼테이션이며, 따라서 일 실시예에서 제2 세그먼테이션(

)을 생성하기 위해, 원본 이미지에서 기존 의상이 제거된 의상제거 세그먼테이션(Sa), 및 포즈(P)와 피팅의상(c)이 입력데이터로서 활용된다. 피팅 의상(c)은 제2 세그먼테이션(

)에 반영되어야 할 의상 정보를 주기 위해 입력되고, 의상제거 세그먼테이션(Sa)은 제2 세그먼테이션(

) 생성을 위한 기본 이미지로서 이용되고, 포즈(P)는 피팅 의상(c)의 어느 부위가 어느 각도로 틀어지는지 등의 포즈 정보를 알려주기 위해 입력될 수 있다. Second segmentation (

) is the segmentation in which the fitting garment (c) is reflected, and thus, in one embodiment, the second segmentation (

), the clothes removal segmentation (Sa) in which the existing clothes are removed from the original image, and the pose (P) and the fitting clothes (c) are used as input data. The fitting clothes (c) are the second segmentation (

) is input to give clothing information to be reflected in the second segmentation (Sa)

) is used as a basic image for generation, and the pose P may be input to inform pose information such as which part of the fitting garment c is twisted at which angle.

)은 예컨대 도6에 도시한 이미지와 같이 생성되고, 제1 세그먼테이션(S)과의 비교에서 알 수 있듯이, 제2 세그먼테이션(

)의 상의(주황색) 영역은 기존 의상이 아니라 피팅 의상과 유사한 영역을 갖게 됨을 이해할 것이다. 또한 제2 세그먼테이션(

)에서도 예컨대 얼굴, 헤어, 상의, 하의, 팔, 배경 등이 각각 미리 지정된 클래스가 되고 제2 세그먼테이션(

)은 이 클래스의 개수만큼의 채널로 이루어진 이미지 데이터이며, 도6에서는 편의상 각 클래스를 각기 다른 컬러로 표시하여 하나의 이미지로 나타내었음을 이해할 것이다. Second segmentation (

) is generated, for example, as in the image shown in FIG. 6, and as can be seen from the comparison with the first segmentation S, the second segmentation (

), it will be understood that the area of the upper garment (orange) will have an area similar to the fitting garment, not the existing garment. Also, the second segmentation (

), for example, face, hair, top, bottom, arm, background, etc. are each a predefined class, and the second segmentation (

) is image data composed of as many channels as the number of this class, and it will be understood that each class is displayed in a different color for convenience in FIG. 6 and is represented as one image.

)의 차이와 관련하여, 의상제거 이미지(Ia)는 피팅 의상(c)이 크더라도 피팅 의상을 수용할 영역을 가능한 최대한 크게 만들기 위해, 원본 이미지에서 삭제하는 영역이 비교적 많은 반면, 의상제거 세그먼테이션(Sa)은 세그먼테이션부(200)에서 제2 세그먼테이션(

)을 만들기 위한 입력 데이터로 사용하기 위한 것이며 기존 의상만 제거하면 되므로 많은 영역을 제거하지 않아도 된다. 그리고 제2 세그먼테이션(

)은 피팅 의상(c)이 반영된 세그먼테이션이며 의상 변형부(300)와 이미지 합성부(400)에서 사용될 수 있다. Costume removal image (Ia), clothing removal segmentation (Sa), and second segmentation (

) in relation to the difference in ), the removal of clothing image (Ia) has relatively many areas to be deleted from the original image, in order to make the area to accommodate the fitting clothing as large as possible even if the fitting clothing (c) is large, whereas the removal of clothing segmentation ( Sa) is the second segmentation in the segmentation unit 200 (

) to be used as input data for making ), and there is no need to remove many areas because only the existing clothes need to be removed. and the second segmentation (

) is a segmentation in which the fitting garment (c) is reflected, and may be used in the clothes transforming unit 300 and the image synthesizing unit 400 .

8 is a view for explaining the operation of the clothes transforming unit 300 according to an embodiment, and FIG. 9 is a flowchart for explaining the operation of the clothes transforming unit 300 . The clothes transforming unit 300 is a functional unit that transforms the fitting clothes based on the pose of the person in the original image.

c) 및 피팅 의상(c)을 입력 데이터로 사용할 수 있다. 이 경우, 피팅 의상(c)은 와핑의 대상이 되는 데이터이고, 채널(

c)은 변형되는 의상이 차지하는 영역을 설정하기 위한 데이터이다. Referring to FIGS. 8 and 9 , first, in step S310, input data is input to a deep learning algorithm that performs clothing transformation to generate an output variable θ indicating the degree of transformation (warping) of the clothes. In one embodiment, the channel representing the costume (class) of the second segmentation (

c) and fitting clothes (c) can be used as input data. In this case, the fitting clothes (c) are data to be warped, and the channel (

c) is data for setting the area occupied by the clothes to be transformed.

c), 및 피팅 의상(c)을 딥러닝 알고리즘의 입력 데이터로서 사용할 수 있다. 이 경우 포즈(P)는 의상의 어느 부위가 어느 각도로 틀어져서 변형되는지를 알려주기 위한 데이터로 사용되고 의상제거 이미지(Ia)는, 의상에 가려지지 않아야 할 부분의 정보를 주기 위해 사용될 수 있다. 예를 들어, 치마나 바지 등의 하의 때문에 피팅 의상(상의) 아래쪽이 제거되어야 하는데, 이 경우 단순히 피팅 의상을 사람의 포즈에 따라 변형해야 할 뿐만 아니라 피팅 의상의 아래쪽 영역을 제거해야 하며, 따라서 이렇게 제거되어야 하는 부분까지 반영하기 위해 의상제거 이미지(Ia)가 사용된다. Alternatively, as shown in Fig. 8, a channel representing the costume (class) of the costume removal image (Ia), the pose (P), and the second segmentation (

c), and fitting clothes (c) can be used as input data of the deep learning algorithm. In this case, the pose P may be used as data for informing which part of the clothes is deformed by being distorted at what angle, and the clothes removal image Ia may be used to give information about parts that should not be covered by the clothes. For example, the bottom of the fitting outfit (top) needs to be removed because of the bottom of the skirt or pants. In order to reflect even the part to be removed, the clothing removal image Ia is used.

c)을 딥러닝 알고리즘의 인코더(310)에 입력하고 또한 피팅 의상(c)을 별도의 딥러닝 알고리즘의 인코더(310)에 입력 데이터로서 입력하여 각각 다단의 컨볼루션 연산 및 다운샘플링이 이루어지고 그 후 각각의 결과를 결합(Concat)하여 디코딩하면 출력변수(θ)를 얻을 수 있다. 여기서 출력변수(θ)는 의상의 변형(와핑: warping)의 정도를 정의하는 값이고, 출력변수(θ)와 피팅 의상(c)을 예컨대 공지의 와핑 알고리즘(320)에 입력하여 변형된 의상(W)을 출력할 수 있다. As above, the channel (Ia), the pose (P), and the channel (

c) is input to the encoder 310 of the deep learning algorithm, and the fitting costume (c) is input as input data to the encoder 310 of a separate deep learning algorithm, so that multi-stage convolution operation and downsampling are performed, respectively, and the After that, each result is combined (Concat) and decoded to obtain an output variable (θ). Here, the output variable (θ) is a value defining the degree of deformation (warping) of the clothes, and the output variable (θ) and the fitting clothes (c) are input to, for example, a known warping algorithm 320 to transform the clothes ( W) can be printed.

As described above, the deep learning algorithm composed of the encoder and the decoder can use a known machine learning algorithm, and can be implemented by appropriately learning with the learning data according to the embodiment of the present invention.

Fig. 10 schematically shows in a block diagram a garment transforming unit 300 according to an alternative embodiment.

The embodiment of Fig. 10 (a) uses the clothes removal image Ia and the fitting clothes c as input data. For example, an output variable θ is generated by inputting the clothes removal image Ia and the fitting clothes c to the encoder 310 of the deep learning algorithm, and warping the fitting clothes c based on the output variable θ Thus, the deformed fitting clothes W can be output. In another alternative embodiment, the original image I may be used instead of the costume removal image Ia.

The embodiment of FIG. 10( b ) may use the original image I, the channel Sc indicating the first segmentation clothes class, and the fitting clothes c as input data. For example, by inputting the original image (I) and the channel (Sc) representing the clothing class of the first segmentation and the fitting clothing (c) to the encoder 310 of the deep learning algorithm, respectively, an output variable (θ) is generated, The deformed fitting garment W may be output by warping the fitting garment c based on the output variable θ.

c), 및 피팅 의상(c)을 사용할 수 있다. 예를 들어, 원본 이미지(I)와 제2 세그먼테이션의 의상 클래스를 나타내는 채널(

c) 및 피팅 의상(c)을 각각 딥러닝 알고리즘의 인코더(310)에 입력하여 출력변수(θ)를 생성하고, 출력변수(θ)에 기초하여 피팅 의상(c)을 와핑하여 변형된 피팅 의상(W)을 출력할 수 있다. 도10(b)와 도10(c)의 또 다른 대안적 실시예에서, 원본 이미지(I) 대신 의상제거 이미지(Ia)를 사용할 수도 있다. 도11은 일 실시예에 따른 이미지 합성부(400)의 동작을 설명하는 도면이고 도12는 이미지 합성부(400)의 동작을 설명하는 흐름도이다. The embodiment of Fig. 10(c) shows an original image (I) as input data, and a channel (

c), and fitting clothes (c) can be used. For example, the channel representing the original image (I) and the clothing class of the second segmentation (

c) and the fitting garment (c) are input to the encoder 310 of the deep learning algorithm, respectively, to generate an output variable (θ), and based on the output variable (θ), the fitting garment (c) is warped to transform the fitting garment (W) can be printed. In another alternative embodiment of Figs. 10(b) and 10(c), the costume-removed image Ia may be used instead of the original image I. 11 is a diagram for explaining the operation of the image synthesizing unit 400 according to an exemplary embodiment, and FIG. 12 is a flowchart for explaining the operation of the image synthesizing unit 400 .

)의 의상 영역 중 피팅 의상(W)의 영역이 중첩되지 않는 비정렬 영역(Mmisalign)을 생성하고, 이 비정렬 영역(Mmisalign), 원본 이미지(I) 또는 원본 이미지를 이미지 처리한 이미지(예컨대 Ia), 및 변형된 피팅 의상(W)을 이미지 합성 알고리즘(410)에 입력 데이터로 입력하여 합성 이미지(

)를 생성할 수 있다. In one embodiment, the first segmentation (S) or the second segmentation (S)

), an unaligned area (Mmisalign) is created in which the area of the fitting garment (W) does not overlap, and the misaligned area (Mmisalign), the original image (I), or an image obtained by processing the original image (eg, Ia ), and the transformed fitting garment (W) as input data to the image synthesis algorithm 410 to obtain a composite image (

) can be created.

) 중 하나)의 의상 영역과 상기 변형된 피팅 의상(W) 영역이 중첩되는 정렬 영역(Malign)을 우선 생성하고, 상기 세그먼테이션의 의상 영역에서 상기 정렬 영역(Malign)을 제외하여 비정렬 영역(Mmisalign)을 생성할 수 있다. At this time, in order to create a misaligned region (Mmisalign), for example, segmentation (eg, the first segmentation (S) or the second segmentation (S))

) first) to create an alignment region (Malign) in which the garment region of one) and the deformed fitting garment (W) region overlap, and excluding the alignment region (Malign) from the segmentation garment region (Mmisalign) ) can be created.

)일 수 있으며, 바람직하게는 제2 세그먼테이션(

)을 사용한다. 이 경우 도12의 단계(S410)와 같이, 제2 세그먼테이션(

)의 의상 영역의 채널(

c) 중 변형된 의상(W)이 커버하지 못하는 비정렬 영역(Mmisalign) 생성하고, 그 후 단계(S420)에서, 제1 데이터 세트(Ia, P, W) 및 제2 데이터 세트(

, Mmisalign)를 입력 데이터로서 이미지 합성 알고리즘에 입력하여 합성 이미지(

)를 생성할 수 있다. The segmentation used at this time is the first segmentation (S) or the second segmentation (

), preferably the second segmentation (

) is used. In this case, as in step S410 of FIG. 12 , the second segmentation (

) of the clothing area's channel (

c) A misaligned region (Mmisalign) that is not covered by the deformed garment W is generated, and then, in step S420, the first data set (Ia, P, W) and the second data set (

, Mmisalign) as input data to the image synthesis algorithm,

) can be created.

)의 의상 영역의 채널(

c)과 변형된 피팅 의상(W) 영역이 중첩되는 부분을 정렬 영역(Malign)으로서 우선 생성하고, 제2 세그먼테이션(

)의 의상 영역 채널(

c)에서 정렬 영역(Malign)을 제외한 영역을 비정렬 영역(Mmisalign)으로 정의할 수 있다. At this time, the misaligned region Mmisalign may be created as shown in FIG. 13(a). That is, the second segmentation (

) of the clothing area's channel (

c) and the area where the deformed fitting garment (W) overlaps are first created as an alignment area (Malign), and the second segmentation (

) of the clothing area channel (

In c), an area excluding the alignment area Malign may be defined as a misalignment area Mmisalign.

) 및 비정렬 영역(Mmisalign)을 포함할 수 있다. 여기서 제1 데이터 세트는 가상 피팅의 결과물로서 출력하기 위해 합성해야 할 이미지를 제공한다. 즉 기본적으로 원본 이미지(I) 및 변형된 피팅 의상(W)을 포함할 수 있으나, 바람직하게는, 원본 이미지(I)에서 기존 의상(c)을 제거한 의상제거 이미지(Ia) 및 변형된 피팅 의상(W)을 입력 데이터로 할 수 있다. 이 때 의상제거 이미지(Ia)는 합성 이미지에서도 유지해야 할 영역을 지정하는 역할을 할 수 있다. 또한 본 발명의 일 실시예에서, 포즈(P)에 관한 정보도 제1 데이터 세트에 포함될 수 있다. 이 때 포즈(P) 정보는 피팅 의상을 원본 이미지에 합성시 피팅 의상의 텍스쳐를 생성할 때 포즈(P)를 참고하여 생성하도록 입력하는 것으로, 예컨대 이미지 합성 알고리즘이 새로 생성해야 할 부분, 즉 비정렬 영역(Mmisalign)에 텍스쳐를 만들 때 포즈(P)를 참고하여 생성할 할 수 있다. On the other hand, the first data set input to the image synthesis algorithm 410 in step S420 includes the clothes removed image (Ia), the pose (P), and the deformed fitting clothes (W), and the second data set is the second data set 2 segmentation (

) and a misaligned region (Mmisalign). Here, the first data set provides an image to be synthesized to output as a result of virtual fitting. That is, it may basically include the original image (I) and the deformed fitting clothes (W), but preferably, the clothes removal image (Ia) and the modified fitting clothes (Ia) in which the existing clothes (c) are removed from the original image (I) (W) can be used as input data. In this case, the clothes removal image Ia may serve to designate an area to be maintained even in the composite image. In addition, in an embodiment of the present invention, information about the pose P may also be included in the first data set. At this time, the pose (P) information is input so that when the texture of the fitting clothes is generated when the fitting clothes are synthesized with the original image, they are generated by referring to the pose (P). When creating a texture in the alignment area (Mmisalign), it can be created by referring to the pose (P).

) 및 비정렬 영역(Mmisalign)을 포함한다. The second data set input to the image synthesizing algorithm 410 in step S420 serves to provide information on the region to be maintained and the region to be newly created. In one embodiment, the second data set is a second segmentation (

) and a misaligned region (Mmisalign).

) 데이터를 분리되어 입력할 수 있다. 즉 제2 세그먼테이션(

)을 피팅 의상 영역(

c)과 이 영역(

c)을 제외한 나머지 영역(

_c)으로 분리하고 이렇게 분리된 제2 세그먼테이션(

) 데이터와 비정렬 영역(Mmisalign)을 이미지 합성 알고리즘(410)에 입력할 수 있다. 제2 세그먼테이션(

)을 피팅 의상 영역(

c)과 나머지 영역(

_c)으로 분리하여 입력하는 것은, 이미지 합성 알고리즘(410)이 이렇게 입력 데이터를 개별적으로 입력받는 경우 피팅 의상을 합성할 영역을 보다 정확히 인식할 수 있기 때문이다. 그러나 대안적 실시예에서는 제2 세그먼테이션(

)을 위와 같이 분리하지 않고 하나의 데이터로 입력할 수도 있음은 물론이다. In one embodiment, when the second data set is input to the image synthesis algorithm, the second segmentation (

) data can be entered separately. That is, the second segmentation (

) to the fitting garment area (

c) and this area (

c) except for the area (

_c) and the second segmentation (

) data and the misaligned region (Mmisalign) may be input to the image synthesis algorithm 410 . Second segmentation (

) to the fitting garment area (

c) and the rest of the area (

The reason for inputting separately as _c) is that when the image synthesis algorithm 410 receives input data individually in this way, it is possible to more accurately recognize a region for synthesizing fitting clothes. However, in an alternative embodiment, the second segmentation (

Of course, it is also possible to input as one data without separating the ) as above.

Meanwhile, as shown in FIGS. 14 and 15 ( a ), in an embodiment, the image synthesis algorithm may be implemented as a decoder in which a plurality of convolution operation blocks are connected in multiple stages. In the embodiment of Fig. 14, the image synthesis algorithm is configured as a decoder in which a plurality of residual blocks (ResBlk) 415 including convolution and upsampling operations are connected in multiple stages, in this case, the first data set and the second data set are Each residual block of the plurality of residual blocks is input as input data, respectively.

The residual block (ResBlk) algorithm is a technique known as a technique for lowering the difficulty of optimization when learning a deep learning algorithm. That is, since it is difficult to directly learn the mapping result (eg, H) inherent in the actual mapping, the difference between the input value (x) and the result value (H), that is, (F: F(x)=H(x)- This is a technique for learning x) by looking at it as the residual part of learning, so it is expressed as a block diagram in which the input (x) is input to the convolution stage, and at the same time the input is branched and connected directly to the output stage.

In an embodiment of the present invention, such a residual block (ResBlk) technique is employed, and each residual block is composed of an alignment recognition residual block (ALIAS ResBlk) 415 as shown in FIG. 15(b), and each The alignment recognition residual block 415 has a structure in which the input (hi) is inputted to a path that passes through the alignment recognition normalization stage 417 twice and a path that passes once, and then sums the results to output (hi+1). . In this case, the first data set may be input to the input terminal of each alignment recognition residual block 415 , and the second data set may be input to each alignment recognition normalization 417 .

In addition, in one embodiment, each alignment-aware normalization 417 is applied to the present invention by adopting a batch normalization (Batch Norm) structure of a known SPADE (Spatially-Adaptive Denormalization) algorithm, for example, the structure shown in FIG. can have In general, batch normalization is a technique for improving the performance of the algorithm by normalizing the output for each layer of the deep learning algorithm so that the output value maintains non-linearity, and as shown in FIG. It calculates γ (scaling parameter) and β (shift parameter) according to the result of multiple convolution operations, multiplies these values with the output values of each layer, adds them, and outputs them to the next layer.

c)과 나머지 영역(

_c) 및 비정렬 영역(Mmisalign)을 γ와 β를 생성하기 위한 컨볼루션 연산의 입력 데이터로 입력하고 γ와 β의 연산 앞단에 비정렬 영역(Mmisalign)을 표준화하여 입력할 수 있다. In an embodiment of the present invention, the alignment-aware normalization 417 inputs the second data set as input data to the alignment-aware normalization 417 block. That is, as shown in Fig. 16, the fitting clothing area (

c) and the rest of the area (

_c) and the misaligned region (Mmisalign) can be input as input data of a convolution operation for generating γ and β, and the misaligned region (Mmisalign) can be standardized and input before the operation of γ and β.

Of course, in an alternative embodiment, some data of the first data set and the second data set input to the image synthesizing unit 400 may be changed or omitted. For example, Fig. 17 is a block diagram schematically illustrating the operation of the image synthesizing unit 400 according to an exemplary embodiment.

) 대신 제1 세그먼테이션(S)을 사용하는 실시예를 나타내었다. 도17(b)는 제1 데이터 세트에서 포즈(P)를 생략하고 변형된 피팅의상(W) 대신 피팅의상(c)의 이미지를 사용하여 이미지를 합성하는 실시예를 나타내었다. 도17(c)는 제1 데이터 세트에서 의상제거 이미지(Ia) 대신 원본 이미지(I)를 사용하고 포즈(P)를 생략한 경우의 실시예를 나타내었다. 이와 같이 구체적 실시 형태에 따라 이미지 합성부(400)에 입력하는 제1 데이터 세트와 제2 데이터 세트의 일부 데이터를 다양한 조합으로 변경하거나 생략하여 이미지를 합성할 수 있음을 이해할 것이다. Fig. 17(a) omits the pause P in the first data set and shows the second segmentation (P) in the second data set.

) instead of the first segmentation (S) is shown. Fig. 17(b) shows an example in which the pose (P) is omitted from the first data set and images are synthesized using the image of the fitting outfit (c) instead of the modified fitting outfit (W). Fig. 17(c) shows an embodiment in which the original image I is used instead of the costume removal image Ia in the first data set and the pose P is omitted. As described above, it will be understood that, according to a specific embodiment, an image may be synthesized by changing or omitting some data of the first data set and the second data set input to the image synthesizing unit 400 in various combinations.

18 is a view for explaining the effects of the present invention according to an embodiment.

In general, as the resolution of the synthetic image on which the virtual fitting is completed increases, a lot of awkward parts are seen between the human area of the original image and the area of the deformed fitting clothing. By inputting as one of the input data of the composite image, the image synthesis algorithm 410 recognizes the unaligned area and creates and fills the unaligned area with a texture, thereby realizing a natural fitting outfit.

Figure 18 (a) is a composite image according to the prior art. The red area marked on the shoulder of a person corresponds to the misalignment area (Mmisalign). In the conventional fitting technology, the image synthesis algorithm does not properly process this portion. As a result, the texture was not properly created and an unnatural output was displayed.

On the other hand, FIG. 18( b ) is a composite image according to an embodiment of the present invention. The image synthesis algorithm 410 recognizes a red misaligned area (Mmisalign) and creates and fills the texture in this area. It can be seen that a more natural output result was displayed compared to that.

As described above, those of ordinary skill in the art to which the present invention pertains can understand that various modifications and variations are possible from the description of this specification. Therefore, the scope of the present invention should not be limited to the described embodiments and should be defined by the claims described below as well as the claims and equivalents.

10: computer device
100: preprocessor
200: segmentation unit
300: costume transformation part
400: image synthesizing unit

Claims (9)

A virtual fitting method for synthesizing fitting clothes with an original image using a computer device, comprising:
generating a segmentation from the original image;
generating a deformed fitting garment by deforming the fitting garment (c) based on the original image;
generating a misalignment region (Mmisalign) in which regions of the deformed fitting clothes do not overlap among the segmentation regions of the existing clothes; and
and generating a composite image by inputting input data including the original image and fitting clothes into an image synthesis algorithm. The method according to claim 1,
In the generating of the unaligned region, an alignment region (Malign) in which the segmentation garment region and the deformed fitting clothing region overlap is created, and the unaligned region is excluded from the segmentation garment region A virtual fitting method using a computer device, characterized in that it generates a. The method according to claim 2,
The segmentation is a first segmentation (S) in which the original image is segmented or a second segmentation (S) in which fitting clothes are included as a class instead of existing clothes in the first segmentation (

), characterized in that, a virtual fitting method using a computer device. The method of claim 3, wherein the computer device comprises:
In the first segmentation, the clothes removal segment (Sa) is generated by removing the existing clothes region, and the second segmentation (Sa) is performed using data including the clothes removal segmentation (Sa) and the fitting clothes.

), a virtual fitting method using a computer device, characterized in that it generates. The method of claim 1,
In the step of generating the composite image, a first data set including the original image or a costume removal image in which the existing clothes are removed from the original image, and the deformed fitting clothes, and a second data set including the unaligned region A virtual fitting method using a computer device, characterized in that a data set is used as input data of the image synthesis algorithm. The method according to claim 5,
Extracting pose information (P) including feature points of the body from the original image, and removing a predetermined region from the original image based on the pose information (P) to generate the clothes removal image , a virtual fitting method using a computer device. The method according to claim 5,
Extract the pose information (P) including the feature points of the body from the original image, remove a predetermined region from the original image based on the pose information (P), and maintain it in the composite image among the segments of the segmentation class A virtual fitting method using a computer device, characterized in that the clothes removal image is generated by overlapping images of a preset class to be performed. The method according to claim 5,
The image synthesis algorithm consists of a decoder in which a plurality of residual blocks (ResBlk) including convolution and upsampling operations are connected in multiple stages, and the first data set and the second data set are each residual block of the plurality of residual blocks. A virtual fitting method using a computer device, characterized in that each is input as input data to the . A computer-readable recording medium in which a computer program for executing the virtual fitting method according to any one of claims 1 to 8 is recorded.

Images