JP7497059B2 - IMAGE PROCESSING DEVICE, TRAINED MODEL, IMAGE COLLECTION DEVICE, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on U.S. Provisional Application No. 62/809,088, filed February 22, 2019, the contents of which are incorporated herein by reference.

The present invention relates to an image processing device, a trained model, an image collection device, an image processing method, and an image processing program.

Conventionally, a technique is known in which an image of clothing is generated in accordance with the movement of the human body, and the generated image of clothing is superimposed on an image of the human body to allow the user to virtually try on clothing (see, for example, Non-Patent Document 1). In recent years, a technique has also been proposed that uses deep learning to allow the user to virtually try on clothing (see, for example, Non-Patent Document 2).

Kim, J., & Forsythe, S. "Adoption of Virtual Try-on technology for online apparel shopping." Journal of Interactive Marketing, Volume 22, Spring 2008, Pages 45-59. Christoph Lassner, Gerard Pons-Moll, Peter V. Gehler.The IEEE International Conference on Computer Vision (ICCV), July 2017, Pages 853-862.

However, conventional technology had the problem that the computational load became excessive when running a simulation using a large number of parameters set for each target garment.

Note that these issues are not limited to virtually trying on clothing, but are also common when virtually trying on other accessories such as glasses or shoes.

Therefore, the present invention provides an image processing device, a trained model, an image collection device, an image processing method, and an image processing program that can reduce the computational load when trying on virtual clothing items.

An image processing device according to one aspect of the present invention includes an image generating unit that inputs an image of a user wearing the reference wearing item into a trained model trained by machine learning using learning data in which an image of a reference wearing item worn by a wearer in a predetermined posture is associated as input data with an image of a tried-on wearing item worn by the wearer in a posture common to the predetermined posture, and generates an image of the user trying on the tried-on wearing item.

According to this aspect, a trained model is trained using learning data that associates images of a reference wearing item with images of a tried-on wearing item, and an image of a user wearing the reference wearing item is input to this trained model, thereby reducing the computational load when trying on a virtual wearing item.

In the above aspect, the image generation unit may extract an image of the reference wearing item and an image of the user's body part from an image of the user wearing the reference wearing item, input the extracted image of the reference wearing item to the trained model, and generate an image of the user trying on the try-on wearing item by combining the image of the try-on wearing item output from the trained model with the extracted image of the user's body part.

According to this aspect, an image of a user trying on a try-on garment can be accurately synthesized.

In the above aspect, the image generating unit may extract depth information of the image of the reference wearing item and depth information of the image of the user's body part from an image of the user wearing the reference wearing item, and synthesize the image of the tried-on wearing item and the image of the user's body part based on the extracted depth information.

According to this aspect, by using image depth information, an image of a user wearing the tried-on garment can be synthesized more accurately.

In the above aspect, the learning data may be such that an image of the reference wearing item worn by the wearing subject in a predetermined posture is used as the input data, an image of the try-on wearing item of a first variation worn by the wearing subject in a posture common to the predetermined posture is associated as the output data, and an image of the try-on wearing item of a second variation worn by the wearing subject in a posture common to the predetermined posture is associated as the output data with the input data, and the image generation unit may accept a selection of a variation of the try-on wearing item, input an image of a user wearing the reference wearing item to the trained model, and generate an image of a user trying on the try-on wearing item of the selected variation.

According to this aspect, an image of a user trying on the try-on garment can be generated, distinguishing between variations of the try-on garment selected by the user.

In the above aspect, the wearable item may be clothing, and the learning data may include data in which an image of the reference clothing worn by the wearer in a specified posture is used as the input data, and an image of the try-on clothing worn by the wearer in a posture common to the specified posture is used as the output data.

According to this aspect, the computational load when trying on virtual clothing can be reduced by training a trained model using training data that matches images of reference clothing with images of tried-on clothing.

The trained model according to one embodiment of the present invention is trained by machine learning using training data in which an image of a reference wearable item worn by the wearer in a specified posture is used as input data, and an image of a try-on wearable item worn by the wearer in a posture common to the specified posture is used as output data.

According to this aspect, the same effect as the image processing device invention described above can be obtained.

An image collection device according to one embodiment of the present invention comprises a robot mannequin, a camera, a robot control unit which controls the posture of the robot mannequin, and an image capture control unit which captures an image of an accessory with the camera while controlling the posture of the robot mannequin wearing the accessory to a predetermined posture based on the control of the robot control unit, and stores the captured image in a memory unit in association with the posture.

According to this aspect, it is possible to accurately match images of reference wear items with images of try-on wear items, thereby improving the reliability of the learning data used to train the trained model.

An image processing method according to one aspect of the present invention includes an image generation step of inputting an image of a user wearing the reference wearing item into a trained model trained by machine learning using training data in which an image of a reference wearing item worn by a wearer in a predetermined posture is used as input data, and an image of a user trying on the tried-on wearing item is generated by inputting an image of the user wearing the reference wearing item into a trained model trained by machine learning using training data associated with an image of a tried-on wearing item worn by a wearer in a posture common to the predetermined posture.

According to this aspect, the same effect as the image processing device invention described above can be obtained.

An image processing program according to one embodiment of the present invention causes a computer to execute a process of inputting an image of a user wearing the reference wearable item into a trained model trained by machine learning using learning data in which an image of a reference wearable item worn by a wearer in a predetermined posture is used as input data, and an image of a tried-on wearable item worn by the wearer in a posture common to the predetermined posture is used as output data. The trained model is trained by machine learning using learning data in which an image of a user wearing the reference wearable item is used as input data, and an image of a user trying on the tried-on wearable item is generated.

According to this aspect, the same effect as the above-mentioned image processing program invention can be obtained.

According to the present invention, it is possible to reduce the computational load when trying on virtual clothing items.

1 is a block diagram showing a schematic configuration of a first embodiment of an image processing apparatus; FIG. 4 is a diagram showing an example of data content of image data. FIG. 4 is a diagram showing an example of data content of learning data. FIG. 13 is a diagram showing an example of association between images of measurement garments and images of try-on garments. 1A to 1C are diagrams for explaining an example of an image generation process. 13 is a flowchart showing an example of a learning process of a trained model. 11 is a flowchart showing the contents of an image generating process. FIG. 13 is a diagram showing an example of association between images of measurement garments and images of try-on garments. 1A to 1C are diagrams for explaining an example of an image generation process. FIG. 13 is a diagram showing an example of association between images of measurement garments and images of try-on garments. 1A to 1C are diagrams for explaining an example of an image generation process. FIG. 2 illustrates an example of a hardware configuration of an image processing apparatus.

(First embodiment)
Hereinafter, with reference to the drawings, embodiments of an image processing device, a trained model, an image collection device, an image processing method, and an image processing program of the present invention will be described.

As shown in FIG. 1, the image collection device 1 includes, for example, a camera 10, a robot mannequin 20, a display 30, and an image processing device 100. These devices and equipment are connected to each other via communication lines, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configuration may be added.

The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is, for example, fixedly disposed in front of the robot mannequin 20, and captures an image of the robot mannequin 20 wearing clothing, which is an example of an accessory. The camera 10 may be, for example, a stereo camera or a depth camera.

The robot mannequin 20 is, for example, a humanoid robot. The robot mannequin 20 is configured to be drivable with, for example, four degrees of freedom in the shoulders, four degrees of freedom in the elbows, and two degrees of freedom in the waist rotation, and can assume various postures while wearing clothing.

The display 30 is, for example, an LCD display and displays the image generated by the image processing device 100.

The image processing device 100 includes, for example, a control unit 110 and a storage unit 120. The control unit 110 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in a storage device such as an HDD or flash memory of the image processing device 100 in advance, or may be stored in a removable storage medium such as a DVD or CD-ROM, and may be installed in the HDD or flash memory of the image processing device 100 by attaching the storage medium to a drive device.

The control unit 110 includes, for example, a robot control unit 112, a photography control unit 114, a learning unit 116, and an image generation unit 118.

The robot control unit 112 controls the movement of the robot mannequin 20. The robot control unit 112 controls the posture of the robot mannequin 20, for example, by controlling the amount of movement in each degree of freedom of the robot mannequin 20. Each degree of freedom of the robot mannequin 20 includes the degree of freedom of the pose of the robot mannequin 20, such as the degree of freedom of both shoulders, the degree of freedom of both elbows, and the degree of freedom of rotation of the waist, as well as the degree of freedom of the rotation position of the robot mannequin 20 around a central axis extending in the vertical direction. The degree of freedom of the robot mannequin 20 may include, for example, the angle of the robot mannequin 20 relative to the shooting position and direction of the camera 10. Note that the above description is merely an example of the posture of the robot mannequin 20, and is not limited thereto.

The photographing control unit 114 controls the photographing operation of the camera 10. For example, the photographing control unit 114 photographs an image of the robot mannequin 20 while controlling the posture of the robot mannequin 20 to a predetermined posture based on the control of the robot control unit 112. For example, the photographing control unit 114 photographs an image of the robot mannequin 20 for various poses of the robot mannequin 20 while changing the rotational position of the robot mannequin 20 based on the control of the robot control unit 112. The photographing control unit 114 associates the photographed image with the posture of the robot mannequin 20 and stores it in the storage unit 120. For example, the photographing control unit 114 photographs an image of the robot mannequin 20 wearing a measurement garment and an image of the robot mannequin 20 wearing a try-on garment. The photographing control unit 114 divides the image of the measurement garment from the image of the robot mannequin 20 wearing the measurement garment, and stores the divided image in the storage unit 120 as image data 122 in association with the posture of the robot mannequin 20 at the time of photographing the image. In addition, the photography control unit 114 divides the image of the try-on clothing into regions from the image of the robot mannequin 20 wearing the clothing, and stores the divided image in the memory unit 120 as image data 122 in association with the posture of the robot mannequin 20 at the time the image was photographed.

2 is a diagram showing an example of the data contents of the image data 122. In the illustrated example, the data attributes of the image data 122 include, for example, the type of image, image information, and the posture of the robot mannequin 20. The type of image is the type of clothing worn by the robot mannequin 20, and includes, for example, measurement clothing and try-on clothing. The measurement clothing is an example of reference clothing that is referenced when synthesizing the try-on clothing. The measurement clothing is, for example, clothing for obtaining information about the user's body, and is preferably easily distinguishable from the color of the human skin and easily associated with various try-on clothing. The try-on clothing is clothing that is the subject of virtual try-on, and includes, for example, multiple types of clothing with different designs. The image information includes information on the luminance values (R, G, B) for each image position of the image captured by the camera 10. The image information may further include depth information for each image position of the image captured by the camera 10. The posture of the robot mannequin 20 is a parameter defined by a combination of the movement amounts of each degree of freedom of the robot mannequin 20.

The learning unit 116 uses an image of the measurement garment as input data, and stores in the storage unit 120 learning data 124 in which the image of the measurement garment and the image of the try-on garment in which the posture of the robot mannequin 20 is the same are associated as output data. That is, the learning unit 116 associates an image of the measurement garment and an image of the try-on garment taken in the same pose from the same direction, and stores them in the storage unit 120 as learning data 124. For example, the learning unit 116 refers to the image data 122 stored in the storage unit 120, associates the image of the measurement garment with the image of the try-on garment, and stores the learning data 124 in the storage unit 120. The learning unit 116 trains a learning model by machine learning using the learning data 124 to generate a trained model 126. The trained model 126 is, for example, composed of a neural network, which is a type of machine learning model. The learning unit 116 may perform so-called data augmentation, which inflate the number of data in the learning data 124, by performing processes such as translation, enlargement/reduction, rotation, and noise addition on the images of the measurement clothing and the images of the try-on clothing.

Figure 3 is a diagram showing an example of the data contents of the learning data 124. In the illustrated example, the learning data 124 includes a plurality of (N in the illustrated example) pieces of image information "A1" to "AN" of measurement garments in which the robot mannequin 20 has different postures. The learning data 124 also associates each piece of image information "A1" to "AN" of the measurement garments with image information of garments to be tried on that have a common posture on the robot mannequin 20. Specifically, the learning data 124 associates each piece of image information "A1" to "AN" of the measurement garments with image information for each garment (such as "Garment 1" and "Garment 2") that have a common posture on the robot mannequin 20.

Figure 4 is a diagram showing an example of the correspondence between images of measurement clothing and images of try-on clothing. As shown in Figure 4, the learning data 124 corresponds image information of the measurement clothing and image information of the try-on clothing, distinguishing between different postures of the robot mannequin 20. In the illustrated example, the learning data 124 corresponds, for example, image information "A1" of the measurement clothing to image information "B11" of the try-on clothing when the posture of the robot mannequin 20 is "posture 1", and image information "A2" of the measurement clothing to image information "B12" of the try-on clothing when the posture of the robot mannequin 20 is "posture 2".

The image generation unit 118 inputs an image of a user wearing the measurement garment into the trained model 126 to generate an image of the user trying on the try-on garment. For example, the image generation unit 118 extracts an image of the measurement garment and an image of the user's body part from the image of the user wearing the measurement garment, inputs the extracted image of the measurement garment into the trained model 126, and synthesizes the image of the try-on garment output from the trained model 126 with the extracted image of the user's body part.

Figure 5 is a diagram illustrating an example of an image generation process by the image processing device 100.

As shown in the figure, in the learning phase, the image processing device 100 first divides an image of the measurement garment into regions from an image of the robot mannequin 20 wearing the measurement garment, and also divides an image of the try-on garment into regions from an image of the robot mannequin 20 wearing the try-on garment.

Next, the image processing device 100 matches the region-divided image of the measurement clothing with the image of the try-on clothing as training data 124, and generates a trained model 126 by training a training model through machine learning using this training data 124.

Next, in the execution phase, the image processing device 100 divides the image of the user wearing the measurement garment into an image of the measurement garment and an image of the user's body parts. In this case, the image processing device 100 may acquire depth information of the image of the measurement garment and depth information of the image of the user's body parts. This allows accurate division of the image of the measurement garment and the image of the user's body parts. Then, the image processing device 100 inputs the divided image of the measurement garment into the trained model 126. As a result, an image of the try-on garment corresponding to the user's posture is output from the trained model 126.

Next, the image processing device 100 generates an image of the user wearing the try-on clothes by synthesizing the image of the try-on clothes output from the trained model 126 with the image of the user's body parts divided into regions as described above. The image processing device 100 synthesizes the image of the user wearing the try-on clothes so that, for example, the image of the user's body parts is positioned in front of the image of the measurement clothes. The image processing device 100 may generate an image of the user wearing the try-on clothes based on, for example, depth information of the image of the measurement clothes and depth information of the image of the user's body parts. In this case, the image processing device 100 can accurately synthesize an image of the user wearing the try-on clothes, for example, by synthesizing the user's face or neck on the neck part of the clothes or the user's arms on the sleeves of the clothes.

Figure 6 is a flowchart showing an example of a learning process of the trained model 126. It is assumed that, prior to the processing of the flowchart shown in Figure 6, images of the measurement garment for each posture of the robot mannequin 20 have been taken. The processing of the flowchart shown in Figure 6 is executed, for example, when a predetermined operation is triggered when the try-on garment is put on the robot mannequin 20.

As shown in FIG. 6, first, the robot control unit 112 controls the robot mannequin 20 wearing the try-on clothes to a predetermined posture (step S10). Next, the photography control unit 114 uses the camera 10 to take an image of the robot mannequin 20 (step S12). Next, the photography control unit 114 divides the image of the try-on clothes from the image of the robot mannequin 20 into regions (step S14). Next, the photography control unit 114 associates the image of the measurement clothes and the image of the try-on clothes, which have a common posture of the robot mannequin 20, and stores them in the memory unit 120 as learning data 124 (step S16). Next, the learning unit 116 uses the learning data 124 stored in the memory unit 120, as well as data augmented by performing data augmentation on the learning data 124, to train a learning model by machine learning (step S18). Next, the photography control unit 114 determines whether learning has been completed (step S20). Then, when the photography control unit 114 determines that the learning has not been completed, the robot control unit 112 changes the posture of the robot mannequin 20 (step S22), and repeats the processes of steps S12 to S20 until the correspondence between the image of the measurement garment and the image of the try-on garment is completed. On the other hand, when the photography control unit 114 determines that the learning has been completed, the process of this flowchart ends.

Figure 7 is a flowchart showing an example of an image generation process. It is assumed that the user has selected the clothes to be tried on beforehand, prior to the processing of the flowchart shown in Figure 7. The processing of the flowchart shown in Figure 7 is executed, for example, when an image of the user wearing the measurement garment is captured, triggered by a specified operation.

As shown in FIG. 7, first, the image generating unit 118 divides an image of the user wearing the measurement garment into an image of the user's body part and an image of the measurement garment (step S30). Next, the image generating unit 118 inputs the image of the measurement garment divided into regions in the previous step S30 into the trained model 126 (step S32). Next, the image generating unit 118 synthesizes the image of the try-on garment output from the trained model 126 with the image of the user's body part divided into regions in the previous step S30 (step S34). Next, the image generating unit 118 outputs the combined image to the display 30 (step S36). This ends the processing of this flowchart.

As described above, according to the first embodiment, the following effects can be obtained.
(1) The image processing device 100 learns the trained model 126 by machine learning using the training data 124 in which an image of the measurement garment worn by the robot mannequin 20 in a predetermined posture is used as input data and an image of the try-on garment worn by the robot mannequin 20 in a posture common to the predetermined posture is used as output data. The image processing device 100 also inputs an image of a user wearing the measurement garment to the trained model 126 to generate an image of the user trying on the try-on garment. That is, the image of the measurement garment worn by the user is converted into an image of the try-on garment using the trained model 126, and an image of the user trying on the try-on garment is generated using the converted image. This makes it unnecessary to prepare three-dimensional models of the human body and the garment and perform a physical simulation, thereby reducing the calculation load when trying on virtual garments. In addition, it is unnecessary to accurately measure a large number of parameters for each target garment, thereby improving the convenience when trying on virtual garments.

(2) The image processing device 100 extracts an image of the measurement garment and an image of the user's body part from an image of the user wearing the measurement garment, and inputs the extracted image of the measurement garment to the trained model 126. The image processing device 100 also generates an image of the user trying on the try-on garment by synthesizing the image of the try-on garment output from the trained model 126 with the image of the user's body part previously extracted. In other words, by aligning and synthesizing the image of the try-on garment converted using the trained model 126 with the image of the user's body part, it is possible to accurately synthesize an image of the user trying on the try-on garment.

(3) The image processing device 100 extracts depth information of the image of the measurement garment and depth information of the image of the user's body part from an image of the user wearing the measurement garment, and synthesizes the image of the try-on garment and the image of the user's body part based on the extracted depth information. This makes it possible to more accurately synthesize an image of the user trying on the try-on garment.

(4) The image processing device 100 captures an image of the clothing with the camera 10 while controlling the posture of the robot mannequin 20 wearing the clothing to a predetermined posture, and stores the captured image of the clothing in the memory unit 120 in association with the posture of the robot mannequin 20. This makes it possible to accurately match the image of the measurement clothing with the image of the try-on clothing, thereby improving the reliability of the learning data 124.

(5) The image processing device 100 configures the learning data 124 to include images of the robot mannequin 20 viewed from various rotational positions, such as an image of the robot mannequin 20 viewed from the side, in addition to an image of the robot mannequin 20 viewed from the front. This makes it possible to accurately synthesize an image of a user trying on the try-on garment, even when the user is in various postures.

(6) The image processing device 100 constructs the learning data 124 by associating images of the measurement garments with images of the try-on garments. This makes it possible to accurately predict deformation of the try-on garments that accompanies changes in the user's posture, and more accurately synthesize an image of the user wearing the try-on garments.

Second Embodiment
Next, a second embodiment of the image processing device will be described with reference to the drawings. The second embodiment differs from the first embodiment in the method of associating images of measurement garments with images of try-on garments. Therefore, in the following description, the configuration different from the first embodiment will be mainly described, and a duplicate description of the same or corresponding configuration as the first embodiment will be omitted.

The learning unit 116 according to the second embodiment stores in the memory unit 120 learning data 124 in which the image of the measurement garment is used as input data and the image of the measurement garment is associated with multiple variations of try-on garments in which the posture of the robot mannequin 20 is common as output data. The multiple variations include, for example, the color and size of the try-on garment. For example, the learning data 124 uses an image of the measurement garment as input data and associates with an image of a first variation of try-on garments in which the posture of the measurement garment is common to the robot mannequin 20 as output data, and also uses an image of the measurement garment as input data and associates with an image of a second variation of try-on garments in which the posture of the measurement garment is common to the robot mannequin 20 as output data.

8 is a diagram showing an example of the correspondence between the image of the measurement garment and the image of the try-on garment. As shown in FIG. 8, the learning data 124 corresponds image information of the measurement garment and image information of the try-on garment for each posture of the robot mannequin 20. In the illustrated example, the learning data 124 corresponds, for example, image information "A1" of the measurement garment in which the posture of the robot mannequin 20 is "posture 1" with image information "B11α" of the S-size try-on garment. Also, image information "A1" of the measurement garment in which the posture of the robot mannequin 20 is "posture 1" with image information "B11β" of the M-size try-on garment. Also, image information "A1" of the measurement garment in which the posture of the robot mannequin 20 is "posture 1" with image information "B11γ" of the L-size try-on garment.

Figure 9 is a diagram illustrating an example of an image generation process by the image processing device 100.

As shown in the figure, in the execution phase, the image processing device 100 divides an image of a user wearing the measurement garment into an image of the measurement garment and an image of the user's body parts. The image processing device 100 then inputs the divided image of the measurement garment into the trained model 126. In this case, the size of the try-on garment output from the trained model 126 ("M size" in the illustrated example) is selected in advance. As a result, an image of the try-on garment corresponding to the pre-selected size is output from the trained model 126. The image processing device 100 then generates an image of the user wearing the try-on garment by combining the image of the try-on garment output from the trained model 126 with the image of the user's body parts divided as described above.

As described above, according to the second embodiment, in addition to the effects (1) to (6) of the first embodiment, the following effects can be obtained.
(7) The image processing device 100 receives a selection of a variation of the try-on clothing, inputs an image of the user wearing the measurement clothing to the trained model 126, and generates an image of the user trying on the try-on clothing of the previously selected variation. This makes it possible to generate images of the user trying on the try-on clothing, distinguishing between the variations of the try-on clothing selected by the user.

Third Embodiment
Next, a third embodiment of the image processing device will be described with reference to the drawings. The third embodiment differs from the first embodiment in the method of associating images of measurement garments with images of try-on garments. Therefore, in the following description, the configuration different from the first embodiment will be mainly described, and a duplicate description of the same or corresponding configuration as the first embodiment will be omitted.

The learning unit 116 according to the third embodiment uses an image of a measurement garment as input data, and stores in the storage unit 120 learning data 124 in which the image of the measurement garment and the image of a try-on garment that has the same body type and posture as the robot mannequin 20 are associated as output data. The body type of the robot mannequin 20 is defined by parameters such as arm circumference, shoulder width, and waist circumference. The learning data 124 is configured by associating, for example, an image of a measurement garment worn by the robot mannequin 20 that corresponds to a normal body type with an image of a try-on garment that has the same body type and posture as the measurement garment and the robot mannequin 20, and by associating an image of a measurement garment worn by the robot mannequin 20 that corresponds to an obese type with an image of a try-on garment that has the same body type and posture as the measurement garment and the robot mannequin 20. That is, the learning data 124 associates the image of the measurement garment with the image of the try-on garment so as to correspond to users of a plurality of body types, and the number of data in the learning data 124 is inflated.

10 is a diagram showing an example of the correspondence between images of measurement clothing and images of try-on clothing. As shown in FIG. 10, the learning data 124 corresponds image information of measurement clothing and image information of try-on clothing according to the body type and posture of the robot mannequin 20. In the illustrated example, the learning data 124 corresponds, for example, image information "A1" of measurement clothing in which the posture of the robot mannequin 20 is "posture 1" with image information "B11" of try-on clothing for a robot mannequin 20 corresponding to a normal body type. Also, image information "A1X" of measurement clothing in which the posture of the robot mannequin 20 is "posture 1" with image information "B11X" of try-on clothing for a robot mannequin 20 corresponding to an obese type.

Figure 11 is a diagram illustrating an example of an image generation process by the image processing device 100.

As shown in FIG. 11(a), in the execution phase, the image processing device 100 divides an image of a user of a normal body type wearing measurement clothing into an image of the measurement clothing and an image of the user's body parts. The image processing device 100 then inputs the divided image of the measurement clothing into the trained model 126. As a result, an image of try-on clothing that is close to a normal body type is output from the trained model 126. The image processing device 100 then generates an image of the user wearing the try-on clothing by combining the image of the try-on clothing output from the trained model 126 with the image of the user's body parts divided as described above.

As shown in FIG. 11(b), in the execution phase, the image processing device 100 segments an image of an obese user wearing the measurement garment into an image of the measurement garment and an image of the user's body parts. The image processing device 100 then inputs the segmented image of the measurement garment into the trained model 126. As a result, an image of try-on garments that are close to the obese type is output from the trained model 126. The image processing device 100 then generates an image of the user wearing the try-on garment by combining the image of the try-on garment output from the trained model 126 with the image of the user's body parts segmented as described above.

As described above, according to the third embodiment, in addition to the effects (1) to (6) of the first embodiment, the following effects can be obtained.
(8) The image processing device 100 trains the trained model 126 using the training data 124 in which images of measurement garments and images of try-on garments are associated so as to correspond to users of a variety of body types. The image processing device 100 also inputs images of users wearing the measurement garments to the trained model 126 to generate images of users of various body types wearing the try-on garments. In other words, by configuring the training data 124 so as to correspond to users of various body types, it is possible to generate highly realistic images of try-on garments while reducing the computational load.

(Other embodiments)
Each of the above-described embodiments can also be implemented in the following forms.
In the above embodiments, the image processing device 100 controls the robot mannequin 20 to collect images of the measurement garments and images of the try-on garments as the learning data 124. Alternatively, images of the measurement garments and images of the try-on garments may be collected as the learning data 124 by having the subject change their posture while wearing the garments.

- In each of the above embodiments, the trained model 126 is trained using an image of measurement clothing worn by the user. Alternatively, the trained model 126 may be trained using an image of ordinary clothing worn by the user. For example, the user's posture may be estimated based on an image of ordinary clothing worn by the user, and the estimated user's posture may be associated with the image of the ordinary clothing to form the training data 124. When estimating the user's posture based on an image of ordinary clothing, an image of a robot mannequin 20 wearing ordinary clothing may be used, or an image of a subject wearing ordinary clothing may be used.

[Hardware configuration]
FIG. 12 is a diagram showing an example of a hardware configuration of an image processing device 100 according to an embodiment. As shown in the figure, the image processing device 100 includes a communication controller 100-1, a CPU 100-2, a RAM (Random Access Memory) 100-3 used as a working memory, a ROM (Read Only Memory) 100-4 for storing a boot program, a storage device 100-5 such as a flash memory or a HDD (Hard Disk Drive), a drive device 100-6, and the like, all connected to each other via an internal bus or a dedicated communication line. The communication controller 100-1 communicates with components other than the image processing device 100. The storage device 100-5 stores a program 100-5a executed by the CPU 100-2. This program is expanded in the RAM 100-3 by a DMA (Direct Memory Access) controller (not shown) or the like, and executed by the CPU 100-2. This realizes a robot control unit 112, an imaging control unit 114, a learning unit 116, and an image generating unit 118.

The above-described embodiment can be expressed as follows.
A storage device storing a program;
a hardware processor;
The hardware processor executes the program stored in the storage device,
The image processing device is configured to input an image of a user wearing the reference wearing item to a trained model trained by machine learning using learning data in which an image of a reference wearing item worn by a wearer in a predetermined posture is used as input data, and an image of a tried-on wearing item worn by the wearer in a posture common to the predetermined posture is used as output data. The trained model is trained by machine learning using learning data in which an image of a user wearing the reference wearing item is used to generate an image of a user trying on the tried-on wearing item.

The above-described embodiments are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The elements of the embodiments, as well as their arrangement, materials, conditions, shapes, sizes, etc., are not limited to those exemplified and can be modified as appropriate. Furthermore, configurations shown in different embodiments can be partially substituted or combined.

1...image generation system, 10...camera, 20...robot mannequin, 30...display, 100...image processing device, 110...control unit, 112...robot control unit, 114...photography control unit, 116...learning unit, 118...image generation unit, 120...memory unit, 122...image data, 124...learning data, 126...learned model.

Claims (7)

The image generating unit is provided for inputting an image of a user wearing the reference wearing product to a trained model trained by machine learning using training data in which an image of a reference wearing product worn by a wearer in a predetermined posture is used as input data, and an image of a tried-on wearing product worn by a wearer in a posture common to the predetermined posture is used as output data. The image generating unit generates an image of a user wearing the tried-on wearing product by inputting an image of a user wearing the reference wearing product to a trained model trained by machine learning using training data in which an image of a user wearing the reference wearing product is used as input data, and an image of a user trying on the tried-on wearing product is generated.
Image processing device. The image generating unit includes:
Extracting an image of the reference wearing item and an image of a body part of the user from an image of the user wearing the reference wearing item;
The extracted image of the reference wearing item is input to the trained model, and the image of the try-on wearing item output from the trained model is synthesized with the extracted image of the user's body part to generate an image of the user who has tried on the try-on wearing item.
The image processing device according to claim 1 . The image generating unit extracts depth information of the image of the reference wearing item and depth information of the image of a body part of the user from an image of the user wearing the reference wearing item,
synthesizing the image of the try-on garment with an image of a body part of the user based on the extracted depth information;
The image processing device according to claim 2 . The learning data is obtained by associating an image of the reference wearing item worn by the wearing subject in a predetermined posture as the input data, an image of the try-on wearing item of a first variation worn by the wearing subject in a posture common to the predetermined posture as the output data, and an image of the try-on wearing item of a second variation worn by the wearing subject in a posture common to the predetermined posture as the output data with the input data;
The image generating unit includes:
Accepting a selection of a variation of the try-on garment;
An image of a user wearing the reference wearing item is input to the trained model to generate an image of a user trying on the try-on wearing item of the selected variation.
The image processing device according to claim 1 . The wearable item is clothing,
The learning data includes data obtained by associating an image of a reference garment worn by a subject in a predetermined posture as the input data with an image of a try-on garment worn by the subject in a posture common to the predetermined posture as the output data.
The image processing device according to claim 1 . The image generating step includes inputting an image of a user wearing the reference wearing product to a trained model trained by machine learning using training data in which an image of a reference wearing product worn by a wearer in a predetermined posture is used as input data, and an image of a user trying on the try-on wearing product worn by the wearer in a posture common to the predetermined posture is used as output data.
Image processing methods. On the computer,
An image of a user wearing the reference wearing item is input to a trained model trained by machine learning using training data in which an image of a reference wearing item worn by a wearer in a predetermined posture is used as input data, and an image of a user trying on the try-on wearing item is generated by inputting an image of the user wearing the reference wearing item into the trained model.
Execute the process,
Image processing program.

Images