WO2024009721A1

WO2024009721A1 - Image processing device, and image processing method

Info

Publication number: WO2024009721A1
Application number: PCT/JP2023/022231
Authority: WO
Inventors: 倫晶有定; 新吾堀内; 大暉市原; 祥彦静野; 浩之木村; 裕貴中山; 喜貴千賀
Original assignee: 株式会社Ｎｔｔデータ
Priority date: 2022-07-08
Filing date: 2023-06-15
Publication date: 2024-01-11
Also published as: JP2024008557A

Abstract

The present invention reduces unnaturalness in a synthesized image and obviates the need for manual correction by a person. This image processing device: divides a face and complete hands in a model image for synthesis into regions bounded by edges, on the basis of edge information pertaining to a mask image of a face and hands in the model image for synthesis and edge information pertaining to a clothing image for synthesis; calculates, for each divided region, the rate of consistency between a second 3D clothed avatar and portions that correspond to images of the face and complete hands in the model image for synthesis, the face and hands having been divided into the regions; ad generates a face/hand image for synthesis for which a before/after relationship between clothing and a person has already been assessed. The image processing device also: generates, for a first 3D clothed avatar, a shadow image in which rendering is executed to reflect settings data that is associated with the model image for synthesis; outputs a first clothed model image in which the clothing image is superposed on the model image for synthesis; superposes the face/hand image for synthesis on the first clothed model image; and furthermore superposes the shadow image on the resultant image to generate a final clothed model image.

Description

Image processing device and image processing method

The present invention relates to an image processing device and an image processing method. More specifically, the present invention attaches 3D clothing data created by a user to a 3D avatar in a 3DCG environment, converts only the clothing image into 2D data, and automatically converts the 2D data of the clothing image and the 2D data of the human. The present invention relates to an image processing apparatus and an image processing method for synthesizing images and outputting them as a 2D data wearing image.

Traditionally, in the apparel and advertising industries, real models have been photographed wearing clothing, and the images have been used for sales promotions and marketing. However, such a method requires a large amount of human resources, money, and time to generate one image, and is not suitable for generating many images.

For this reason, 3D avatars and a technology called GAN (Generative Adversarial Network) are being used. An avatar is a character used as a user's alter ego on a network, and includes two-dimensional (2D) image avatars and three-dimensional (3D) avatars. GAN is a method of generating images by learning using two neural networks. GAN technology has made it possible for computers to generate an unlimited number of similar images as if they were taken of the real thing.

By using 3D avatars and GAN technology, efforts are being made in the apparel and advertising industries to utilize non-existent models (virtual models) in sales promotions, marketing, and other business operations. In particular, in the apparel industry, 3D models of clothing are created, and the 3D clothing data is converted into 2D data and composited with the model's image. Because the image may sometimes be hidden away, corrections were made manually to make it look natural.

Japanese Patent Application Publication No. 2011-186774

Because there is a limit to the number of images that can be created using methods that perform image synthesis using human hands, many companies are researching image synthesis methods that automatically attach 3D clothing data to human images.

The technique disclosed in Patent Document 1 was known as an image synthesis technique for synthesizing clothing data with a mannequin image. However, even if the technique of Patent Document 1 is used, there is a problem in that the combination of the clothing image and the model image still looks unnatural. For example, when an image of a human body from the neck up and an image of clothing are combined, the lining of the back of the neck is displayed, leaving the combined image unnatural.

In addition, when automatically applying 3D clothing data to a 2D image of a person, conventional technology frequently suffers from accuracy problems such as the clothing area that should originally be on the person becoming invisible. However, there were technical issues, such as the need for manual corrections in the end.

The present invention was made to solve such problems, and it involves dressing a 3D avatar with 3D clothing data created by a user in a 3DCG environment, converting only the clothing image into 2D data, and converting the clothing image into 2D data. An object of the present invention is to provide an image processing device and an image processing method that automatically synthesize images of 2D data and human 2D data and output the 2D data as a worn image.

An image processing device that is one aspect of the present invention includes
Means for generating a first 3D clothed avatar, a second 3D clothed avatar, and a synthetic clothing image using setting data, a 3D avatar, and 3D clothing data associated with the synthetic model image;
An image of the entire expressed human body part of the synthetic model image is surrounded by edges based on edge information of a mask image of the expressed human body part associated with the synthetic model image and edge information of the synthetic clothing image. and for each divided region, the matching rate of the corresponding portion between the second 3D clothed avatar and the image of the entire expressed human body part of the synthesis model image divided into the regions. means for generating a composite expressed human body part image in which the anteroposterior relationship between the clothing and the human body has been determined by calculating the
means for generating a shadow image when rendering is performed on the first 3D clothed avatar by reflecting setting data associated with the synthesis model image;
means for superimposing the synthetic clothing image on the synthetic model image to output a first clothed model image;
and means for superimposing the expressed human body part image for synthesis on the first clothed model image, and further superimposing the negative image on the first clothed model image to generate a final clothed model image.

According to the present invention, by using two types of edge images, 3D image data, and model images, the unnaturalness that conventionally occurs in a composite image is reduced, and manual correction by humans is no longer necessary.

A detailed understanding of the embodiments disclosed herein can be obtained from the following description, illustrated in conjunction with the accompanying drawings.
1 is an overall configuration diagram of an image processing system 1 including an image processing device 10, a user terminal 11, a 3D scanner 12, and an imaging device 13 according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an overview of processing executed by an image processing device 10, a user terminal 11, and an imaging device 13 according to an embodiment of the present invention. 1 is a system configuration diagram of an image processing apparatus 10 according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a data structure of live photographic data 106 according to an embodiment of the present invention. It is a diagram showing an example of a data structure of a 3D avatar 108 according to an embodiment of the present invention. It is a diagram showing an example of a data structure of a 3D clothed avatar 109 according to an embodiment of the present invention. It is a diagram showing an example of a data structure of a clothing image for synthesis 110 according to an embodiment of the present invention. FIG. 2 is a diagram showing a processing flow in which the image processing device 10 generates a 3D clothed avatar and a synthetic clothing image. FIG. 3 is a diagram illustrating a processing flow in which the image processing device 10 generates a composite face/hand image in which the anteroposterior relationship between clothing and a human body has been determined. 3 is a diagram showing a processing flow in which the image processing device 10 generates a shadow image. FIG. FIG. 3 is a diagram showing a processing flow in which the image processing device 10 generates a final clothed image. (a) is a diagram illustrating an image in which edge information is extracted from mask images of faces and hands, and clothing images for synthesis, and (b) is a diagram illustrating an image in which two pieces of edge information are combined. (a) is a diagram showing an example in which the hand part of the synthesis model image is divided into several regions surrounded by edges, and (b) is a diagram showing the conventional method of the generated synthesis face/hand image. FIG. 2 is a diagram illustrating an example of a hand image of the technique and the present invention.

(overall structure)
FIG. 1 is an overall configuration diagram of an image processing system 1 including an image processing device 10, a user terminal 11, a 3D scanner 12, and an imaging device 13 according to an embodiment of the present invention. The image processing device 10 is connected to a user terminal 11 and an imaging device 13 via a network 14 so as to be able to communicate with each other. The user terminal 11 is connected to the 3D scanner 12 via an arbitrary network such as a LAN or WAN so that they can communicate with each other. Although FIG. 1 shows only one image processing device 10, one user terminal 11, one 3D scanner 12, and one imaging device 13 for simplicity of explanation, there may be a plurality of these devices.

In this specification, the image processing device 10 will be described as one device or system, but the various processes executed by the image processing device 10 may be configured to be distributed and executed by multiple devices or systems. .

The image processing device 10 executes image composition processing described in this specification. More specifically, the image processing device 10 dresses the 3D clothing data created by the user on a 3D avatar in a 3DCG environment, converts only the clothing image into 2D data, and converts the 2D data of the clothing image and the 2D data of the human being into 2D data. The system automatically synthesizes the images and outputs a 2D data wearing image.

The user terminal 11 can be any type of device (e.g., PC, tablet terminal, etc.) capable of operating in a wired or wireless environment used by the user, but is not limited to a specific device or device. There isn't. The user terminal 11 can generate 3D clothing data using a third-party application, and can generate 3D avatar data using a 3D scanner 12 or the like. The user terminal 11 transmits 3D clothing data and 3D avatar data to the image processing device 10, transmits various instructions regarding image composition processing via an application provided by the image processing device 10, and performs image processing on the composition results. It can be received from the device 10.

The 3D scanner 12 is a device that has a function of generating 3D avatar data in response to instructions from the user terminal 11.

The imaging device 13 is a device that takes a live photograph of a real model, and is a device that takes an image of the model using one or more cameras, and can include any studio device. In order to easily identify the captured image data of the model, the floor, wall, etc. of the imaging location may be any background such as a blue background or a green background.

The network 14 is any communication network responsible for communication between the image processing device 10, the user terminal 11, the 3D scanner 12, and the imaging device 13, and includes the Internet, an intranet, a leased line, any network system, etc., and is particularly limited to It will not be done.

(Functional configuration of image processing device 10)
The image processing device 10 uses images of real models and various setting data to dress 3D avatars with 3D clothing data created by the user in a 3DCG environment, converts only the clothing images into 2D data, and creates clothing images. automatically synthesizes the 2D data of the person and the 2D data of the person, and outputs the 2D data wearing image.

Hereinafter, various functions provided by the image processing device 10 will be explained with reference to FIG. 2. FIG. 2 is a diagram illustrating an overview of processing executed by the image processing device 10, the user terminal 11, and the imaging device 13 according to the embodiment of the present invention. S1 in FIG. 2 is executed by the imaging device 13, S2 is executed by the user terminal 11, and S3 to S6 are executed by the image processing device 10. Note that the embodiments described in this specification will be described using tops (upper garments) as an example of the type of clothing, but the present invention is also applicable to other types of clothing (e.g., bottoms). I hope you understand that.

(S1: Processing details when photographing a live-action model by the imaging device 13)
A user uses the imaging device 13 to photograph an actual model. As will be described later, a model image of an actual model is used as a model image for synthesis in image synthesis processing that will be described later. The model image for synthesis is 2D image data.

The imaging device 13 transmits the synthesis model image, camera setting data (camera angle, distance, etc.) and illumination setting data (brightness, etc.) at the time of imaging to the image processing device 10. The image processing device 10 stores the compositing model image, camera setting data, and illumination setting data received from the imaging device 13 in the live photographing data 106.

(S2: Processing details of 3D clothing data and 3D avatar generation by user terminal 11)
The user terminal 11 has an application such as a third party, generates 3D clothing data, and also communicates with the 3D scanner 12 to generate a 3D avatar with the same pose as the actual model. The pose of the 3D avatar may be the same as that of the real model at this stage, or it may be set as a basic pose and changed to the same pose during processing in S3, which will be described later.

The user terminal 11 transmits the 3D clothing data and 3D avatar to the image processing device 10, and the image processing device 10 stores the received 3D clothing data and 3D avatar in 3D clothing data 107 and 3D avatar 108, respectively.

(S3: Processing content for generating 3D clothed avatar and synthetic clothing image)
The image processing device 10 provides the user terminal 11 with an application for generating a 3D clothed avatar and a synthetic clothing image. In response to an instruction from the user terminal 11, the image processing device 10 outputs a 2D image of only clothes (synthesis clothing image) and two types of 3D clothed avatars for use in the compositing process.

The image processing device 10 reads 3D clothing data from the 3D clothing data 107 and reads 3D avatars from the 3D avatar 108 in response to instructions from the user terminal 11. In response to an instruction from the user terminal 11, the image processing device 10 can also read out a model image from the live-action photography data 106 and change the pose of the 3D avatar to the same pose as the pose of the read model image. In response to an instruction from a user terminal 11, the image processing device 10 superimposes 3D clothing data on a 3D avatar in a 3DCG (computer graphics) space, performs predetermined position calculations, and determines the size and placement position of the 3D clothing data. and place the 3D clothing data at the appropriate position of the 3D avatar. Through this process, the 3D clothing data is put on the 3D avatar.

In response to an instruction from the user terminal 11, the image processing device 10 executes a cross simulation of 3D clothing data on a 3D avatar wearing 3D clothing data in a 3DCG space according to the body shape and pose of the 3D avatar, The first 3D clothed avatar is stored in the 3D clothed avatar 109.

The image processing device 10 reflects the camera setting data and lighting setting data read from the live-action shooting data 106 on the first 3D clothed avatar in the 3DCG space, and uses predetermined shader setting parameters. 3D rendering processing is performed, and the second 3D clothed avatar is stored in the 3D clothed avatar 109.

The image processing device 10 generates a 2D clothing image (also referred to as a "synthesis clothing image") based on the 3D clothing data excluding the 3D avatar, and stores it in the synthesis clothing image 110.

(S4: Generation process of face and hand images for synthesis)
The user uses any application to mechanically or manually generate face and hand mask images from the synthesis model image using a method such as binarization. That is, in response to a mask image generation instruction from the user terminal 11, the image processing device 10 generates mask images of the face and hands from the synthesis model image received from the imaging device 13. The image processing device 10 extracts edge information of a face and hand mask image that has been generated in advance using an arbitrary filter. The described embodiment is explained using tops as an example of the type of clothing, so the parts of the human body that are exposed in the clothed state (exposed human body parts) are the face and/or hands, but other It should be understood that in the case of different types of clothing, the body parts exposed may vary depending on the type of clothing (eg, feet and/or ankles in the case of bottoms).

The image processing device 10 extracts key points of the face and hands from the synthesis model image, and sets a search range using the extracted key points as a bounding box. The image processing device 10 searches for edges while moving from one side of the 3D face or hand to the other side (for example, from the left outer part to the right outer part), and reaches the other side of the face or hand. and continues searching the edge until it turns around. The image processing device 10 extracts an image of the entire face and hand of the synthesis model image from the searched edge range.

The image processing device 10 executes depth information extraction processing to extract depth information (also referred to as "depth information") in the first 3D clothed avatar.

The image processing device 10 extracts edge information of the clothing image for synthesis using an arbitrary filter. The image processing device 10 can extract edge information using depth information, because if the edges of the clothing image for synthesis are extracted as they are, wrinkles caused by the texture will become edge noise.

The image processing device 10 combines the edge information of the mask image of the face and hand with the edge information of the clothing image for synthesis, and creates several images of the entire face and hand of the model image for synthesis based on the combined images. To divide. The image processing device 10 determines the face and hand to be finally synthesized while calculating the matching rate with the corresponding area in the second 3D clothed avatar for each divided area of the entire face and hand of the synthesis model image. Extract images (face and hand images for synthesis).

(S5: Shadow image generation process)
The image processing device 10 reflects the camera setting data and proof setting data used when generating the synthesis model image on the first 3D clothed avatar read from the 3D clothed avatar 109, and calculates the shadow created when rendering is performed. (shade) information is generated as a shade image and stored in the 3D clothed avatar 109.

(S6: Clothes composition processing that generates the final clothed image)
The image processing device 10 executes a first clothing composition process of superimposing a composition clothing image on a composition model image and outputting a first clothed model image. The image processing device 10 executes a second clothed image generation process that generates a final clothed model image by superimposing the synthetic face and hand images on the first clothed model image and further superimposing the shadow image. do.

(System configuration of image processing device 10)
Next, the system configuration of the image processing device 10 will be explained. FIG. 3 is a system configuration diagram of the image processing device 10 according to the embodiment of the present invention. The image processing device 10 may be configured to be placed on a cloud system or on an in-house network. As shown in FIG. 3, the image processing device 10 includes a control section 101, a main storage section 102, an auxiliary storage section 103, and an interface (IF) section 104, which are interconnected by a bus 120 or the like, like a general computer. , and an output section 105. The auxiliary storage unit 103 stores programs that implement each function of the image processing device 10 and data handled by the programs. The auxiliary storage unit 103 includes live photographed data 106, 3D clothing data 107, 3D avatar 108, 3D clothing avatar 109, and synthetic clothing image 110 in a file/database format. The image processing device 10 can read or update information stored in the live-action photography data 106, 3D clothing data 107, 3D avatar 108, 3D clothed avatar 109, and synthetic clothing image 110. Each program stored in the auxiliary storage unit 103 is executed by the image processing device 10.

The control unit 101, also called a central processing unit (CPU), controls each component of the image processing device 10 and calculates data, and reads various programs stored in the auxiliary storage unit 103 to the main storage unit 102. and execute it. The main storage unit 102 is also called main memory, and stores various received data, computer-executable instructions, and data after arithmetic processing using the instructions. The auxiliary storage unit 103 is a storage device such as a hard disk (HDD) or SSD (Solid State Drive), and stores data and programs for a long period of time.

Although the embodiment of FIG. 3 describes an embodiment in which the control unit 101, the main storage unit 102, and the auxiliary storage unit 103 are provided inside the same computer, as another embodiment, the image processing apparatus 10 By using a plurality of main storage units 102 and auxiliary storage units 103, it is also possible to implement parallel distributed processing by a plurality of computers. Furthermore, as another embodiment, a plurality of servers for the image processing apparatus 10 may be installed, and one auxiliary storage unit 103 may be shared by the plurality of servers.

The IF unit 104 serves as an interface for transmitting and receiving data with other systems and devices, and also provides an interface for receiving various commands and input data (various masters, tables, etc.) from the system operator. The output unit 105 provides a display screen for displaying processed data, a printing means for printing the data, and the like.

Components similar to the control unit 101, main storage unit 102, auxiliary storage unit 103, IF unit 104, and output unit 105 also exist in the user terminal 11 and the imaging device 13.

The live-action shooting data 106 stores a model image (2D image data) of a real model, a mask image of the face and hands of the real model, and camera setting data and lighting setting data at the time of live-action shooting. FIG. 4 is a diagram illustrating an example of the data structure of the live-action photography data 106 according to the embodiment of the present invention. The live-action shooting data 106 can include a live-action shooting ID 401, a model image 402, a mask image 403, camera setting data 404, and lighting setting data 405, but is not limited to these data items and may include other data items. can also be included.

The live-action shooting ID 401 is an identifier that identifies a model at the time of live-action shooting and data associated with the model. The model image 402 is 2D model image data of a real model, and is also called a "synthesis model image." The mask image 403 is a mask image of the model's face and hands generated from the synthesis model image. Camera setting data 404 indicates camera setting data at the time of live-action photography, such as camera angle and distance. Lighting setting data 405 indicates lighting setting data, such as brightness, at the time of live-action photography.

Returning to FIG. 3, the 3D clothing data 107 stores 3D clothing data generated by the user. The 3D clothing data may be stored in association with attribute information (eg, clothing category, color, shape, etc.) to facilitate image selection.

The 3D avatar 108 stores 3D avatar data generated by the user. The 3D avatar is created by the user so as to have the same pose as the model image at the time of live-action photography. FIG. 5 is a diagram showing an example of the data structure of the 3D avatar 108 according to the embodiment of the present invention. The 3D avatar 108 can include a 3D avatar ID 501, a 3D avatar 502, and a live-action shooting ID 401, but is not limited to these data items and can also include other data items.

The 3D avatar ID 501 is an identifier that identifies a 3D avatar. 3D avatar 502 shows 3D avatar data. The live-action shooting ID 401 is an identifier that identifies the live-action shooting associated with the 3D avatar. The live-action shooting ID 401 associates the corresponding live-action model's pose, mask image, camera and lighting setting data with the 3D avatar.

Returning to FIG. 3, the 3D clothed avatar 109 stores image data of a 3D clothed avatar obtained by superimposing 3D clothing data on the 3D avatar and performing predetermined processing. FIG. 6 is a diagram showing an example of the data structure of the 3D clothed avatar 109 according to the embodiment of the present invention. The 3D clothed avatar 109 can include a 3D clothed avatar ID 601, a first 3D clothed avatar 602, a second 3D clothed avatar 603, shadow information 604, a shadow image 605, a 3D avatar ID 501, and a live-action shooting ID 401. It is not limited to these data items and can also include other data items.

The 3D clothed avatar ID 601 is an identifier that identifies the 3D clothed avatar generated by the image processing device 10. The first 3D clothed avatar 602 shows image data of a 3D clothed avatar that has undergone 3D cloth simulation. A second 3D clothed avatar 603 represents image data of a 3D clothed avatar that has been subjected to 3D rendering processing by reflecting camera setting data and certification setting data on the first 3D clothed avatar. The shadow information 604 and the shadow image 605 are the shadow information and shadow information generated when rendering is executed by reflecting the camera setting data and proof setting data when generating the synthesis model image for the first 3D clothed avatar. The shadow images are shown respectively. The 3D avatar ID 501 is an identifier for identifying the 3D avatar from which the 3D clothed avatar is generated, and the live-action shooting ID 401 is an identifier for identifying the live-action shooting associated with the 3D avatar. The 3D avatar ID 501 and live-action shooting ID 401 make it easier to acquire various setting data and the like when shooting a real model.

Returning to FIG. 3, the synthetic clothing image 110 stores 2D clothing data generated based on the 3D clothing data of the second 3D clothing avatar. FIG. 7 is a diagram illustrating an example of a data structure of a clothing image for synthesis 110 according to an embodiment of the present invention. The synthetic clothing image 110 can include a synthetic clothing image ID 701, a synthetic clothing image 702, a live-action photography ID 401, and a 3D clothed avatar ID 601, but is not limited to these data items and may include other data items. can also be included.

The compositing clothing image ID 701 is an identifier that identifies 2D clothing image data used in the image compositing process according to the embodiment of the present invention. A clothing image for composition 702 indicates 2D clothing image data used for image composition processing. The live-action shooting ID 401 is an identifier that identifies the live-action shooting associated with the 3D clothed avatar from which the synthetic clothing image is generated. The 3D clothed avatar ID 601 is an identifier of a 3D clothed avatar associated with 3D cloth data that is the original data of the clothing image for synthesis.

(Explanation about various flows)
With reference to FIGS. 8 to 11, the image processing device 10 creates a final clothed model using a compositing clothing image (2D), a compositing model image (2D), various setting data, 3D avatars, and 3D clothing data. The processing flow for generating an image will be explained. 8 to 11 show the processing contents of S3 to S6 in FIG. 2, respectively. Either of S4 and S5 may be performed first.

(S3: Processing details for generating 3D clothed avatar and synthetic clothing image)
FIG. 8 shows data generated by the image processing device 10 through the processing described above with reference to S1 and S2 of FIG. A processing flow for generating a 3D clothed avatar and a synthetic cloth image using data is shown.

In the process of FIG. 8, the image processing device 10 provides the user terminal 11 with an application for generating a 3D clothed avatar image, and performs the process based on user instructions received via the user terminal 11.

In S801, the user terminal 11 selects a synthesis model image, 3D clothing data, and 3D avatar to be processed through the provided application, and sends a selection instruction to the image processing device 10. In response to a selection instruction from the user terminal 11, the image processing device 10 reads the selected model image from the live-action photography data 106, reads 3D clothing data from the 3D clothing data 107, and converts the selected 3D avatar into a 3D avatar 108. Read from. The image processing device 10 can change the pose of the 3D avatar to match the pose of the read model image. Through this processing, the pose of the model image and the pose of the 3D avatar match, the model image and the 3D avatar are associated, and the selected live-action shooting ID 401 is stored in the 3D avatar 108.

The user terminal 11 transmits a placement instruction to the image processing device 10 to place the 3D clothing data at an appropriate position of the 3D avatar on the 3DCG space of the application. The image processing device 10 adjusts the size and placement position of the 3D clothing data by superimposing the 3D clothing data on the 3D avatar in the 3DCG space and performing predetermined position calculations in accordance with placement instructions from the user terminal 11. and place the 3D clothing data at an appropriate position on the 3D avatar. Through this process, the 3D clothing data is put on the 3D avatar.

In S802, the user terminal 11 transmits a simulation instruction to the image processing device 10 to perform a cloth simulation on a 3D avatar wearing 3D clothing data. Cloth simulation refers to a technology that physically simulates the movement of cloth such as clothing. For example, physical calculations are performed on the cloth, such as simulating the wrinkles in clothing when a 3D avatar wears it.

In response to a simulation instruction from the user terminal 11, the image processing device 10 executes a cross simulation of 3D clothing data in a 3DCG space on a 3D avatar wearing 3D clothing data according to the body shape and pose of the 3D avatar. , the cross-simulated 3D clothed avatar is stored in the first 3D clothed avatar 602 of the 3D clothed avatar 109 .

In S803, the image processing device 10 reads camera setting data and lighting setting data associated with the model image selected in S801 from the live-action photography data 106. The image processing device 10 reflects the read camera setting data and lighting setting data on the cross-simulated 3D clothed avatar (first 3D clothed avatar) in the 3DCG space of the application, and applies a predetermined shader. ) Execute the 3D rendering process using the setting parameters, and store the 3D rendered avatar (second 3D clothed avatar) in the second 3D clothed avatar 603 of the 3D clothed avatar 109.

In S804, the image processing device 10 extracts 3D clothing data by removing the 3D avatar from the 3D clothing avatar (second 3D clothing avatar) that has undergone 3D rendering processing. The image processing device 10 generates a 2D clothing image (herein referred to as a "synthesis clothing image") based on the extracted 3D clothing data, and stores it in the synthesis clothing image 110.

(S4: Processing details for generating face and hand images for synthesis)
In FIG. 9, the image processing device 10 uses a face and hand mask image, a composition clothing image, and a 3D clothed avatar to generate a composite face and hand image in which the front-back relationship between the clothing and the human body has been determined. The processing flow is shown. Note that this processing flow assumes that the image processing device 10 communicates with the user terminal 11 through an arbitrary application to generate mask images of the face and hands from the synthesis model image. Additionally, in this specification, the term "hand" is used to indicate any of the wrist, palm, and fingers of a human body from the shoulder to the fingertips, but these may vary depending on the design of the clothing. .

In S901, the image processing device 10 extracts key points of the face and hands from the synthesis model image associated with the live-action shooting ID 401 to be processed, and sets a search range using the extracted key points as a bounding box. The image processing device 10 searches for edges while moving from one side of the face or hand to the other side (for example, from the left outer part to the right outer part), and reaches the other side of the face or hand. Continue searching the edge until it turns around. The image processing device 10 extracts the entire face and hand image of the synthesis model image from the searched edge range.

The image processing device 10 extracts edge information of a face and hand mask image that has been generated in advance using an arbitrary filter. The upper part of FIG. 12(a) shows an image of extracting edge information from mask images of faces and hands.

In S902, the image processing device 10 reads the first 3D clothed avatar 602 from the 3D clothed avatar 109, and performs depth information extraction processing on the read first 3D clothed avatar 602. Through this process, the image processing device 10 can acquire depth information at each position of the first 3D clothed avatar 602. Depth information makes it possible to distinguish between wrinkles and contour lines in clothing.

In S903, the image processing device 10 extracts edge information of the clothing image for synthesis using an arbitrary filter. If the edge information of the clothing image for synthesis is extracted as is, wrinkles due to the texture of the clothing may become noise, so the image processing device 10 extracts the edge information of the clothing image for synthesis using the depth information acquired in S902. Can be done. The lower part of FIG. 12(a) shows an image of extracting edge information from a clothing image for synthesis.

Note that the order of the processing in S901 and the processing in S902 and S903 is not particularly limited and may be performed first. That is, the processing in S902 and S903 may be performed after the processing in S901, or the processing in S901 may be performed after the processing in S902 and S903. Alternatively, both may be processed in parallel.

In S904, the image processing device 10 combines the edge information of the clothing image for synthesis with the edge information of the face and hand mask images. FIG. 12(b) shows an image in which two pieces of edge information are combined.

In S905, the image processing device 10 divides the entire face and hand image of the synthesis model image extracted in S901 into regions surrounded by edges, based on the edge information combined in S904. FIG. 13A is an example showing that the hand portion of the synthesis model image is divided into several regions surrounded by edges.

In S906, the image processing device 10 reads the second 3D clothed avatar 603 from the 3D clothed avatar 109, and combines the read second 3D clothed avatar 603 with the compositing model image that has been divided into several regions. The corresponding parts of the face and the entire hand image (for example, the thumbs of both left hands) are compared for each divided region. The image processing device 10 calculates the matching rate between the two, and determines that a portion where the matching rate is equal to or higher than a predetermined threshold value (X) is an actually visible portion. Based on the determination results for each area, the image processing device 10 determines a certain area of the entire face and hand image of the synthesis model image as a visible part (visible part) or as an invisible part (visible part). The face and hand images to be finally synthesized are extracted. In the example of FIG. 13(b), since the matching rate for the thumb portion of the left hand was less than the threshold value (X), the image processing device 10 determines that the image of the thumb portion of the left hand should not be combined with the face and hand images to be finally combined. Processing is being performed not to include it.

The threshold (X) can be changed based on depth information. Therefore, the image processing apparatus 10 can change the threshold value (X) of each position based on the depth information of each position acquired in S902. Therefore, the value of the threshold (X) can change for each divided region surrounded by edges.

In S907, the image processing device 10 generates a face/hand image for synthesis based on the face and hand images to be finally synthesized for each region extracted in S906. FIG. 13(b) shows an example of the generated synthetic face/hand images of the prior art and the hand image of the present invention. As shown in FIG. 13(b), the thumb portion is visible because the conventional general image synthesis process does not perform the above-described matching determination. In the actual pose, the thumb is hidden behind the folds of the clothing and cannot be seen, resulting in an unnatural image. On the other hand, when a match is determined according to the present invention, the thumb part is hidden behind the folds of the clothing and cannot be seen. The image processing device 10 determines that the matching rate for this thumb portion is less than the threshold (X), and does not include this thumb portion in the face/hand image for synthesis since it is an invisible portion.

(S5: Shadow image generation process)
In FIG. 10, the image processing device 10 renders the cross-simulated 3D clothed avatar (first 3D clothed avatar) by reflecting the camera setting data and proof setting data when generating the synthesis model image. The processing flow for generating a shade image based on shade information generated when executing the process is shown.

In S1001, the image processing device 10 reads the first 3D clothed avatar 602 from the 3D clothed avatar 109 based on the live-action shooting ID 401 to be processed. The image processing device 10 also queries the live-action shooting data 106 based on the live-action shooting ID 401 and reads out the corresponding camera setting data 404 and illumination setting data 405.

In S1002, the image processing device 10 performs rendering on the read first 3D clothed avatar 602 by reflecting the corresponding camera setting data 404 and lighting setting data 405, and determines whether or not light is shining on it. Calculate and perform shading.

In S1003, the image processing device 10 generates a shadow image based on shadow information that is a shading calculation result. The image processing device 10 stores the shadow information and the shadow image as the shadow information 604 and the shadow image 605 of the 3D clothed avatar 109, respectively.

(S6: Clothes composition processing that generates the final clothed image)
FIG. 11 shows a first clothing composition process in which the image processing device 10 outputs a first clothed model image by superimposing a composition clothing image on a composition model image associated with the live-action shooting ID 401 to be processed; A second clothed compositing process that generates a final clothed model image by superimposing the compositing face/hand image on the first clothed model image that is the output of the first clothed compositing process, and further overlapping the shadow image. The processing flow for generating the final clothed image by executing the following is shown.

In S1101, the image processing device 10 executes a first clothing composition process. More specifically, the image processing device 10 reads the model image 402 from the live-action photography data 106 based on the live-action photography ID 401 to be processed, and queries the clothing image for composition 110 using the live-action photography ID 401. The composite clothing image 702 is read out. The image processing device 10 generates a first clothed model image by superimposing the synthetic clothing image on the synthetic model image.

In S1102, the image processing device 10 executes a second clothing composition process. More specifically, the image processing device 10 generates the final clothed model image by superimposing the synthetic face/hand image and the shadow image on the generated first clothed model image. do. The image processing device 10 provides the generated final clothed model image to the user terminal 11.

(Advantages of the present invention)
The above-described processing enables the image processing device 10 to perform image synthesis processing while estimating the context of a person and clothing more precisely. According to the present invention, it is difficult to identify the front and back relationship between a person and clothing, and problems such as low accuracy of output images such as the clothing area that should normally be on the person becoming invisible can be solved. .

(Other embodiments)
Although humans have been described above as an example, the principles of the present invention are also applicable to animals other than humans. Recently, clothing for animals kept as pets has been on sale. By creating a composite image of such pet animal clothing using the principles of the present invention, the composite image can be used for advertising and marketing.

In addition, although the above explanation has been given using the human face and hands as an example, the principles of the present invention can also be applied to parts of the human body other than the face and hands, depending on the type of clothing, to create a composite image. be able to generate. For example, the parts of the human body that are exposed when wearing tops and bottoms are different. In the case of tops, the exposed body parts may be the face and/or hands, and in the case of bottoms, the exposed body parts may be the hands and/or ankles. In this specification, "exposed human body parts" refer to the face, hands, feet, ankles, etc., depending on the type of clothing.

Furthermore, although the principles of the present invention have been described above with reference to objects such as a human body and clothing, the objects are not limited to the human body and clothing. For example, the target objects may be a human body and a vehicle (car, motorcycle, bicycle, etc.). Furthermore, the number of objects may be three or more. In relation to the above example, it is also possible to generate a composite image using small items such as accessories and bags as the third object. That is, the present invention makes it possible to increase the accuracy of the output composite image even if the context of a plurality of objects is complicated.

Although the principles of the present invention have been described above with reference to exemplary embodiments, those skilled in the art will appreciate that various embodiments with changes in configuration and details can be realized without departing from the gist of the present invention. You will understand. That is, the present invention can be implemented as, for example, a system, device, method, program, storage medium, or the like.

1 Image processing system 10 Image processing device 11 User terminal 12 3D scanner 13 Imaging device 14 Network 101 Control unit 102 Main storage unit 103 Auxiliary storage unit 104 Interface (IF) unit 105 Output unit 106 Live-action shooting data 107 3D clothing data 108 3D avatar 109 3D clothed avatar 110 Clothes image for synthesis

Claims

Means for generating a first 3D clothed avatar, a second 3D clothed avatar, and a synthetic clothing image using setting data, a 3D avatar, and 3D clothing data associated with the synthetic model image;
An image of the entire expressed human body part of the synthetic model image is surrounded by edges based on edge information of a mask image of the expressed human body part associated with the synthetic model image and edge information of the synthetic clothing image. and for each divided region, the matching rate of the corresponding portion between the second 3D clothed avatar and the image of the entire expressed human body part of the synthesis model image divided into the regions. means for generating a composite expressed human body part image in which the anteroposterior relationship between the clothing and the human body has been determined by calculating the
means for generating a shadow image when rendering is performed on the first 3D clothed avatar by reflecting setting data associated with the synthesis model image;
means for superimposing the synthetic clothing image on the synthetic model image to output a first clothed model image;
An image processing device comprising: means for superimposing the expressed human body part image for synthesis on the first clothed model image, and further superimposing the negative image on the first clothed model image to generate a final clothed model image.
The means for generating a synthetic expressed human body part image in which the anteroposterior relationship between clothing and the human body has been determined is as follows:
means for determining that a corresponding portion of an image of the entire expressed human body part of the synthesis model image divided into the regions is a visible portion when the calculated matching rate is equal to or higher than a threshold; The image processing apparatus according to claim 1, further comprising: means, wherein the visible portion is a portion visible on clothing.
Further comprising means for acquiring depth information of the first 3D clothed avatar,
edge information of the synthetic clothing image is extracted using the depth information;
An image processing device according to claim 2.
The image processing device according to claim 3, wherein the threshold associated with the area surrounded by edges differs based on the depth information.
means for extracting key points of expressed human body parts from the synthesis model image and setting a search range using the extracted key points as bounding boxes;
means for searching for edges in the synthetic model image while moving from one side of the expressed human body part to the other side, and continuing to search for edges until reaching the other side of the expressed human body part and turning back;
The image processing apparatus according to claim 1, further comprising means for extracting an image of the entire human body part expressed in the synthesis model image from the range of the searched edges.
When the type of clothing is a top, the exposed human body part is a face and/or a hand;
When the type of clothing is bottoms, the exposed human body parts are feet and/or ankles;
An image processing device according to claim 1.
An image processing method executed by an image processing device, the method comprising:
Generating a first 3D clothed avatar, a second 3D clothed avatar, and a composite clothing image using setting data, a 3D avatar, and 3D clothing data associated with the composite model image;
An image of the entire expressed human body part of the synthetic model image is surrounded by edges based on edge information of a mask image of the expressed human body part associated with the synthetic model image and edge information of the synthetic clothing image. and for each divided region, the matching rate of the corresponding portion between the second 3D clothed avatar and the image of the entire expressed human body part of the synthesis model image divided into the regions. By calculating the above, a expressed human body part image for synthesis in which the anteroposterior relationship between the clothing and the human body has been determined,
generating a shadow image when rendering is performed on the first 3D clothed avatar by reflecting setting data associated with the synthesis model image;
superimposing the synthetic clothing image on the synthetic model image to output a first clothed model image;
An image processing method comprising: superimposing the expressed human body part image for synthesis on the first clothed model image, and further superimposing the negative image on the first clothed model image to generate a final clothed model image.