WO2014081394A1 - Procédé, appareil et système pour la modélisation virtuelle de vêtements - Google Patents

Procédé, appareil et système pour la modélisation virtuelle de vêtements Download PDF

Info

Publication number
WO2014081394A1
WO2014081394A1 PCT/SG2013/000496 SG2013000496W WO2014081394A1 WO 2014081394 A1 WO2014081394 A1 WO 2014081394A1 SG 2013000496 W SG2013000496 W SG 2013000496W WO 2014081394 A1 WO2014081394 A1 WO 2014081394A1
Authority
WO
WIPO (PCT)
Prior art keywords
avatar
user
colour
channel
frames
Prior art date
Application number
PCT/SG2013/000496
Other languages
English (en)
Inventor
Miaolong Yuan
Ishtiaq Rasool KHAN
Farzam Farbiz
Original Assignee
Agency For Science, Technology And Research
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agency For Science, Technology And Research filed Critical Agency For Science, Technology And Research
Priority to SG11201503974SA priority Critical patent/SG11201503974SA/en
Publication of WO2014081394A1 publication Critical patent/WO2014081394A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/16—Cloth
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2012—Colour editing, changing, or manipulating; Use of colour codes

Definitions

  • the present invention generally relates to a method for virtual clothes modeling such as for virtual try-pn of clothes by a user in retail shopping.
  • the present invention also relates to an apparatus and a system for virtual clothes modeling.
  • Physical try-on of clothes can be a time consuming procedure in retail shopping. It typically takes several tries before a shopper can decide on the design, color and size of the apparel that suits the shopper. Virtual try-on of clothes can help to speed-up this process as the shopper is able to view the clothes on his/her body without physically wearing them. This may eliminate the need for physical try-on or at least narrow down his/her selections before physical try-on.
  • Some conventional systems use image processing techniques, such as changing the color of the user's clothes worn by the user or segmenting the user's clothes worn by the user and re-texturing them using extracted shading and shape deformation information based on the clothes which the user would like to try on.
  • Some other conventional systems reconstruct a look-alike 3D model of the user and the clothes are simulated on the reconstructed 3D model.
  • scanning or computer vision-based systems can be used to reconstruct the user's 3D model.
  • a major disadvantage is that the user must physically wear the clothes and that only the clothes color/texture can be changed.
  • AR augmented reality
  • sensing technologies capable of providing high quality synchronized videos of both color and depth such as MICROSOFT KINECT produced by MICROSOFT.
  • Some conventional systems using the MICROSOFT KINECT use deformable 3D cloth models which follow the user's movements with related cloth simulation.
  • MICROSOFT KINECT is normally used for games and other applications, and such conventional systems using MICROSOFT KINECT have not been found to be sufficiently accurate for alignment of the virtual clothes with the user's body.
  • a method for virtual clothes modelling comprising:
  • said generating the avatar comprises configuring the avatar to match the one or more characteristics of the user including a skin colour of the user, and configuring the avatar to match the skin colour of the user comprises:
  • the avatar is three-dimensional and the input and output image streams are two-dimensional
  • said synthesizing the plurality of frames with the clothed avatar to generate the output image stream comprises aligning the clothed avatar with each of the plurality of frames based on a set of three-dimensional to two-dimensional point correspondences between the clothed avatar and the user in the plurality of frames.
  • the one or more characteristics further include a body size of the user.
  • superimposing the virtual clothes on the avatar comprises:
  • said applying a colour to the avatar comprises:
  • said determining the colour to apply to the avatar comprises:
  • u denotes initial colour of the avatar
  • a u denotes initial colour of the avatar
  • b u denote the / channel, a channel, and b channel of the initial colour of the avatar, respectively
  • a u , and b u denote intermediate values of the / channel, a channel and b channel of the colour of the avatar, respectively, and
  • a f , and b f denote the / channel, a channel, and b channel of the colour of the facial area, respectively,
  • ⁇ * denote said standard deviation of the / channel, a channel and b channel of the colour of the facial area, respectively,
  • a nc ' ⁇ bf denote said mean values of the / channel, a channel and b channel of the colour of the facial area, respectively, and
  • l u ' , a u ' , and b u ' denote the / channel, a channel, and b channel of the colour to apply to the avatar, respectively.
  • the facial area is a cheek area of the face of the user.
  • said aligning the clothed avatar comprises computing a projection matrix (P) for projecting each of the three-dimensional points of the avatar to a two- dimensional point on a current frame (j) of the plurality of frames corresponding to each respective two-dimensional point of the user.
  • the three-dimensional points of the avatar and the two-dimensional points of the user comprise points located at joints and along bones of a skeleton model of the avatar and the user.
  • the projection matrix (P) for the f* frame is computed by minimising residual errors according to:
  • M i denotes the three-dimensional point of the avatar
  • ⁇ ⁇ ' , ,. denotes the projection of the three-dimensional points of the avatar given the project matrix P J for the j th frame;
  • J Avatar and Juser denote points located at said joints from the skeleton models of the avatar and the user, respectively;
  • B Avatar and B User denote points located along said bones of the skeleton models of the avatar and the user, respectively;
  • ⁇ denotes a predetermined value functioning as a regularizer
  • W j and w B are predetermined weights for E j and E B , respectively.
  • the three-dimensional points of the avatar and the two-dimensional points of the user further comprise points located at facial features and silhouette contours of the avatar and the user.
  • the method is configured to provide one or more of the following options: the avatar is invisible, therefore, synthesizing the plurality of frames with the clothed avatar generates the output image stream that on display shows the user wearing the virtual clothes,
  • the avatar is visible, therefore,- synthesizing the plurality of frames with the clothed avatar generates the output image stream that on display shows the clothed avatar resembling the user wearing the virtual clothes, and
  • the image stream generated is a real-time image stream of the user trying on the virtual clothes.
  • an apparatus for virtual clothes modelling comprising:
  • a receiver module for receiving a plurality of frames of an input image stream of a user
  • an avatar generator module for generating an avatar based on one or more characteristics of the user
  • a superimposing module for superimposing virtual clothes on the avatar
  • a synthesizing module for sythesizing the plurality of frames with the clothed avatar to generate an output image stream that on display appears to resemble the user trying on the virtual clothes
  • the avatar generator module is operable to configure the avatar to match the one or more characteristics of the user including a skin colour of the user, and configuring the avatar to match the skin colour of the user comprises:
  • synthesizing the plurality of frames with the clothed avatar to generate the output image stream comprises aligning the clothed avatar with each of the plurality of frames based on a set of three-dimensional to two-dimensional point correspondences between the clothed avatar and the user in the plurality of frames
  • the one or more characteristics further includes a body size of the user.
  • the superimposing module is configured to:
  • said applying a colour to the avatar comprises:
  • said determining the colour to apply to the avatar comprises:
  • u denotes initial colour of the avatar
  • a u , and b u denote the / channel, a channel, and b channel of the initial colour of the avatar, respectively, W 201
  • a u , and b u denote intermediate values of the / channel, a channel and b channel of the colour of the avatar, respectively,
  • a f , and b f denote the / channel, a channel, and b channel of the colour of the facial area, respectively,
  • a f ' , ⁇ ° , and ⁇ x* denote said standard deviation of the / channel, a channel and b channel of the colour of the facial area, respectively,
  • ⁇ [ , ⁇ ° , and * denote said standard deviation of the / channel, a channel and b channel of the colour of the avatar, respectively,
  • u , a u ' , and b u ' denote the / channel, a channel, and b channel of the colour to apply to the avatar, respectively.
  • the facial area is a cheek area of the face of the user.
  • said aligning the clothed avatar comprises computing a projection matrix (P) for projecting each of the three-dimensional points of the avatar to a two- dimensional point on a current frame (j) of the plurality of frames corresponding to each respective two-dimensional point of the user.
  • the three-dimensional points of the avatar and the two-dimensional points of the user comprise points located at joints and along bones of a skeleton model of the avatar and the user, respectively.
  • the projection matrix (P) for the j th frame is computed by minimising residual errors according to:
  • M t denotes the three-dimensional point of the avatar
  • ⁇ ( ⁇ ] , , ) denotes the projection of the three-dimensional points of the avatar given the project matrix P j for the j th frame;
  • d Juser denote points located at said joints from the skeleton models of the avatar and the user, respectively;
  • ⁇ Avatar and Buser denote points located along said bones of the skeleton models of the avatar and the user, respectively;
  • ⁇ denotes a predetermined value functioning as a regularize ⁇ and and w B are predetermined weights for E j and E B , respectively.
  • the three-dimensional points of the avatar and the two-dimensional points of the user further comprise points located at facial features and silhouette contours of the avatar and the User, respectively.
  • the apparatus is configured to provide one or more of the following options: the avatar is invisible, therefore, synthesizing the plurality of frames with the clothed avatar generates the output image stream that on display shows the user wearing the virtual clothes,
  • the avatar is visible, therefore, synthesizing the plurality of frames with the clothed avatar generates the output image stream that on display shows the clothed avatar resembling the user wearing the virtual clothes, and
  • synthesizing the plurality of frames with the clothed avatar generates the output image stream that on display shows the body portion of the avatar wearing the virtual clothes while a head portion of the user remains in the output image stream aligned on the body portion of the avatar.
  • the output image stream generated is a real-time image stream of the user trying on the virtual clothes.
  • a system for virtual clothes modelling comprising:
  • an image capturing unit communicatively coupled to the apparatus for capturing an image stream of the user and sending the image stream to the apparartus
  • a display communicatively coupled to the image capturing unit for displaying the image stream generated by the apparatus to the user.
  • a computer program product embodied in a computer-readable storage medium, comprising instructions executable by a computer processor to perform a method for virtual clothes modelling comprising:
  • said generating the avatar comprises configuring the avatar to match the one or more characteristics of the user including a skin colour of the user, and configuring the avatar to match the skin colour of the user comprises:
  • the avatar is three-dimensional and the input and output image streams are two-dimensional
  • said synthesizing the plurality of frames with the clothed avatar to generate the output image stream comprises aligning the clothed avatar with each of the plurality of frames based on a set of three- dimensional to two-dimensional point correspondences between the clothed avatar and the user in the plurality of frames.
  • Fig. 1 depicts a flow diagram generally illustrating a method for virtual clothes modelling according to an example embodiment of the present invention
  • Fig. 2 depicts a schematic block diagram of an example system including an example apparatus for virtual clothes modelling configured to perform the method shown in Fig. 1 according to an example embodiment of the present invention
  • Figs. 3A to 3C depict three exemplary try-on scenarios according to embodiments of the present invention
  • Fig. 4 depicts a block diagram illustrating a detailed architecture of an apparatus for virtual clothes modelling according to an example embodiment of the present invention
  • Fig. 5A depicts an example generic avatar generated by the generic avatar generator module
  • Figs. 5B to 5F depict example customized avatars configured to have various sizes based on the body size of different users
  • Figs 6A and 6B depict the clothed avatars generated for two different users, respectively;
  • Figs. 7A to 7C illustrate a technique for extracting a facial area of interest to be used as a suitable exemplar patch for colour transfer to the avatar
  • Figs. 8A to 8C illustrate an example of aligning a 3D avatar with a 2D image of the user according to an example embodiment of the present invention
  • Figs. 9A and 9B illustrate another example of aligning a 3D avatar with a 2D image of the user according to another example embodiment of the present invention
  • Fig. 10 illustrates a conventional general purpose computer which may be used to implement the method described herein in example embodiments of the present invention
  • Figs. 11 to 13 illustrate exemplary output images for various poses of a first; a second and a third scenario produced by the system according to example embodiments of the present invention.
  • Embodiments of the present invention seek to provide a method, an apparatus and a system for virtual clothes modelling which can be used, e.g., for virtual try-on of clothes by a user/customer in retail shopping. Details of the method, apparatus and system according to exemplary embodiments of the present invention will be described hereinafter.
  • apparatus for performing the operations of the methods.
  • Such apparatus may be specially constructed for the required purposes, or may comprise a general purpose computer or other device selectively activated or reconfigured by a computer program stored in the computer.
  • the algorithms and displays presented herein are not inherently related to any particular computer or other . apparatus.
  • Various general purpose machines may be used with programs in accordance with the teachings herein.
  • the construction of more specialized apparatus to perform the required method steps may be appropriate.
  • the structure of an exemplary conventional general purpose computer will be described later with reference to Fig. 10.
  • the present specification also implicitly discloses a computer program, in that it would be apparent to the person skilled in the art that the individual steps of the method described herein may be put into effect by computer code.
  • the computer program is not intended to be limited to any particular programming language and implementation thereof. It will be appreciated that a variety of programming languages and coding thereof may be used to implement the teachings of the disclosure contained herein.
  • the computer program is not intended to be limited to any particular control flow. There are many other variants of the computer program, which can use different control flows without departing from the spirit or scope of the invention.
  • Such a computer program may be stored on any computer readable medium.
  • the computer readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a general purpose computer.
  • the computer program when loaded and executed on such a general-purpose computer effectively results in an apparatus that implements the steps of the method described herein.
  • a module is a functional hardware unit designed for use with other components or modules.
  • a module may be implemented using discrete electronic components, or it can form a portion of an entire electronic circuit such as an Application Specific Integrated Circuit (ASIC). Numerous other possibilities exist.
  • ASIC Application Specific Integrated Circuit
  • Fig. 1 depicts a flow diagram generally illustrating a method 100 for virtual clothes modelling according to an example embodiment of the present invention.
  • the method 100 comprises a step 104 of receiving a plurality of frames of an input image stream of a user, a step 108 of generating an avatar based on one or more characteristics of the user, a step 112 of superimposing virtual clothes on the avatar, and a step 116 of synthesizing the plurality of frames with the clothed avatar to generate an output image stream that on display appears to resemble the user trying on the virtual clothes.
  • a step 104 of receiving a plurality of frames of an input image stream of a user a step 108 of generating an avatar based on one or more characteristics of the user, a step 112 of superimposing virtual clothes on the avatar, and a step 116 of synthesizing the plurality of frames with the clothed avatar to generate an output image stream that on display appears to resemble the user trying on the virtual clothes.
  • the user is a person/customer who is trying on the virtual clothes (virtual try-on) and wishes to see how the clothes would look on himself/herself when worn.
  • the virtual clothes may be digital versions of the physical clothes at a retail shop and the user may indicate the physical clothes that he/she are interested in trying and tries on the corresponding virtual clothes instead of the physical clothes.
  • the user may browse through a corpal collection of virtual clothes (e.g., online shopping) and tries on any of the virtual clothes of interest.
  • the image stream may be generated by an image capturing unit such as a digital camera/sensor, preferably a RGB-D camera/sensor such as MICROSOFT KINECT produced and made commercially available by MICROSOFT.
  • the avatar is a digital three-dimensional object representing or resembling the user or a part thereof.
  • the avatar is configured to match one or more characteristics of the user, including a skin colour of the user.
  • configuring the avatar to match the skin colour of the user includes locating facial features of the user, extracting a colour of a facial area of the user, and applying a. colour to the avatar computed based on the colour of the facial area.
  • the avatar may also be configured to match a body size of the user. The manner in which the avatar is configured will be described in further detail later below.
  • the avatar is three- dimensional and the input and output image streams are two-dimensional, whereby synthesizing the plurality of frames with the clothed avatar to generate the output image stream includes aligning the clothed avatar with each of the plurality of frames based on a set of three-dimensional to two-dimensional point correspondences between the clothed avatar and the user in the plurality of frames. This alignment technique will be described in further detail later below.
  • the apparatus 200 comprises a receiver module 204 for receiving a plurality of frames of an input image stream 203 of a user 202, an avatar generator module 208 for generating an avatar based on one or more characteristics of the user 202, a superimposing module 212 for superimposing virtual clothes on the avatar, a synthesizing module 216 for synthesizing the plurality of frames with the clothed avatar to generate an output image stream 220 that on display appears to resemble the user 202 trying on the virtual clothes.
  • the apparatus 200 for virtual clothes modelling may be specially constructed for the required purposes, or may be implemented in a general purpose computer or other devices.
  • each of the above modules may be software modules realised by a computer program or a set of instructions executable by a computer processor 1018 to perform the required functions, or may be hardware modules being a functional hardware unit designed to perform the required functions.
  • a combination of hardware and software modules may be implemented in the apparatus 200.
  • Fig. 2 also schematically illustrates a system 250 for virtual clothes modelling comprising the apparatus 200, an image capturing unit 254 communicatively coupled to the apparatus 200 (in particular, the receiver module 204) for capturing an input image stream 203 of the user 202 and sending the image stream 203 to the apparatus 200, and a display unit 258 communicatively coupled to the apparatus 200 (in particular, the synthesizing module 216) for displaying the output image stream 220 generated by the apparatus 200 to the user 202.
  • the image capturing unit 254 is digital camera/sensor and preferably a RGB-D camera/sensor such as MICROSOFT KINECT produced and made commercially available by MICROSOFT.
  • the display unit 258 can be any device with a display screen capable of receiving and displaying an image stream, such as but not limited to a TV, a projector, or a computer device with a display screen.
  • Figs. 3A to 3C illustrate three exemplary try-on scenarios preferred in the embodiments of the present invention.
  • a first try-on scenario illustrated in Fig. 3A the virtual clothes 304 are arranged so as to appear on a body of the user 202.
  • the virtual clothes 304 are superimposed on the avatar of the user 202 but the avatar is made invisible in the output image 220 thus providing the effect that the user is wearing the virtual clothes.
  • a second try-on scenario illustrated in Fig. 3B the avatar wearing the virtual clothes 304 (i.e., clothed avatar 308) is visible and shown entirely in the output image, including the body and head portions.
  • a headless clothed avatar 312 i.e., the clothed avatar 308 as shown in Fig. 3B but without the head portion 309 is shown and is aligned/superimposed with an image of the user's head portion. That is, in the output image 220 generated, the body portion 312 of the clothed avatar 308 is shown while the head portion 309 of the avatar 308 is hidden. Therefore, the head portion 316 of the user 202 remains in the output image 220 aligned on the body portion 312 of the clothed avatar 308.
  • the virtual clothes 304 are simulated on the avatar 308 and follow the user's 202 movements in real-time to provide an enhanced virtual try-on experience close to in a real life situation where the user stands in front of a mirror.
  • the apparatus 200 or the system 250 may be configured to provide one or more of the above-described try-on scenarios.
  • An apparatus 400 will now be described according to a preferred embodiment with reference to Fig. 4 which efficiently integrates the above three try-on scenarios together to advantageously provide the user or operator with the options of selecting one or more of the try-on scenarios for the virtual try-on experience.
  • the apparatus 200 may be configured to provide more or less try-on options/scenarios without deviating from the scope of present invention.
  • Fig. 4 depicts a block diagram illustrating an architecture of the apparatus 400 according to the example embodiment.
  • the apparatus 400 comprises an avatar generator module 404 including a generic avatar generator module 406 for generating a generic/original avatar, an avatar body customization module 408 for customizing a body size of the avatar based on the body size of the user 202, and a skin tone matching module 410 for customizing the colour of the avatar based on a skin colour of the user 202.
  • the apparatus 400 also comprises a receiver module 420 for receiving a plurality of frames of an input image stream 203 of the user 202.
  • the receiver module 420 includes a body measurement module 422 for determining a body size of the user 202 based on the input image 203 of the user 202 received from the image capturing unit 426 and sending the measurements to the avatar body customization module 408, and a user segmentation module 424 for extracting a skin area of the input image of the user 202 and sending the extract skin area to the skin tone matching module 410. Accordingly, a customized avatar 412 may be generated based on a body size and a skin colour of the user 202.
  • the apparatus 400 further comprises a superimposing and synthesizing module 430 for superimposing virtual clothes on the customized avatar, and synthesizing the plurality of frames of the input image stream 203 with the clothed avatar 308 to generate an output image stream 220 that on display appears to resemble the user trying on the virtual clothes in real time.
  • the superimposing and synthesizing " module 430 comprises a user pose detection module 432 for detecting a pose of the user 202, a two- dimensional to three-dimensional (2D-3D) alignment module 434 for aligning the customized avatar from the avatar generator module 404 with the pose of the user 202, and a clothes simulation module 436 for superimposing the virtual clothes 304 to the customized avatar 412.
  • the output image stream 220 may be generated by a rendering module 438 that on display appears to resemble the user trying on the virtual clothes in real time.
  • the image capturing unit 426 may also capture a background image 320 behind the user 202 and the superimposing and synthesizing module 430 may comprise a background reconstruction module 442 for reconstructing the background image 320 in the output image stream 220.
  • the superimposing and synthesizing module 430 may further comprise a first avatar modification module 446 for hiding the avatar (i.e., making the avatar invisible) as described above in the first try-on scenario illustrated in Fig. 3A such that the virtual clothes 304 appear to the user 202 to be on his/her body, and a second avatar modification module 450 for hiding a part of the avatar 308, in particular, the head portion 309 as described above in the third try-on scenario illustrated in Fig. 3C such that the body portion 312 of the clothed avatar 308 is shown while the head portion 316 of the user 202 remains in the output image 220 aligned on the body portion 312 of the clothed avatar 308.
  • a first avatar modification module 446 for hiding the avatar (i.e., making the avatar invisible) as described above in the first try-on scenario illustrated in Fig. 3A such that the virtual clothes 304 appear to the user 202 to be on his/her body
  • a second avatar modification module 450 for hiding a part of the avatar 308, in
  • the output image stream 220 from the rendering module 438 does not need to be modified as shown in Fig. 4 since the entire clothed avatar 308 resembling the user 202 is displayed.
  • the virtual clothes 304 on the avatar follow the user's movements, therefore giving the user 202 a perception of trying on the virtual clothes 304 in front of a mirror in real time.
  • Fig. 5A illustrates an example generic/original avatar generated by the generic avatar generator module 406.
  • Figs. 5B to 5F illustrate example customized/modified avatars configured to have various sizes based on the body size of different users 202.
  • the size of the customized avatar is configured so as to match to body size of the user 202, i.e., the same size or in proportion thereto.
  • A. Niswar, I. Khan, and F. Farbiz "Avatar Customization based on Human Body Measurements", Proceeding of the SIGGRAPH Asia 2012 Posters, the contents of which are hereby incorporated by cross-reference.
  • a scaling technique is implemented which scales different sections of the virtual clothes separately and then suitably translating them to retain the connectivity of the different sections. This is because scaling of the virtual clothes to appear to fit well on the user's body is an important requirement for a realistic virtual try-on. In this regard, a global scaling generally would not provide a satisfactory result since different users usually have different body shapes.
  • superimposing the virtual clothes 304 on the avatar to produce a clothed avatar 308 comprises segmenting the virtual clothes 304 into a plurality of sections, scaling each section of the virtual clothes 304 to match a size of each corresponding section of the avatar, and aligning the scaled virtual clothes 304 with respect to the avatar to superimpose the virtual clothes 304 on the avatar.
  • the virtual clothes 304 are scaled to fit on the avatar to generate a clothed avatar 308.
  • the clothed avatar 308 may be generated in a setup or an offline stage (e.g., each of the plurality of virtue clothes 304 is pre-fitted/superimposed onto a respective avatar before the user virtual try-on to produce a plurality of clothed avatars 308). Therefore, when the avatar is customized according to the size of the user during the user virtual try-on, the virtual clothes 304 on the avatar is also automatically scaled proportionally since the virtual clothes 304 have already been fitted on the avatar.
  • Figs 6A and 6B show the clothed avatars 308 generated for two different users 202, respectively.
  • the clothed avatar 308 is customised according to the size of the user 202 and thus the virtual clothes 304 on the user 202 are also scaled proportionally.
  • the method of transferring the user's skin tone to the avatar 308 generally comprises the following three steps.
  • a first step the facial features of the user 202 are located using a facial feature extraction technique such as the Active Shape Model (ASM) technique.
  • ASM Active Shape Model
  • the ASM technique is known in the art and thus will not be described in detail herein.
  • piecewise linear curves are used to represent facial areas of the user 202 and one or more facial areas of interest are extracted.
  • the facial area of interest is the cheek area 712 of the user 202 and the cheek patches 716 are extracted from the image of the user 202 as shown in Figs. 7A to 7C and described in further detail below.
  • a global colour transfer method is applied to transfer the colour of extracted facial area to the avatar 308.
  • a common technique to acquire face skin patches is to transfer RGB image of the user to the YCbCr colour space and threshold the chrominance component.
  • the pixels of the image with their chromatic value falling in a certain range are classified as skin-color pixels.
  • some background clothes or hair may have similar colours as the face.
  • the lips and the areas between the eyebrows and the upper eyelids may be misclassified as skin areas which are not ideal/suitable skin patches to be used for skin tone transfer.
  • Another problem in skin tone extraction is that the appearance of the skin is usually quite different under different viewing and lighting conditions.
  • the cheek area 712 is the largest flat skin area on the face 704, and is least affected by shadows from nose, forehead and hair.
  • the cheek area 712 generally reflects the illuminated light uniformly compared to other areas on the face 704. Therefore, the cheek area 712 is chosen as a suitable or an ideal exemplar patch 716.
  • facial features of the face 704 are detected using ASM, in which face shape can be formed, with a set of landmarks that represent distinguishable points such as eye pupils, nose and lips, etc.
  • piecewise linear curves 708 are used to represent facial areas 704 of the user as shown in Fig. 7B and then cheek patches 716 are extracted from the image of the face 704 as shown in Fig. 7C.
  • a total of 76 landmarks are marked on the image of the face 704, and joined by lines 708 in a predetermined order so as to obtain a piecewise linear representation of the face 704.
  • Fig. 7B shows 20 landmarks and their connecting lines 708 used to enclose the left and right cheek areas 712.
  • Fig. 7C shows the two cheek patches 716 extracted from the image of the face 704.
  • the RGB colour components of the extracted face skin patches are decomposed into lightness (/) and chrominance layers (a first colour axis (a) channel, e.g., in the range from cyan colour to magenta/red colour and a second colour axis (b) channel, e.g., in the range of blue to yellow colour) by converting them to the CIELab colour space.
  • the CIELab colour space is known in the art and thus will not be described in detail herein.
  • the " ⁇ channel is the lightness layer
  • the "a" and "b” channels are the colour layers.
  • the CIELAB colour space is chosen because it performs better than other colour spaces in separating lightness from colour, and is approximately perceptually uniform.
  • the colour (i.e., the initial skin colour) of the avatar 308 is also converted into the CIELAB colour space, having the lightness (/) channel, the first colour axis (a) channel and the second colour axis (b) channel.
  • the above computed intermediate values of the skin colour of the avatar are then scaled with weighting factors (i.e.,— ⁇ ,— - , and—r ) determined by the respective standard deviations, and the mean/average values of the user's face skin are then added to the scaled intermediate values to generate new 7", "a", and "b" colour components to be converted to the RGB colour space and applied to the avatar 308.
  • weighting factors i.e.,— ⁇ ,— - , and—r
  • the new skin colour components ( l u ' , u ' ,b u ' ) for the avatar 308 are converted back to the RGB colour space to give a new UV map which has a similar colour tone as the face skin (i.e., the cheek area) of the user 202.
  • a technique which seeks to efficiently align a 3D avatar 308 with a user's image in real-time by using a RGB-D sensor, such as MICROSOFT KINECT.
  • a set of 3D-2D i.e., 3D to 2D
  • the user 202 may stand and look straight at a display with one of a number of predetermined poses, such as a T- pose, at the beginning.
  • Two skeleton models 804, 808 are then employed and aligned with respect to each other, i.e., a skeleton model 808 for the avatar 810 which is preferably already built-in in the 3D avatar generated, and a skeleton model 804 for the user 202 which may be obtained or generated based on the image from the RGB-D camera.
  • homogeneous vectors are used to represent the 2D image point (m) and the 3D point (M) as follows:
  • the parameters P J are estimated for the current incoming frame j that minimize the sum of squared distances between the projections of the 3D points of the avatar 308 and the corresponding 2D image points of the user 202.
  • P J is estimated through minimizing the actual residual errors and can be determined as follows: where E, and E R are defined as follo
  • Equations (6) and (7) ⁇ ( ⁇ ⁇ ⁇ ) denotes the projection of 3D avatar points given the parameters P J in the j th incoming/current frame. As shown in Figures 8A and 8B,
  • J Avatar anc! J user are ⁇ ° ⁇ ⁇ * ⁇ 830, 828 located at the joints of the skeletons 808, 804 of the avatar 308 and the user 202, respectively.
  • B Avatar and B User are points 838, 834 located along the bones of the skeletons 808, 804 of the avatar 810 and the user 202, respectively.
  • only the sampling points on the upper arms, the shoulder and the spine are used due to their stable poses captured from the RGB-D camera, and the different hip positions of the two skeletons 808, 804.
  • the number of sampling points may be reduced or increased as appropriate. functions as a regularizer which is used to avoid, over-fitting and ⁇
  • Equation (5) can be solved using the Levenberg-Marquardt (LM) minimization method.
  • Fig. 8C shows an example of the initial alignment based on the above-described technique.
  • the LM minimization method is known in the art and thus will not be described in detail herein.
  • ; , /W ⁇ between the 3D avatar 308 and the incoming frame j of the user 202 are established.
  • feature points and silhouette contours are used to establish more 3D-2D point correspondences.
  • contour correspondences are established between the projected surface of the avatar 308 and the user's 2D image silhouette by searching for the 3D vertices of the avatar 308 whose projections are nearest to user's contour points.
  • the image of the user 202 from the RGB- D camera 254 is segmented and the user silhouette contours are then extracted.
  • Various known methods may be used to extract the user silhouette contours from an image and thus will not be described in detail herein.
  • a machine learning-based keypoint recognition method is used, in which matching is formulated as an image classification problem, where each class is trained with all the possible appearances that an image feature may take under large perspective and scale variations.
  • a machine learning-based keypoint recognition method reference is made to V. Lepetit and P. Fua, "Keypoint recognition using randomized trees", IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 9, pp. 1465-1497, 2006, the contents of which are hereby incorporated by cross- reference.
  • a major advantage of the learning-based matching method is that it is very fast for the image tracking problems and therefore suitable for the applications that require real-time performance.
  • E c and E F are same as E y and E B , but relate to silhouette contours and feature points instead of joints and bones, respectively.
  • P j using Equation (8) and the robust M-estimators.
  • Figs. 9A and 9B show an example of the alignment based on the above-described technique further involving the feature points and silhouette contours.
  • the method of the example embodiments described herein can be implemented on a computer system 1000, schematically shown in Fig. 10. It may be implemented as software, such as a computer program being executed within the computer system 1000, and instructing the computer system 1000 to conduct the method of the example embodiment.
  • the computer system 1000 comprises a computer module 1002, input modules such as a keyboard 1004 and mouse 1006 and a plurality of output devices such as a display 1008, and printer 1010.
  • the computer module 1002 is connected to a computer network 1012 via a suitable transceiver device 1014, to enable access to e.g. the Internet or other network systems such as Local Area Network (LAN) or Wide Area Network (WAN).
  • LAN Local Area Network
  • WAN Wide Area Network
  • the computer module 1002 in the example includes a processor 1018, a Random Access Memory (RAM) 1020 and a Read Only Memory (ROM) 1022.
  • the computer module 1002 also includes a number of Input/Output (I/O) interfaces, for example I/O interface 1024 to the display 1008, and I/O interface 1026 to the keyboard 1004.
  • I/O interface 1024 to the display 1008
  • I/O interface 1026 to the keyboard 1004.
  • the components of the computer module 1002 typically communicate via an interconnected bus 1028 and in a manner known to the person skilled in the relevant art.
  • the application program may be supplied to the user of the computer system 1000 encoded on a data storage medium such as a CD/DVD-ROM or flash memory carrier and read utilising a corresponding data storage medium drive of a data storage device 1030.
  • the application program may also be supplied to the user of the computer system 1000 via a network such as the Internet, LAN, or WAN.
  • the application program is read and controlled in its execution by the processor 1018.
  • Intermediate storage of program data may be accomplished using RAM 1020.
  • a computer program product embodied in a computer-readable storage medium 1030, comprising instructions executable by a computer processor 1018 to perform a method for virtual clothes modelling comprising receiving a plurality of frames of an input image stream of a user, generating an avatar based on one or more characteristics of the user, superimposing virtual clothes on the avatar, and synthesizing the plurality of frames with the clothed avatar to generate an output image stream that on display appears to resemble the user trying on the virtual clothes, wherein generating the avatar comprises configuring the avatar to match the one or more characteristics of the user including a skin colour of the user, and configuring the avatar to match the skin colour of the user comprises locating facial features of the user, extracting a colour of a facial area of the user, and applying a colour to the avatar computed based on the colour of the facial area, and/or wherein the avatar is three-dimensional and the image stream is two-dimensional, and said synthesizing the plurality of frames with the clothed avatar to generate the image stream
  • the virtual try-on apparatus 200 of the system 250 described herein was implemented in a 2.53GHz Intel Xeon(R) with 24GB RAM running Visual Studio 2010.
  • a MICROSOFT KINECT camera was used as the RGB-D sensor for user image collection, pose detection, body measurements, user segmentation and user's face skin color detection.
  • the user 202 stands in front of a display with a standard or predetermined pose (such as a T-pose).
  • the system 250 automatically establishes the relevant 3D-2D correspondences for the keyframe according to the techniques as described above.
  • various sizes of the user's body e.g., the user's height, shoulder width, waist height and arm lengths
  • the avatar is then customized according to the extracted size and the skin color.
  • the user 202 may also key in to the system 250 additional body sizes for a more accurate avatar customization.
  • the user 202 can select his/her favourite virtual clothes for virtually trying-on.
  • the selected virtual clothes will be superimposed to the customized avatar and synthesized with the input image stream 203 of the user 202 in real-time such that on display appears to resemble the user 202 trying on the virtual clothes 304 and following the user's movements.
  • Figs. 11 to 13 illustrate exemplary output images for various poses of the first, second and third scenarios produced by the system 250, respectively.
  • Fig. 11 illustrates the first try-on scenario where the virtual clothes 304 are arranged so as to appear on a body of the user 202 (the virtual clothes 304 are superimposed on the avatar of the user 202 but the avatar is made invisible thus providing the effect that the user is wearing the virtual clothes)
  • Fig. 12 illustrates the second try-on scenario where the avatar 308 wearing the virtual clothes 304 (i.e., clothed avatar 308) is visible and shown entirely, including the body and head portions (the user 202 is able to see the clothed avatar 308 resembling the user 202 wearing the virtual clothes 304),
  • Fig. 11 illustrates the first try-on scenario where the virtual clothes 304 are arranged so as to appear on a body of the user 202 (the virtual clothes 304 are superimposed on the avatar of the user 202 but the avatar is made invisible thus providing the effect that the user is wearing the
  • FIG. 13 illustrates the third try-on scenario where a headless clothed avatar 312 (i.e., the clothed avatar 308 as shown in Fig. 3B but without the head portion) is shown and is superimposed with an image of the user's head portion.
  • a headless clothed avatar 312 i.e., the clothed avatar 308 as shown in Fig. 3B but without the head portion
  • the method 100 and system 250 for virtual clothes modelling described herein according to the example embodiments are efficient and sufficiently accurate in aligning the virtual clothes 304 with the user's body.
  • This * provides an enhanced virtual clothing experience.
  • This also advantageously avoids or at least minimise the need for physical try-ons, which may thus result in increased sales by a retail shop.
  • Additional advantages include the possibility of side-by-side comparison of various clothes and simultaneous viewing of outfits from different angles for further enhancing user experience.
  • Interactive virtual try-on can also be an interesting feature for advertisement and/or attracting crowd.

Landscapes

  • Engineering & Computer Science (AREA)
  • Architecture (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé pour la modélisation virtuelle de vêtements comprenant la réception d'une pluralité de trames d'un flux d'images d'entrée d'un utilisateur, la génération d'un avatar d'après une ou plusieurs caractéristiques de l'utilisateur, la superposition de vêtements virtuels sur l'avatar et la synthèse de la pluralité de trames avec l'avatar vêtu pour générer un flux d'images de sortie qui apparaît sur l'affichage comme ressemblant à l'utilisateur essayant les vêtements virtuels, la génération de l'avatar comprenant la configuration de l'avatar pour qu'il corresponde à ladite ou auxdites caractéristiques de l'utilisateur y compris une couleur de peau de l'utilisateur, et la configuration de l'avatar pour qu'il corresponde à la couleur de peau de l'avatar comprenant la localisation des traits du visage de l'utilisateur, l'extraction d'une couleur d'une zone faciale de l'utilisateur, et l'application d'une couleur à l'avatar réalisé d'après la couleur de la zone du visage, et/ou l'avatar étant en trois dimensions et les flux d'images d'entrée et de sortie étant en deux dimensions, et la synthèse de la pluralité de trames avec l'avatar vêtu pour générer le flux d'images de sortie comprenant l'alignement de l'avatar vêtu avec chacune de la pluralité de trames d'après un ensemble de correspondances de points en trois dimensions et deux dimensions entre l'avatar vêtu et l'utilisateur dans la pluralité de trames. L'invention concerne également un appareil et un système de modélisation virtuelle de vêtements.
PCT/SG2013/000496 2012-11-22 2013-11-22 Procédé, appareil et système pour la modélisation virtuelle de vêtements WO2014081394A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
SG11201503974SA SG11201503974SA (en) 2012-11-22 2013-11-22 Method, apparatus and system for virtual clothes modelling

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SG2012085858 2012-11-22
SG201208585-8 2012-11-22

Publications (1)

Publication Number Publication Date
WO2014081394A1 true WO2014081394A1 (fr) 2014-05-30

Family

ID=50776428

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SG2013/000496 WO2014081394A1 (fr) 2012-11-22 2013-11-22 Procédé, appareil et système pour la modélisation virtuelle de vêtements

Country Status (2)

Country Link
SG (1) SG11201503974SA (fr)
WO (1) WO2014081394A1 (fr)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160086202A1 (en) * 2014-09-24 2016-03-24 National Tsing Hua University Method and electronic device for rating outfit
WO2016064121A1 (fr) * 2014-10-21 2016-04-28 삼성전자 주식회사 Dispositif d'essayage virtuel et procédé d'essayage virtuel associé
WO2016086266A1 (fr) 2014-12-05 2016-06-09 Myfiziq Limited Imagerie d'un corps
CN106339390A (zh) * 2015-07-09 2017-01-18 中兴通讯股份有限公司 基于人体特征数据的匹配方法及装置
WO2018052367A1 (fr) * 2016-09-14 2018-03-22 Ayne Pte. Ltd. Système et procédé de numérisation et d'essayage de vêtement
CN109771947A (zh) * 2019-01-31 2019-05-21 网易（杭州）网络有限公司 游戏角色的换装方法、装置、计算机存储介质和电子设备
WO2019147359A1 (fr) * 2018-01-27 2019-08-01 Walmart Apollo, Llc Système de conception de vêtement augmenté
US10380794B2 (en) 2014-12-22 2019-08-13 Reactive Reality Gmbh Method and system for generating garment model data
US10388050B2 (en) * 2015-03-31 2019-08-20 Seiren Co., Ltd. System and method for creating customized clothing that combines clothing shapes, clothing patterns, dolls and the user's head image
WO2021112888A1 (fr) * 2019-12-04 2021-06-10 Roblox Corporation Détection d'objets virtuels contrefaits
US11321856B1 (en) 2020-12-18 2022-05-03 Roblox Corporation Detection of inauthentic virtual objects
WO2022173579A1 (fr) * 2021-02-11 2022-08-18 Qualcomm Incorporated Capture d'image dans des environnements de réalité étendue

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6546309B1 (en) * 2000-06-29 2003-04-08 Kinney & Lange, P.A. Virtual fitting room
EP1959394A2 (fr) * 2005-11-15 2008-08-20 Reyes Infografica, S.L. Procede de production et d'utilisation d'une cabine d'essayage de vetements virtuelle et systeme associe
WO2012110828A1 (fr) * 2011-02-17 2012-08-23 Metail Limited Procédés et systèmes mis en œuvre par ordinateur pour créer des modèles corporels virtuels pour visualisation de l'ajustement d'un vêtement
US20120287122A1 (en) * 2011-05-09 2012-11-15 Telibrahma Convergent Communications Pvt. Ltd. Virtual apparel fitting system and method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6546309B1 (en) * 2000-06-29 2003-04-08 Kinney & Lange, P.A. Virtual fitting room
EP1959394A2 (fr) * 2005-11-15 2008-08-20 Reyes Infografica, S.L. Procede de production et d'utilisation d'une cabine d'essayage de vetements virtuelle et systeme associe
WO2012110828A1 (fr) * 2011-02-17 2012-08-23 Metail Limited Procédés et systèmes mis en œuvre par ordinateur pour créer des modèles corporels virtuels pour visualisation de l'ajustement d'un vêtement
US20120287122A1 (en) * 2011-05-09 2012-11-15 Telibrahma Convergent Communications Pvt. Ltd. Virtual apparel fitting system and method

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160086202A1 (en) * 2014-09-24 2016-03-24 National Tsing Hua University Method and electronic device for rating outfit
WO2016064121A1 (fr) * 2014-10-21 2016-04-28 삼성전자 주식회사 Dispositif d'essayage virtuel et procédé d'essayage virtuel associé
US10152829B2 (en) 2014-10-21 2018-12-11 Samsung Electronics Co., Ltd. Virtual fitting device and virtual fitting method thereof
WO2016086266A1 (fr) 2014-12-05 2016-06-09 Myfiziq Limited Imagerie d'un corps
CN106999107A (zh) * 2014-12-05 2017-08-01 麦费兹克公司 身体成像
EP3226763A4 (fr) * 2014-12-05 2018-07-04 Myfiziq Limited Imagerie d'un corps
US10380794B2 (en) 2014-12-22 2019-08-13 Reactive Reality Gmbh Method and system for generating garment model data
US10388050B2 (en) * 2015-03-31 2019-08-20 Seiren Co., Ltd. System and method for creating customized clothing that combines clothing shapes, clothing patterns, dolls and the user's head image
CN106339390A (zh) * 2015-07-09 2017-01-18 中兴通讯股份有限公司 基于人体特征数据的匹配方法及装置
WO2018052367A1 (fr) * 2016-09-14 2018-03-22 Ayne Pte. Ltd. Système et procédé de numérisation et d'essayage de vêtement
WO2019147359A1 (fr) * 2018-01-27 2019-08-01 Walmart Apollo, Llc Système de conception de vêtement augmenté
CN109771947A (zh) * 2019-01-31 2019-05-21 网易（杭州）网络有限公司 游戏角色的换装方法、装置、计算机存储介质和电子设备
WO2021112888A1 (fr) * 2019-12-04 2021-06-10 Roblox Corporation Détection d'objets virtuels contrefaits
US11238316B2 (en) 2019-12-04 2022-02-01 Roblox Corporation Detection of counterfeit virtual objects
US11731053B2 (en) 2019-12-04 2023-08-22 Roblox Corporation Detection of counterfeit virtual objects
US11321856B1 (en) 2020-12-18 2022-05-03 Roblox Corporation Detection of inauthentic virtual objects
US11715219B2 (en) 2020-12-18 2023-08-01 Roblox Corporation Detection of inauthentic virtual objects
WO2022173579A1 (fr) * 2021-02-11 2022-08-18 Qualcomm Incorporated Capture d'image dans des environnements de réalité étendue
US11644890B2 (en) 2021-02-11 2023-05-09 Qualcomm Incorporated Image capturing in extended reality environments

Also Published As

Publication number Publication date
SG11201503974SA (en) 2015-06-29

Similar Documents

Publication Publication Date Title
WO2014081394A1 (fr) Procédé, appareil et système pour la modélisation virtuelle de vêtements
US11423556B2 (en) Methods and systems to modify two dimensional facial images in a video to generate, in real-time, facial images that appear three dimensional
Habermann et al. Deepcap: Monocular human performance capture using weak supervision
US9348950B2 (en) Perceptually guided capture and stylization of 3D human figures
Achenbach et al. Fast generation of realistic virtual humans
Zhang et al. Detailed, accurate, human shape estimation from clothed 3D scan sequences
CN109690617B (zh) 用于数字化妆镜的系统和方法
Ichim et al. Dynamic 3D avatar creation from hand-held video input
Yuan et al. A mixed reality virtual clothes try-on system
JP6389888B2 (ja) 鏡における仮想化の装置、システム、及び方法
US9147207B2 (en) System and method for generating image data for on-line shopping
JP5976103B2 (ja) プレイヤーの外観を反映するアバターの生成
JP2020526809A (ja) 仮想顔化粧の除去、高速顔検出およびランドマーク追跡
Tsoli et al. Breathing life into shape: Capturing, modeling and animating 3D human breathing
KR20180100476A (ko) 이미지 및 뎁스 데이터를 사용하여 3차원(3d) 인간 얼굴 모델을 발생시키는 가상 현실 기반 장치 및 방법
Zhao et al. 3-D reconstruction of human body shape from a single commodity depth camera
US20150235305A1 (en) Garment modeling simulation system and process
JP2010507854A (ja) ビデオイメージシーケンスを仮想シミュレーションする方法及び装置
JP2002133446A (ja) 顔画像処理方法およびシステム
KR20160098560A (ko) 동작 분석 장치 및 방법
Wenninger et al. Realistic virtual humans from smartphone videos
CN116523579A (zh) 一种显示设备、虚拟试衣系统及方法
Richter et al. Real-time reshaping of humans
Danieau et al. Automatic generation and stylization of 3d facial rigs
CN114373044A (zh) 生成脸部三维模型的方法、装置、计算设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13856184

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 13856184

Country of ref document: EP

Kind code of ref document: A1