WO2010042990A1 - Commercialisation en ligne de produits pour le visage à l'aide d'un suivi du visage en temps réel - Google Patents

Commercialisation en ligne de produits pour le visage à l'aide d'un suivi du visage en temps réel Download PDF

Info

Publication number
WO2010042990A1
WO2010042990A1 PCT/AU2009/001360 AU2009001360W WO2010042990A1 WO 2010042990 A1 WO2010042990 A1 WO 2010042990A1 AU 2009001360 W AU2009001360 W AU 2009001360W WO 2010042990 A1 WO2010042990 A1 WO 2010042990A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
face
image
rendering
eyewear
Prior art date
Application number
PCT/AU2009/001360
Other languages
English (en)
Inventor
Timothy James Henry Edwards
Nicholas John Langdale-Smith
Original Assignee
Seeing Machines Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from AU2008905350A external-priority patent/AU2008905350A0/en
Application filed by Seeing Machines Limited filed Critical Seeing Machines Limited
Publication of WO2010042990A1 publication Critical patent/WO2010042990A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality

Definitions

  • the present invention relates to the field of real time facial tracking and, in particular, to the online marketing of facial products. Background
  • Unites States Patent 7016824 to Waupotitsch et al. discloses systems and methods for an interactive try-on platform for eyeglasses for online commerce.
  • a fully-textured 3D face model and a 3D representation of a selected pair of glasses are used.
  • the visualisation system is not based on live images.
  • the 3D face model is generated in advance.
  • an interactive system comprising: a video capture device for capturing a video image of the face of a system user; a rendering system for rendering a synthetically generated 3D model of eyewear on the face of the video image; and a display system for displaying to the user a view of the rendered image.
  • the system operates in real time.
  • the rendering system preferably can include tracking of the 3D location and orientation of a user's face.
  • an interactive system comprising: a video capture device for capturing a video image of the face of a system user; a rendering system for rendering a synthetically generated 3D model of an object a user may wear on the face of the video image; and display system for displaying to the user a view of the rendered image.
  • the object can in some embodiments, include one of a necklace, sunglasses, hair style, cosmetics, cosmetic procedures, tattoos, beauty treatments or any other procedure that affects the facial visual appearance of the user.
  • a method of providing interactive feedback to a user of the projected facial appearance of an object comprising the steps of: (a) capturing a real time image of the user's face; (b) projecting a 3D object on to the face within the image; (c) rendering the 3D object on the face to create a rendered image; (d) displaying the rendered image to the user in a real time feedback loop.
  • a synthetic face can be rendered for certain cosmetic treatments or procedures where the texture and/or geometry of the face image require alteration.
  • Fig. 1 illustrates a customer using the system as a virtual mirror to preview sunglasses
  • FIG. 2 illustrates a diagram of the system when used for online marketing
  • FIG. 3 illustrates a diagram of a generalized algorithm of virtual mirror system. Description of Preferred and Other Embodiments
  • the preferred embodiments provide a system to display a live image of the customer being tracked with a synthetically generated 3D model of eyewear, for the purpose of seeing the item on the face to improve the sales and marketing of eyewear.
  • the system provides robust tracking with precise feature location and registration at large angles of head rotation.
  • the customer need not remove their existing glasses during face tracking.
  • the system can be used in conjunction with an online store, to allow the customer to virtually "try on” the eyewear in a realistic and convenient way.
  • the preferred embodiments are robust at large angles of head rotation so that the customer can see themselves with eyewear from the side.
  • the virtual mirror is indistinguishable from a real mirror.
  • the preferred embodiment is directed to the skilled computer graphics programmer or skilled computer vision programmer and in particular, to the skilled 3D real time graphics programmer and familiarity with such system is assumed.
  • familiarity with such systems as those described in United States Patent US - A -
  • the preferred embodiments utilize accurate calibrated 3D face-tracking technology with automatic feature recognition such as provided by faceAPI TM (available from www.faceapi.com) in an online eyewear sales portal which allows customers to virtually "try on” eyewear.
  • faceAPI TM available from www.faceapi.com
  • Fig. 1 illustrates an operational environment of the preferred embodiment 1.
  • a user 2 interacts with a computer application projected on screen 3.
  • the application shows the user with a pair of sunglasses on 5.
  • Imagery of the user is captured via the camera 4.
  • Fig. 2 illustrates schematically the operational environment of the preferred embodiment.
  • the captured images 4 are fed to a face tracking application 7 running on computer 6.
  • the application 7 is in turn interconnected to the internet 10 and a backend application 1 1 which in turn interacts with a database of 3D models of eyewear 12.
  • the system could also function utilizing a local network between the application 7 and the backend application 11.
  • any of the interconnections between components of the system can also be network connections.
  • the customer sits in front of a web-camera, which produce a video feed of their face.
  • the application on computer 6 finds and tracks the customer's facial coordinates as they move.
  • the system renders a currently selected 3D model of eyewear over their face in real-time.
  • the system ideally allows robust tracking and precise feature location and registration at large angles. Different 3D models can be downloaded for rendering on demand from database 12.
  • the 3D model rendering allows the customer to rotate their head and observe how the eyewear would appear on them from the side and other angles, much like they would do in a real store. Furthermore, it allows the customer to 'try on' as many of the eyewear articles listed in the online catalogue as they would like with ease. Furthermore, it allows the customer to send videos or pictures of their appearance, whilst wearing the virtual glasses, to friends for feedback.
  • the catalogue of glasses can include highly realistic 3D models of the glasses for sale.
  • the system further allows an indication of the frame/face fit to be calculated and presented to the customer. For instance, the customer could be presented with a selection of eyewear from the catalogue that would ensure an appropriate fit to their face.
  • the basics of the application within the computer 6 can be as shown in the flow chart of Fig. 3.
  • the camera inputs a live image of the user 31. This is utilized by a face tracking system 32 to determine the 3D location data of the face 33. This data is then utilized in conjunction with a downloaded 3D model of the eyewear, to render the eyewear in real time on the user's face, with the output being displayed to the display 36.
  • the person need not remove their existing glasses; by tracking the glasses outline and using 2D image recovery, the real glasses can be removed and replaced with virtual ones, in real-time. This ensures the synthetically rendered image is convincing.
  • An embodiment includes tracking aspects of humans using image information obtained from a camera or set of cameras. These aspects can include any one or more of: the size, shape, texture and orientation of the onlooker's head, face, eyes (including eye features), and facial features; presence, shape and orientation of glasses, sunglasses or other head adornments (including jewellery, hats); skin colour and texture; and shadow geometry.
  • An another embodiment the system can be located in a kiosk or at a touch screen at a public place or in-store that allows the user to "try on” items that are suitable for their face type and proportions.
  • Another embodiment allows customers to preview other items or procedures they are considering purchasing like hair styles, cosmetics, cosmetic procedures, tattoos, beauty treatments or any other procedure that affects visual appearance.
  • One or more beauty metrics for the customer, before and/or after the simulation, can be calculated and displayed for the customer.
  • the virtual mirror can include a replay feature with the ability to pause and initiate other standard video editing capabilities, or one or more display screens positioned around the customer. This would allow the user to see themselves from angles of rotation that usually would not be possible with a standard mirror (such as the back of their own head for previewing hairstyles).
  • an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention.
  • a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising. [0036] Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limitative to direct connections only. The terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other.
  • a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means.
  • Coupled may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • Strategic Management (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Development Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Game Theory and Decision Science (AREA)
  • Software Systems (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention porte sur un suivi du visage qui est utilisé pour afficher une image en direct de la personne suivie avec un modèle en 3D, généré par synthèse, d’un article de lunetterie (par exemple des lunettes de soleil ou de vue) superposé sur le visage de la personne pour fournir un miroir virtuel pour la personne, par exemple pour améliorer les ventes et la commercialisation d’articles de lunetterie sur Internet. La personne n'a pas besoin de retirer ses propres lunettes existantes; par le suivi du contour des lunettes et l’utilisation d'une récupération d'image en 2D, toutes lunettes réelles sont retirées et remplacées par des lunettes virtuelles. Ceci assure que l'image rendue de façon synthétique est convaincante. Facultativement, étant donné que des traits du visage sont suivis en 3D avec une grande précision et exactitude, l'utilisateur peut également sélectionner des lunettes qui s’accordent bien aux proportions de son visage.
PCT/AU2009/001360 2008-10-16 2009-10-15 Commercialisation en ligne de produits pour le visage à l'aide d'un suivi du visage en temps réel WO2010042990A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
AU2008905350 2008-10-16
AU2008905350A AU2008905350A0 (en) 2008-10-16 Online marketing of facial products using real-time face tracking

Publications (1)

Publication Number Publication Date
WO2010042990A1 true WO2010042990A1 (fr) 2010-04-22

Family

ID=42106127

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/AU2009/001360 WO2010042990A1 (fr) 2008-10-16 2009-10-15 Commercialisation en ligne de produits pour le visage à l'aide d'un suivi du visage en temps réel

Country Status (1)

Country Link
WO (1) WO2010042990A1 (fr)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013177459A1 (fr) * 2012-05-23 2013-11-28 1-800 Contacts, Inc. Systèmes et procédés de rendu de produits d'essayage virtuel
GB2517270A (en) * 2013-07-30 2015-02-18 Holition Ltd Locating and augmenting object features in images
US9236024B2 (en) 2011-12-06 2016-01-12 Glasses.Com Inc. Systems and methods for obtaining a pupillary distance measurement using a mobile computing device
US9286715B2 (en) 2012-05-23 2016-03-15 Glasses.Com Inc. Systems and methods for adjusting a virtual try-on
US9483853B2 (en) 2012-05-23 2016-11-01 Glasses.Com Inc. Systems and methods to display rendered images
US9507175B2 (en) 2012-10-16 2016-11-29 3M Innovative Properties Company Methods and devices for evaluating eyewear fit
WO2017085160A1 (fr) * 2015-11-18 2017-05-26 Sirona Dental Systems Gmbh Procédé pour visualiser une situation dentaire
WO2018002533A1 (fr) * 2016-06-30 2018-01-04 Fittingbox Procédé d'occultation d'un objet dans une image ou une vidéo et procédé de réalité augmentée associé
FR3053509A1 (fr) * 2016-06-30 2018-01-05 Fittingbox Procede d’occultation d’un objet dans une image ou une video et procede de realite augmentee associe
CN108932654A (zh) * 2018-06-12 2018-12-04 苏州诚满信息技术有限公司 一种虚拟试妆指导方法及装置
US10685457B2 (en) 2018-11-15 2020-06-16 Vision Service Plan Systems and methods for visualizing eyewear on a user

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2003081536A1 (fr) * 2002-03-26 2003-10-02 So-Woon Kim Systeme et procede de simulation tridimensionnelle de port de lunettes
US6792401B1 (en) * 2000-10-31 2004-09-14 Diamond Visionics Company Internet-based modeling kiosk and method for fitting and selling prescription eyeglasses
EP1939812A2 (fr) * 2006-12-29 2008-07-02 Cheng-Hsien Yang Système terminal de simulation d'essai et fonctionnement et application de procédé correspondant
US20080201641A1 (en) * 2007-02-21 2008-08-21 Yiling Xie Method And The Associated Mechanism For 3-D Simulation Stored-Image Database-Driven Spectacle Frame Fitting Services Over Public Network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6792401B1 (en) * 2000-10-31 2004-09-14 Diamond Visionics Company Internet-based modeling kiosk and method for fitting and selling prescription eyeglasses
WO2003081536A1 (fr) * 2002-03-26 2003-10-02 So-Woon Kim Systeme et procede de simulation tridimensionnelle de port de lunettes
EP1939812A2 (fr) * 2006-12-29 2008-07-02 Cheng-Hsien Yang Système terminal de simulation d'essai et fonctionnement et application de procédé correspondant
US20080201641A1 (en) * 2007-02-21 2008-08-21 Yiling Xie Method And The Associated Mechanism For 3-D Simulation Stored-Image Database-Driven Spectacle Frame Fitting Services Over Public Network

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
"Proc. of International Conference on Image Processing (ICIP 2007)", vol. 2, September 2007, article EISERT, P. ET AL.: "Virtual Mirror: Real-Time Tracking of Shoes in Augmented Reality Environments", pages: 557 - 560 *
"Virtual Fitting Rooms at IFC 2008", 16 November 2009 (2009-11-16), Retrieved from the Internet <URL:http://www.photonics.com/content/ReadArticle.aspx?ArticleID=34939> *
"Virtual Mirror Augmented Reality Applications by Fraunhofer", 19 November 2009 (2009-11-19), Retrieved from the Internet <URL:http://ip.hhi.de/comvisionG2/assets/pdfs/FlyerVirtualMirror2.pdf> *

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9236024B2 (en) 2011-12-06 2016-01-12 Glasses.Com Inc. Systems and methods for obtaining a pupillary distance measurement using a mobile computing device
US10147233B2 (en) 2012-05-23 2018-12-04 Glasses.Com Inc. Systems and methods for generating a 3-D model of a user for a virtual try-on product
WO2013177459A1 (fr) * 2012-05-23 2013-11-28 1-800 Contacts, Inc. Systèmes et procédés de rendu de produits d'essayage virtuel
US9208608B2 (en) 2012-05-23 2015-12-08 Glasses.Com, Inc. Systems and methods for feature tracking
US9235929B2 (en) 2012-05-23 2016-01-12 Glasses.Com Inc. Systems and methods for efficiently processing virtual 3-D data
US9286715B2 (en) 2012-05-23 2016-03-15 Glasses.Com Inc. Systems and methods for adjusting a virtual try-on
US9311746B2 (en) 2012-05-23 2016-04-12 Glasses.Com Inc. Systems and methods for generating a 3-D model of a virtual try-on product
US9378584B2 (en) 2012-05-23 2016-06-28 Glasses.Com Inc. Systems and methods for rendering virtual try-on products
US9483853B2 (en) 2012-05-23 2016-11-01 Glasses.Com Inc. Systems and methods to display rendered images
US9507175B2 (en) 2012-10-16 2016-11-29 3M Innovative Properties Company Methods and devices for evaluating eyewear fit
US10529078B2 (en) 2013-07-30 2020-01-07 Holition Limited Locating and augmenting object features in images
GB2517270B (en) * 2013-07-30 2020-06-03 Holition Ltd Locating and augmenting object features in images
US9990734B2 (en) 2013-07-30 2018-06-05 Holition Limited Locating and augmenting object features in images
GB2517270A (en) * 2013-07-30 2015-02-18 Holition Ltd Locating and augmenting object features in images
WO2017085160A1 (fr) * 2015-11-18 2017-05-26 Sirona Dental Systems Gmbh Procédé pour visualiser une situation dentaire
US10980422B2 (en) 2015-11-18 2021-04-20 Dentsply Sirona Inc. Method for visualizing a tooth situation
CN109983501A (zh) * 2016-06-30 2019-07-05 试戴盒子公司 一种隐藏图像或视频中的对象的方法和相关的增强现实方法
WO2018002533A1 (fr) * 2016-06-30 2018-01-04 Fittingbox Procédé d'occultation d'un objet dans une image ou une vidéo et procédé de réalité augmentée associé
US9892561B2 (en) 2016-06-30 2018-02-13 Fittingbox Method of hiding an object in an image or video and associated augmented reality process
FR3053509A1 (fr) * 2016-06-30 2018-01-05 Fittingbox Procede d’occultation d’un objet dans une image ou une video et procede de realite augmentee associe
CN109983501B (zh) * 2016-06-30 2023-08-25 试戴盒子公司 一种隐藏图像或视频中的对象的方法和相关的增强现实方法
CN108932654A (zh) * 2018-06-12 2018-12-04 苏州诚满信息技术有限公司 一种虚拟试妆指导方法及装置
CN108932654B (zh) * 2018-06-12 2021-03-26 苏州诚满信息技术有限公司 一种虚拟试妆指导方法及装置
US10685457B2 (en) 2018-11-15 2020-06-16 Vision Service Plan Systems and methods for visualizing eyewear on a user

Similar Documents

Publication Publication Date Title
WO2010042990A1 (fr) Commercialisation en ligne de produits pour le visage à l&#39;aide d&#39;un suivi du visage en temps réel
AU2018214005B2 (en) Systems and methods for generating a 3-D model of a virtual try-on product
KR102399289B1 (ko) 안경의 가상 시착 시스템 및 방법
US10235810B2 (en) Augmented reality e-commerce for in-store retail
US20200066052A1 (en) System and method of superimposing a three-dimensional (3d) virtual garment on to a real-time video of a user
US9990780B2 (en) Using computed facial feature points to position a product model relative to a model of a face
US6792401B1 (en) Internet-based modeling kiosk and method for fitting and selling prescription eyeglasses
JP2019102068A (ja) ユーザの身体的特徴に基づいた推薦システム
CN107492007A (zh) 一种虚拟现实的店铺展示方法和装置
CN108427498A (zh) 一种基于增强现实的交互方法和装置
US20110234581A1 (en) Methods and systems for three-dimensional rendering of a virtual augmented replica of a product image merged with a model image of a human-body feature
JP2003030276A (ja) 眼鏡用の対話式試着プラットフォーム
CN107533727A (zh) 全息交互式零售系统
US20170323374A1 (en) Augmented reality image analysis methods for the virtual fashion items worn
US20220044311A1 (en) Method for enhancing a user&#39;s image while e-commerce shopping for the purpose of enhancing the item that is for sale
Loker et al. Dress in the third dimension: Online interactivity and its new horizons
CN115803750B (zh) 使用参考框架的眼镜的虚拟试戴系统
CN113610612B (zh) 一种3d虚拟试衣方法、系统及存储介质
Zhang et al. A virtual try-on system for prescription eyeglasses
KR102506352B1 (ko) 전자상거래를 위한 3d 인체 측정 데이터 기반의 디지털 트윈 아바타 제공 시스템
CN106570747A (zh) 结合手势识别的眼镜在线适配方法及系统
CN104299143A (zh) 虚拟试戴方法及装置
US20190066197A1 (en) System and Method for Clothing Promotion
WO2022217097A1 (fr) Procédé et appareil de mannequin virtuel permettant l&#39;ajustement de vêtements vendus en ligne
EP2455895A1 (fr) Dispositif d&#39;aide d&#39;utilisateur dans le choix de vêtements

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09820109

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 09820109

Country of ref document: EP

Kind code of ref document: A1