EP3026636A1 - Procédé et appareil de génération d'un modèle de visage 3d personnalisé - Google Patents

Procédé et appareil de génération d'un modèle de visage 3d personnalisé Download PDF

Info

Publication number
EP3026636A1
EP3026636A1 EP15195543.2A EP15195543A EP3026636A1 EP 3026636 A1 EP3026636 A1 EP 3026636A1 EP 15195543 A EP15195543 A EP 15195543A EP 3026636 A1 EP3026636 A1 EP 3026636A1
Authority
EP
European Patent Office
Prior art keywords
face
personalized
face model
face image
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP15195543.2A
Other languages
German (de)
English (en)
Other versions
EP3026636B1 (fr
Inventor
Jungbae Kim
Youngkyoo Hwang
Seon Min Rhee
Jaejoon HAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP3026636A1 publication Critical patent/EP3026636A1/fr
Application granted granted Critical
Publication of EP3026636B1 publication Critical patent/EP3026636B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships

Definitions

  • Example embodiments relate to technology for generating a three-dimensional (3D) face model based on an input image.
  • a 3D face modeling method includes a method of modeling a 3D face of a user using a 3D depth camera or a 3D scanner.
  • the method using a 3D depth camera models a 3D face of a user using a stereo matching scheme.
  • the method using a 3D scanner models a 3D face of a user using 3D data scanned through the 3D scanner.
  • a 3D coordinate value and a color value are acquired with respect to each of sampling points on the surface of the face of the user, and 3D face modeling is performed on the user face based on the acquired 3D coordinate value and color value.
  • a method of generating a three-dimensional (3D) face model includes extracting feature points of a face from input images comprising a first face image and a second face image; deforming a generic 3D face model to a personalized 3D face model based on the feature points; projecting the personalized 3D face model to each of the first face image and the second face image; and refining the personalized 3D face model based on a difference in texture patterns between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected.
  • the refining may include extracting a correspondence point between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected; and generating a comparison based on a texture pattern of the first face image and a texture pattern of the second face image in a peripheral area of the correspondence point.
  • the refining may include refining a shape of the personalized 3D face model to make a similarity between the texture pattern of the first face image and the texture pattern of the second face image exceed a threshold degree of similarity in the peripheral area of the correspondence point.
  • the refining may include includes, iteratively, determining if a first condition is satisfied, and refining a shape of the personalized 3D face model based on the difference in texture patterns, until the first condition is satisfied.
  • the refining may include determining a pose deformation parameter and a shape control parameter that reduce the difference in texture patterns between the first face image and the second face image in the peripheral area of the correspondence point; and applying the determined pose deformation parameter and shape control parameter to the personalized 3D face model.
  • the shape control parameter may control a spatial location of each of vertices constituting the personalized 3D face model.
  • the projecting may include projecting the personalized 3D face model to the first face image based on a facial pose included in the first face image; and projecting the personalized 3D face model to the second face image based on a facial pose included in the second face image.
  • the deforming may include acquiring the personalized 3D face model by mapping landmark points of the generic 3D face model to the extracted feature points.
  • the input images may be two-dimensional (2D) images, and the input images may include at least one frontal image captured from a front of a user face and at least one side face image captured from a side of the user face.
  • 2D two-dimensional
  • a non-transitory computer-readable medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including, extracting feature points of a face from input images comprising a first face image and a second face image, deforming a generic 3D face model to a personalized 3D face model based on the feature points, projecting the personalized 3D face model to each of the first face image and the second face image, and refining the personalized 3D face model based on a difference in texture patterns between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected.
  • an apparatus for generating a three-dimensional (3D) face model includes a memory storing computer-readable instructions; and one or more processors configured to execute the computer-readable instruction to, extract feature points of a face from input images comprising a first face image and a second face image, deform a generic 3D face model to a personalized 3D face model based on the feature points, and refine the personalized 3D face model based on a difference in texture patterns between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected.
  • the one or more processors may be configured to extract a correspondence point between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected, and compare a texture pattern of the first face image and a texture pattern of the second face image in a peripheral area of the correspondence point.
  • the one or more processors may be configured to refine a shape of the personalized 3D face model to make a similarity between the texture pattern of the first face image and the texture pattern of the second face image exceed a threshold degree of similarity in the peripheral area of the correspondence point.
  • the one or more processors may be configured to, iteratively, determine if a first condition is satisfied, and refine a shape of the personalized 3D face model based on the difference in texture patterns, until the first condition is satisfied.
  • the one or more processors may be configured to, determine a pose deformation parameter and a shape control parameter that reduce the difference in texture patterns between the first face image and the second face image in the peripheral area of the correspondence point, and apply the determined pose deformation parameter and shape control parameter to the personalized 3D face model.
  • the shape control parameter may control a spatial location of each of vertices constituting the personalized 3D face model.
  • the one or more processors may be further configured to execute the computer-readable instructions to store the refined personalized 3D face model.
  • the one or more processors may be configured to acquire the personalized 3D face model by mapping landmark points of the generic 3D face model to the extracted feature points.
  • the input images may be two-dimensional (2D) images, and the input images may include at least one frontal image captured from a front of a user face and at least one side face image captured from a side of the user face.
  • 2D two-dimensional
  • Example embodiments of the inventive concepts are described herein with reference to schematic illustrations of idealized embodiments (and intermediate structures) of the inventive concepts. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and/or tolerances, are to be expected. Thus, example embodiments of the inventive concepts should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing.
  • the cross-sectional view(s) of device structures illustrated herein provide support for a plurality of device structures that extend along two different directions as would be illustrated in a plan view, and/or in three different directions as would be illustrated in a perspective view.
  • the two different directions may or may not be orthogonal to each other.
  • the three different directions may include a third direction that may be orthogonal to the two different directions.
  • the plurality of device structures may be integrated in a same electronic device.
  • an electronic device may include a plurality of the device structures (e.g., memory cell structures or transistor structures), as would be illustrated by a plan view of the electronic device.
  • the plurality of device structures may be arranged in an array and/or in a two-dimensional pattern.
  • FIG. 1 is a diagram to describe the overall operation of an apparatus 100 for generating a personalized three-dimensional (3D) face model according to example embodiments.
  • the personalized 3D face model generating apparatus 100 generates a personalized 3D face model based on a plurality of input images.
  • Input images may include input images captured from a user face to be modeled at different angles or in different directions.
  • face images may include at least one frontal face image captured from a front of the user face and at least one side face image captured from a side of the user face.
  • a face image refers to a two-dimensional (2D) image captured from the user face, and may include a global area or a partial area of the user face.
  • the 3D face model generating apparatus 100 may include or be implemented by one or more circuits or circuitry (e.g., hardware) specifically structured to carry out some or all of the operations described herein as being performed by the 3D face model generating apparatus 100 (or an element thereof).
  • the 3D face model generating apparatus 100 may include or be implemented by a memory and one or more processors executing computer-readable code (e.g., software) that is stored in the memory and includes instructions corresponding to some or all of the operations described herein as being performed by the 3D face model generating apparatus 100 (or an element thereof).
  • the 3D face model generating apparatus 100 may be implemented by, for example, a combination of the above-referenced hardware and processors executing computer-readable code.
  • the term 'processor' may refer to, for example, a hardware-implemented data processing device having circuitry that is physically structured to execute desired operations including, for example, operations represented as code and/or instructions included in a program.
  • desired operations including, for example, operations represented as code and/or instructions included in a program.
  • hardware-implemented data processing device include, but are not limited to, a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor; a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).
  • processors executing program code are programmed processors, and thus, are special-purpose computers.
  • the user may take a plurality of 2D face images using a camera.
  • the taken 2D face images may be input to the personalized 3D face model generating apparatus 100.
  • Face images in which the user face is taken in different directions may be acquired by taking a photo while fixing the user face and changing a location of the camera, or by taking a photo while fixing the camera and changing a direction of the user face.
  • the personalized 3D face model generating apparatus 100 may generate a personalized 3D face model by three-dimensionally modeling the user face based on the input face images.
  • the personalized 3D face model generating apparatus 100 may generate the personalized 3D face model by selecting one of generic 3D face models stored in a generic 3D face model database 110 and by matching a shape of the selected generic 3D face model to a shape of the user face represented in each of the face images.
  • a generic 3D face model refers to a statistical 3D shape model generated based on training data.
  • the personalized 3D face model generating apparatus 100 may generate the personalized 3D face model by extracting feature points of a face from the face images and by matching the extracted feature points to the generic 3D face model.
  • the "feature point of a face” or “facial feature point” may be a "facial keypoint” or a "facial landmark”.
  • the personalized 3D face model generating apparatus 100 may generate a personalized 3D face model by extracting, from each of face images, feature points corresponding to eyebrows, eyes, a nose, lips, a chin, ears, and/or the face contour, and by deforming a generic 3D face model to be matched to the extracted feature points.
  • the personalized 3D face model generating apparatus 100 may generate a high-precision personalized 3D face model by quickly generating a coarse personalized 3D face model based on feature points extracted from face images and by refining a shape of the personalized 3D face model based on texture information of the face images.
  • the personalized 3D face model generating apparatus 100 may project the personalized 3D face model to each of the face images and may determine a correspondence point between the face images using the personalized 3D face model.
  • the personalized 3D face model generating apparatus 100 may compare texture patterns of the face images in a peripheral area of the correspondence point, and may refine a shape of the personalized 3D face model, so that a similarity between the texture patterns of the face image may increase.
  • a difference in texture patterns may be present in a correspondence area between a frontal face image to which the personalized 3D face model is projected and a side face image to which the personalized 3D face model is projected.
  • a texture corresponding to a nose end of the personalized 3D face model may represent a texture of a nose in the frontal face image, and may represent a texture of a background in the side face image.
  • the personalized 3D face model generating apparatus 100 may refine a shape of the personalized 3D face model so that the difference in texture patterns (i.e., between the frontal face image and the side face image) may decrease.
  • the personalized 3D face model generating apparatus 100 may reduce a height of the nose of the personalized 3D face model until a texture of a nose end area is changed from the texture of the background to the texture of the nose in the side face image to which the personalized 3D face model is projected.
  • the personalized 3D face model generating apparatus 100 may compare texture patterns of face images in another correspondence area based on the personalized 3D face model, and may refine the shape of the personalized 3D face model so that the difference in texture patterns may decrease.
  • the personalized 3D face model acquired through the aforementioned refining process may be utilized in a variety of application fields, for example, a facial recognition system and a security/surveillance system.
  • FIG. 2 is a block diagram illustrating an apparatus 200 for generating a personalized 3D face model according to example embodiments.
  • the personalized 3D face model generating apparatus 200 includes a feature point extractor 210, a personalized 3D face model generator 220, and a personalized 3D face model refiner 230.
  • the personalized 3D face model generating apparatus 200 receives input images including face images of a user and generates a personalized 3D face model representing a shape of the user face.
  • the input images include two or more face images.
  • the personalized 3D face model generating apparatus 200 generates a personalized 3D face model based on a first face image and a second face image.
  • the personalized 3D face model generating apparatus 200 may also generate the personalized 3D face model based on three or more face images.
  • the personalized 3D face model generating apparatus 200 may generate a personalized 3D face model based on a group of images including a single frontal face image and a plurality of side face images.
  • the 3D face model generating apparatus 200 may include or be implemented by one or more circuits or circuitry (e.g., hardware) specifically structured to carry out some or all of the operations described herein as being performed by the 3D face model generating apparatus 200 (or an element thereof).
  • the 3D face model generating apparatus 200 may include or be implemented by a memory and one or more processors executing computer-readable code (e.g., software) that is stored in the memory and includes instructions corresponding to some or all of the operations described herein as being performed by the 3D face model generating apparatus 200 (or an element thereof).
  • the 3D face model generating apparatus 200 may be implemented by, for example, a combination of the above-referenced hardware and processors executing computer-readable code.
  • the feature point extractor 210 extracts facial feature points from input images including the first face image and the second face image.
  • the first face image may be a frontal face image captured from a front of a user face and the second face image may be a side face image captured from a side of the user face.
  • the feature point extractor 210 may detect a facial area of the user by performing face tracking on the input images and may extract the facial feature points within the detected facial area.
  • Locations of feature points extracted from each of the face images may be defined in advance, for example, by a manufacture or user of the 3D face model generating apparatus 200.
  • the feature point extractor 210 may extract feature points located at eyebrows, eyes, a nose, lips, chin, and/or face contour from input images.
  • the personalized 3D face model generating apparatus 200 may extract feature points from the input images using a variety of feature point extraction methods known in the art.
  • the feature point extractor 210 may extract facial feature points from input images using an active contour model (ACM), an active shape model (ASM), an active appearance model (AAM), a supervised descent method (SDM), and the like.
  • ACM active contour model
  • ASM active shape model
  • AAM active appearance model
  • SDM supervised descent method
  • the personalized 3D face model generator 220 deforms a generic 3D face model to a personalized 3D face model.
  • the personalized 3D face model generating apparatus 200 may generate the personalized 3D face model by adjusting a pose and a shape of the generic 3D face model based on feature points extracted from input images.
  • the generic 3D face model refers to a deformable 3D shape face model generated based on training data.
  • the generic 3D face model includes vertices each having a spatial location. Spatial locations of the vertices may be determined based on a shape control parameter applied to the generic 3D face model.
  • the personalized 3D face model generator 220 may acquire a 3D shape model by extracting 3D coordinates about a shape of the user face based on feature points extracted from input images, and by deforming the generic 3D face model to be matched to the extracted 3D coordinates.
  • the personalized 3D face model generator 220 may generate a further precise personalized 3D face model using feature points extracted from face images taken at different views. For example, when generating a personalized 3D face model based on only feature points extracted from a frontal face image, it may be difficult to accurately determine a height of nose and a shape of cheekbone. Feature points extracted from a face image taken at a different view, such as a side face image, include detailed information about the height of nose and the shape of cheekbone. Thus, it is possible to further precisely model a 3D face model of the user based on the extracted feature points.
  • the generic 3D face model may use, for example, one or more of a Candide face model, a Warter's face model, and a directly designed face model.
  • the personalized 3D face model generator 220 may generate the personalized 3D face model by selecting one of generic 3D face models stored in a generic 3D face model database 250 and by matching a shape of the selected generic 3D face model to feature points extracted from input images.
  • the personalized 3D face model generator 220 may deform a generic 3D face model to a personalized 3D face model by extracting a plurality of landmark points from the generic 3D face model and by matching feature points extracted from input images to the plurality of landmark points extracted from the generic 3D face model.
  • the personalized 3D face model generator 220 may acquire a personalized 3D face model by determining shape control parameters for deforming a shape of a generic 3D face model based on locations of feature points extracted from input images and by applying the determined shape control parameters to the generic 3D face model.
  • Equation 1 Y 2D,k denotes locations of feature points extracted from a k th face image, m denotes the number of face images, and P k denotes a matrix that performs a 2D perspective projection by warping the 3D generic face model based on a face pose estimated from the k th face image.
  • S denotes parameters that represent a mean shape of the generic 3D face model
  • ⁇ has an eigenvalue as shape control parameters to determine a deformation level of a shape of the generic 3D face model
  • V has an eigenvector value as shape control parameters to determine a deformation direction of the shape of the generic 3D face model. The deformation direction of the shape of the generic 3D face model is determined.
  • n denotes the number of vertices that constitute the generic 3D face model.
  • the subscript i denotes an index for identifying a vertex of the shape of the generic 3D face model to which each of ⁇ and V is applied.
  • shape control parameters for matching vertices of the generic 3D face model to feature points extracted from face images by adjusting spatial locations of vertices in the mean shape of the generic 3D face model may be determined.
  • the personalized 3D face model generator 220 may generate a 3D face shape model that represents a shape of the user face by applying the determined shape control parameters to vertices of the generic 3D face model.
  • the personalized 3D face model generator 220 may select a single reference face image from among input images and may generate a 3D face texture model based on texture information of the reference face image.
  • the 3D face shape model refers to a model that includes shape information of the user face
  • the 3D face texture model refers to a model that includes texture information in addition to the shape information of the user face.
  • the 3D face shape model and the 3D face texture model are each types of the personalized 3D face model.
  • the personalized 3D face model generator 220 may determine, as a reference face image, a face image having a least facial occlusion or a face image that represents a pose closest to a frontal face image among input images.
  • the personalized 3D face model generator 220 may generate a 3D face texture model by mapping a texture of a reference face image to the 3D face shape model.
  • the personalized 3D face model refiner 230 may refine a personalized 3D face model by comparing texture patterns of face images.
  • the personalized 3D face model refiner 230 may compare texture patterns of face images using the personalized 3D face model, and may refine a shape of the personalized 3D face model to minimize or, alternatively, reduce a difference in texture patterns.
  • the personalized 3D face model refiner 230 may project the personalized 3D face model to each of the first face image and the second face image.
  • the personalized 3D face model refiner 230 may warp the personalized 3D face model to be matched to a face pose represented in each of face images, and may project the warped personalized 3D face model to the face images.
  • the personalized 3D face model refiner 230 may refine the personalized 3D face model based on a difference in texture patterns between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected.
  • the personalized 3D face model refiner 230 may extract a correspondence point between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected.
  • the personalized 3D face model refiner 230 may compare a texture pattern of the first face image and a texture pattern of the second face image in a peripheral area of the correspondence point, and may refine the shape of the personalized 3D face model based on a result of comparing the texture patterns.
  • the personalized 3D face model refiner 230 may compare texture patterns of peripheral areas for the respective correspondence points, and may adjust spatial locations of vertices constituting the personalized 3D face model, so that the difference in texture patterns is minimize or, alternatively, reduced.
  • the personalized 3D face model When projecting the personalized 3D face model to the first face image, the personalized 3D face model may be warped to be matched to a face pose of the first face image and vertices of the personalized 3D face model may be projected to the first face image.
  • the personalized 3D face model when projecting the personalized 3D face model to the second face image, the personalized 3D face model may be warped to be matched to a face pose of the second face image and vertices of the personalized 3D face model may be projected to the second face image.
  • the personalized 3D face model refiner 230 may identify a location of a correspondence point between face images to which the personalized 3D face model is projected, based on a barycentric coordinate characteristic that a barycenter is maintained irrespective of a deformation of a triangle.
  • the personalized 3D face model refiner 230 may compare the texture pattern of the first face image and the texture pattern of the second face image in the peripheral area of the identified correspondence point, and may determine whether to adjust a location of a vertex of the personalized 3D face model corresponding to the correspondence point.
  • the personalized 3D face model refiner 230 may adjust spatial locations of vertices of the personalized 3D face model to make the texture pattern of the first face image become similar to (e.g., above a threshold degree of similarity with respect to) the texture pattern of the second face image in the peripheral area of the correspondence point.
  • the shape of the personalized 3D face model may be further precisely refined by adjusting the spatial locations of vertices.
  • the personalized 3D face model refiner 230 may determine a pose deformation parameter and a shape control parameter that minimize or, alternatively, reduce a difference in texture patterns between the first face image and the second face image in the peripheral area of the correspondence point.
  • the personalized 3D face model refiner 230 may determine a pose deformation parameter and a shape control parameter to refine a personalized 3D face model according to the following Equation 2.
  • the first face image is determined as the reference face image.
  • R ⁇ arg min R i ⁇ p ⁇ i ⁇ I ref - W I i , PR i S ⁇ p + V p ⁇ p ⁇ 2
  • Equation 2 S p denotes parameters that represent a mean shape of the personalized 3D face model, ⁇ p has an eigenvalue as shape control parameters to determine a deformation level of a shape of the personalized 3D face model.
  • the shape control parameters are used to determine a spatial location of each of vertices constituting the personalized 3D face model.
  • V p has an eigenvector value as shape control parameters to determine a deformation direction of the shape of the personalized 3D face model.
  • the element Ri corresponding to one of pose deformation parameters denotes a matrix to deform a pose of the personalized 3D face model between the second face image and the first face image.
  • P denotes a matrix that performs a 2D perspective projection of the personalized 3D face model to which the shape control parameter and the pose deformation parameter are applied.
  • W denotes a warping function that realigns a location of a texture using shape and location deformation matrices.
  • Ii denotes a texture of the second face image and Iref denotes a texture of the first face image determined as the reference face image.
  • pose deformation parameters and shape control parameters that minimize or, alternatively, reduce a difference in texture patterns between the first face image and the second face image may be determined.
  • the personalized 3D face model refiner 230 may further precisely refine the personalized 3D face model by applying the determined pose deformation parameters and shape control parameters to the personalized 3D face model.
  • the personalized 3D face model refiner 230 may iteratively refine a shape of the personalized 3D face model until the difference in texture patterns between the first face image and the second face image satisfies a predetermined or, alternatively, desired condition.
  • the personalized 3D face model generating apparatus 200 may further include a personalized 3D face model storage 240.
  • the personalized 3D face model storage 240 may store a finally refined personalized 3D face model.
  • the personalized 3D face model storage 240 may store personalized 3D face models corresponding to users, respectively.
  • the personalized 3D face model generating apparatus 200 may update a personalized 3D face model stored in the personalized 3D face model storage 240.
  • the personalized 3D face model generating apparatus 200 may extract feature points of a face from the new face image and may update a previously generated personalized 3D face model based on the extracted feature points.
  • FIG. 3 illustrates face images 310, 320, and 330 to describe a process of extracting feature points from input images according to example embodiments.
  • the face image 320 denotes a frontal face image captured from a front of a user face
  • the face image 310 denotes a left-side face image captured from a left-side of the user face
  • the face image 330 denotes a right-side face image captured from a right-side of the user face.
  • the global 2D shape information and texture information about the user face may be extracted from the face image 320. Further detailed information about a shape of the user face may be extracted from the face images 310 and 330.
  • a personalized 3D face model generating apparatus may extract facial feature points 315, 325, and 335 from the face images 310, 320, and 330, respectively.
  • the personalized 3D face model generating apparatus may detect a facial area from each of the face images 310, 320, and 330, and may extract facial feature points corresponding to eyebrows, eyes, a nose, lips, chin, and/or face contour within the detected facial area.
  • the personalized 3D face model generating apparatus may extract facial feature points from each of the face images 310, 320, and 330 using a variety of methods known in the art, for example, an ASM, an AAM, and an SDM
  • FIG. 4 illustrates face models to describe a process of generating a personalized 3D face model by deforming a generic 3D face model according to example embodiments.
  • the generic 3D face model 410 refers to a deformable 3D shape model generated based on 3D face training data, and also refers to a parametric model capable of representing a shape of a user face using a mean shape and a parameter.
  • a personalized 3D face model generating apparatus may personalize the generic 3D face model 410 based on a plurality of face images captured from the user face.
  • the personalized 3D face model generating apparatus may extract facial feature points from the face images, and may determine a parameter capable of matching the extracted facial feature points to landmark points of the generic 3D face model 410.
  • the personalized 3D face model generating apparatus may generate the personalized 3D face models 420 and 430 by applying the determined parameter to the generic 3D face model 410.
  • Each of the personalized 3D face models 420 and 430 refers to a 3D face model of which a pose or an expression is deformable.
  • the personalized 3D face model 420 refers to a 3D face model that includes only shape information and does not include texture information.
  • the personalized 3D face model 430 refers to a 3D face model that includes both shape information and texture information.
  • the personalized 3D face model generating apparatus may generate the personalized 3D face model 430 to which a texture is applied by mapping the texture extracted from at least one of the face images to the personalized 3D face model 420.
  • the personalized 3D face model 430 that further includes texture information has a higher precision than the personalized 3D face model 420 that includes only the shape information.
  • the personalized 3D face model 430 includes a more number of vertices.
  • FIG. 5 illustrates face images 510, 520, and 530 to describe a process of projecting a personalized 3D face model to input images, comparing texture patterns of the input images, and refining the personalized 3D face model according to example embodiments.
  • the face image 520 refers to a frontal face image
  • the face image 510 refers to a left-side face image
  • the face image 530 refers to a right-side face image.
  • the face image 520 is determined as a reference face image.
  • a personalized 3D face model generating apparatus may project a personalized 3D face model 540 to each of the face images 510, 520, and 530.
  • the personalized 3D face model 540 may represent a user face in a 3D shape using vertices.
  • the personalized 3D face model generating apparatus may warp the personalized 3D face model 540 based on a face pose represented in each of the face images 510, 520, and 530, and then may perform 2D perspective projection on each of the face images 510, 520, and 530.
  • the personalized 3D face model generating apparatus may extract a correspondence point among the face images 510, 520, and 530 to which the personalized 3D face model 540 is projected. For example, a point 570 of the face image 510, a point 550 of the face image 520, and a point 560 of the face point 530 are assumed as a correspondence point. An area in which the same vertex of the personalized 3D face model 540 is located in each of the face images 510, 520, and 530 may be determined as an area in which the correspondence point is located.
  • the personalized 3D face model generating apparatus may compare texture patterns in a peripheral area of a correspondence point between the face image 520 determined as the reference face image and the face image 510, and may refine a shape of the personalized 3D face model 540 based on a difference in the texture patterns.
  • the personalized 3D face model generating apparatus may refine the shape of the personalized 3D face model 540 to minimize or, alternatively, reduce the difference in texture patterns between the face image 520 and the face image 510 in the peripheral area of the correspondence point.
  • the personalized 3D face model generating apparatus may more precisely refine the personalized 3D face model 540 by determining a pose deformation parameter and a shape control parameter to minimize or, alternatively, reduce a difference in texture patterns between the face images 510 and 520 in a peripheral area of each correspondence point, and by applying the determined pose deformation parameter and shape control parameter to the personalized 3D face model 540.
  • the personalized 3D face model generating apparatus may compare texture patterns only in a peripheral area of a correspondence point between the face image 520 determined as the reference face image and the face image 530, and may refine a shape of the personalized 3D face model 540 to minimize or, alternatively, reduce a difference in the texture patterns in the peripheral area of the correspondence point.
  • FIG. 6 is a flowchart illustrating a method of generating a personalized 3D face model according to example embodiments.
  • a personalized 3D face model generating apparatus extracts facial feature points from input images.
  • the personalized 3D face model generating apparatus may detect a facial area of a user by performing face tracking on the input images and may extract the facial feature points within the detected facial area.
  • the personalized 3D face model generating apparatus may extract facial feature points located at eyebrows, eyes, a nose, lips, chin, and/or face contour from input images using an ACM, an ASM, an AAM, an SDM, and the like.
  • the personalized 3D face model generating apparatus may deform a generic 3D face model to the personalized 3D face model based on the extracted feature points.
  • the personalized 3D face model generating apparatus may acquire a 3D shape model by extracting 3D coordinates based on the feature points extracted from the input images and by deforming the generic 3D face model to be matched to the extracted 3D coordinates.
  • the generic 3D face model may use, for example, a Candide face model, a Warter's face model, and a directly designed face model.
  • the personalized 3D face model generating apparatus may generate the personalized 3D face model by selecting one of generic 3D face models and by matching a shape of the selected generic 3D face model to feature points extracted from input images.
  • the personalized 3D face model generating apparatus may acquire a personalized 3D face model by determining shape control parameters for deforming a shape of the generic 3D face model based on locations of feature points extracted from input images and by applying the determined shape control parameters to the generic 3D face model.
  • the personalized 3D face model generating apparatus may project the personalized 3D face model to each of the input images.
  • the personalized 3D face model generating apparatus may warp the personalized 3D face model to be matched to a face pose represented in each of face images, and may project the warped personalized 3D face model to the face images.
  • the personalized 3D face model generating apparatus may compare texture patterns of face images based on the personalized 3D face model and may refine the personalized 3D face model based on a result of comparing the texture patterns. For example, the personalized 3D face model generating apparatus may refine a personalized 3D face model based on a difference in texture patterns between a first face image to which the personalized 3D face model is projected and a second face image to which the personalized 3D face model is projected. The personalized 3D face model generating apparatus may refine a shape of the personalized 3D face model to minimize or, alternatively, reduce the difference in texture patterns between face images based on the personalized 3D face model. Operation 640 will be further described with reference to FIG. 7 .
  • FIG. 7 is a flowchart illustrating in detail operation 640 of refining a personalized 3D face model in a method of generating the personalized 3D face model according to example embodiments.
  • the personalized 3D face model generating apparatus may extract a correspondence point between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected.
  • the personalized 3D face model generating apparatus may extract a correspondence point between face images based on locations of vertices of the personalized 3D face model projected to each of the face images.
  • the personalized 3D face model generating apparatus may refine the shape of the personalized 3D face model to make the texture pattern of the first face image and the texture pattern of the second face image become similar to (e.g., above a threshold degree of similarity with respect to) each other in the peripheral area of the correspondence point.
  • the personalized 3D face model generating apparatus may refine the shape of the personalized 3D face model to minimize or, alternatively, reduce the difference in texture patterns between face images in the peripheral area of the correspondence point.
  • the personalized 3D face model generating apparatus may refine a shape of a personalized 3D face model by determining a pose deformation parameter and a shape control parameter that minimize or, alternatively, reduce a difference in texture patterns between face images and by applying the determined pose deformation parameter and shape control parameter to the personalized 3D face model.
  • the units and/or modules described herein may be implemented using hardware components and software components.
  • the hardware components may include microphones, amplifiers, band-pass filters, audio to digital convertors, and processing devices.
  • a processing device may be implemented using one or more hardware device configured to carry out and/or execute program code by performing arithmetical, logical, and input/output operations.
  • the processing device(s) may include a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor or any other device capable of responding to and executing instructions in a defined manner.
  • the processing device may run an operating system (OS) and one or more software applications that run on the OS.
  • OS operating system
  • the processing device also may access, store, manipulate, process, and create data in response to execution of the software.
  • a processing device may include multiple processing elements and multiple types of processing elements.
  • a processing device may include multiple processors or a processor and a controller.
  • different processing configurations are possible, such a parallel processors.
  • the software may include a computer program, a piece of code, an instruction, or some combination thereof, to independently or collectively instruct and/or configure the processing device to operate as desired, thereby deforming the processing device into a special purpose processor.
  • Software and data may be embodied permanently or temporarily in any type of machine, component, physical or virtual equipment, computer storage medium or device, or in a propagated signal wave capable of providing instructions or data to or being interpreted by the processing device.
  • the software also may be distributed over network coupled computer systems so that the software is stored and executed in a distributed fashion.
  • the software and data may be stored by one or more non-transitory computer readable recording mediums.
  • the methods according to the above-described example embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations of the above-described example embodiments.
  • the media may also include, alone or in combination with the program instructions, data files, data structures, and the like.
  • the program instructions recorded on the media may be those specially designed and constructed for the purposes of example embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts.
  • non-transitory computer-readable media examples include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM discs, DVDs, and/or Blue-ray discs; magneto-optical media such as optical discs; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drives, memory cards, memory sticks, etc.), and the like.
  • program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.
  • the above-described devices may be configured to act as one or more software modules in order to perform the operations of the above-described example embodiments, or vice versa.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Graphics (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)
  • Image Generation (AREA)
EP15195543.2A 2014-11-25 2015-11-20 Procédé et appareil de génération d'un modèle de visage 3d personnalisé Active EP3026636B1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020140165506A KR101997500B1 (ko) 2014-11-25 2014-11-25 개인화된 3d 얼굴 모델 생성 방법 및 장치

Publications (2)

Publication Number Publication Date
EP3026636A1 true EP3026636A1 (fr) 2016-06-01
EP3026636B1 EP3026636B1 (fr) 2020-01-15

Family

ID=54608430

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15195543.2A Active EP3026636B1 (fr) 2014-11-25 2015-11-20 Procédé et appareil de génération d'un modèle de visage 3d personnalisé

Country Status (3)

Country Link
US (2) US9799140B2 (fr)
EP (1) EP3026636B1 (fr)
KR (1) KR101997500B1 (fr)

Cited By (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108537110A (zh) * 2017-03-01 2018-09-14 索尼公司 生成三维人脸模型的基于虚拟现实的装置和方法
CN109376698A (zh) * 2018-11-29 2019-02-22 北京市商汤科技开发有限公司 人脸建模方法和装置、电子设备、存储介质、产品
US10270983B1 (en) 2018-05-07 2019-04-23 Apple Inc. Creative camera
GB2568475A (en) * 2017-11-15 2019-05-22 Cubic Motion Ltd A method of generating training data
US10325416B1 (en) 2018-05-07 2019-06-18 Apple Inc. Avatar creation user interface
US10362219B2 (en) 2016-09-23 2019-07-23 Apple Inc. Avatar creation and editing
US10528243B2 (en) 2017-06-04 2020-01-07 Apple Inc. User interface camera effects
US10602053B2 (en) 2016-06-12 2020-03-24 Apple Inc. User interface for camera effects
US10645294B1 (en) 2019-05-06 2020-05-05 Apple Inc. User interfaces for capturing and managing visual media
US11054973B1 (en) 2020-06-01 2021-07-06 Apple Inc. User interfaces for managing media
US11061372B1 (en) 2020-05-11 2021-07-13 Apple Inc. User interfaces related to time
US11107261B2 (en) 2019-01-18 2021-08-31 Apple Inc. Virtual avatar animation based on facial feature movement
US11128792B2 (en) 2018-09-28 2021-09-21 Apple Inc. Capturing and displaying images with multiple focal planes
US11212449B1 (en) 2020-09-25 2021-12-28 Apple Inc. User interfaces for media capture and management
US11321857B2 (en) 2018-09-28 2022-05-03 Apple Inc. Displaying and editing images with depth information
US11350026B1 (en) 2021-04-30 2022-05-31 Apple Inc. User interfaces for altering visual media
US11468625B2 (en) 2018-09-11 2022-10-11 Apple Inc. User interfaces for simulated depth effects
US11481988B2 (en) 2010-04-07 2022-10-25 Apple Inc. Avatar editing environment
US11706521B2 (en) 2019-05-06 2023-07-18 Apple Inc. User interfaces for capturing and managing visual media
US11722764B2 (en) 2018-05-07 2023-08-08 Apple Inc. Creative camera
US11770601B2 (en) 2019-05-06 2023-09-26 Apple Inc. User interfaces for capturing and managing visual media
US11776190B2 (en) 2021-06-04 2023-10-03 Apple Inc. Techniques for managing an avatar on a lock screen
US11778339B2 (en) 2021-04-30 2023-10-03 Apple Inc. User interfaces for altering visual media
US11921998B2 (en) 2020-05-11 2024-03-05 Apple Inc. Editing features of an avatar
US12008230B2 (en) 2020-09-24 2024-06-11 Apple Inc. User interfaces related to time with an editable background

Families Citing this family (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3186787A1 (fr) * 2014-08-29 2017-07-05 Thomson Licensing Procédé et dispositif pour enregistrer une image dans un modèle
US9633250B2 (en) * 2015-09-21 2017-04-25 Mitsubishi Electric Research Laboratories, Inc. Method for estimating locations of facial landmarks in an image of a face using globally aligned regression
CN108781251A (zh) * 2016-03-15 2018-11-09 株式会社理光 图像处理装置、图像处理方法和图像处理系统
WO2018053703A1 (fr) * 2016-09-21 2018-03-29 Intel Corporation Estimation de la forme et de la texture précises d'un visage à partir d'une image
US20180101984A1 (en) * 2016-10-06 2018-04-12 Google Inc. Headset removal in virtual, augmented, and mixed reality using an eye gaze database
US10528801B2 (en) * 2016-12-07 2020-01-07 Keyterra LLC Method and system for incorporating contextual and emotional visualization into electronic communications
CN106682598B (zh) * 2016-12-14 2021-02-19 华南理工大学 一种基于级联回归的多姿态的人脸特征点检测方法
WO2018117704A1 (fr) 2016-12-23 2018-06-28 Samsung Electronics Co., Ltd. Appareil électronique et son procédé de fonctionnement
KR101963198B1 (ko) * 2017-04-18 2019-07-31 한국과학기술원 3차원 안면 정면화 시스템 및 방법
CN108960020A (zh) * 2017-05-27 2018-12-07 富士通株式会社 信息处理方法和信息处理设备
CN107818310B (zh) * 2017-11-03 2021-08-06 电子科技大学 一种基于视线的驾驶员注意力检测方法
US10769411B2 (en) * 2017-11-15 2020-09-08 Qualcomm Technologies, Inc. Pose estimation and model retrieval for objects in images
US10573349B2 (en) * 2017-12-28 2020-02-25 Facebook, Inc. Systems and methods for generating personalized emoticons and lip synching videos based on facial recognition
CN108596827B (zh) * 2018-04-18 2022-06-17 太平洋未来科技（深圳）有限公司 三维人脸模型生成方法、装置及电子设备
CN110533761B (zh) * 2018-05-23 2024-01-12 华硕电脑股份有限公司 影像显示方法、电子装置及非瞬时电脑可读取记录媒体
US10783704B2 (en) * 2018-09-27 2020-09-22 Disney Enterprises, Inc. Dense reconstruction for narrow baseline motion observations
CN111340932A (zh) * 2018-12-18 2020-06-26 富士通株式会社 图像处理方法以及信息处理设备
CN110111418B (zh) 2019-05-15 2022-02-25 北京市商汤科技开发有限公司 创建脸部模型的方法、装置及电子设备
EP3977417A4 (fr) * 2019-05-31 2023-07-12 Applications Mobiles Overview Inc. Système et procédé de production d'une représentation 3d d'un objet
JP7276968B2 (ja) * 2019-06-20 2023-05-18 Ｎｅｃソリューションイノベータ株式会社 ３次元データ更新装置、顔向き推定装置、３次元データ更新方法およびプログラム
KR102181901B1 (ko) * 2019-07-25 2020-11-23 넷마블 주식회사 애니메이션 생성 방법
CN110719455A (zh) * 2019-09-29 2020-01-21 深圳市火乐科技发展有限公司 视频投影方法及相关装置
KR20210077993A (ko) 2019-12-18 2021-06-28 삼성전자주식회사 사용자 인증 장치, 사용자 인증 방법, 및 사용자 인증을 위한 학습 방법
CN111754637B (zh) * 2020-06-30 2021-01-29 华东交通大学 一种样本相似度抑制的大尺度三维人脸合成系统
US11403800B1 (en) * 2020-07-13 2022-08-02 Amazon Technologies, Inc. Image generation from 3D model using neural network
CN112200905B (zh) * 2020-10-15 2023-08-22 革点科技(深圳)有限公司 一种三维人脸补全的方法
KR102251781B1 (ko) * 2020-12-30 2021-05-14 (주)라이언로켓 인공신경망을 이용한 입모양 합성 장치 및 방법
CN112926434A (zh) * 2021-02-22 2021-06-08 王先峰 一种基于5g架构的人脸识别设备
US11461970B1 (en) * 2021-03-15 2022-10-04 Tencent America LLC Methods and systems for extracting color from facial image
WO2023230696A1 (fr) * 2022-06-02 2023-12-07 Digital Domain Virtual Human (Us), Inc. Systèmes et procédés d'interpolation de textures à haute résolution sur la base d'expressions faciales
CN116503508B (zh) * 2023-06-26 2023-09-08 南昌航空大学 一种个性化模型构建方法、系统、计算机及可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050063582A1 (en) * 2003-08-29 2005-03-24 Samsung Electronics Co., Ltd. Method and apparatus for image-based photorealistic 3D face modeling
US20070091085A1 (en) * 2005-10-13 2007-04-26 Microsoft Corporation Automatic 3D Face-Modeling From Video
WO2012082077A2 (fr) * 2010-12-17 2012-06-21 Agency For Science, Technology And Research Reconstruction de visage 3d indépendamment de la pose à partir d'une image de visage 2d échantillonnée
WO2012126135A1 (fr) * 2011-03-21 2012-09-27 Intel Corporation Procédé de maquillage augmenté à modélisation de visage tridimensionnelle et alignement de points de repère
US20130287294A1 (en) * 2012-04-30 2013-10-31 Cywee Group Limited Methods for Generating Personalized 3D Models Using 2D Images and Generic 3D Models, and Related Personalized 3D Model Generating System

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6208347B1 (en) 1997-06-23 2001-03-27 Real-Time Geometry Corporation System and method for computer modeling of 3D objects and 2D images by mesh constructions that incorporate non-spatial data such as color or texture
EP1039417B1 (fr) 1999-03-19 2006-12-20 Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. Méthode et appareil de traitement d'images basés sur des modèles à métamorphose
JP3639475B2 (ja) * 1999-10-04 2005-04-20 シャープ株式会社 ３次元モデル生成装置および３次元モデル生成方法ならびに３次元モデル生成プログラムを記録した記録媒体
US6807290B2 (en) * 2000-03-09 2004-10-19 Microsoft Corporation Rapid computer modeling of faces for animation
JP4341135B2 (ja) 2000-03-10 2009-10-07 コニカミノルタホールディングス株式会社 物体認識装置
KR100372811B1 (ko) 2000-12-19 2003-02-26 (주) 웹씨인터미디어 2차원 얼굴 정면 이미지를 이용한 3차원 얼굴 모델 생성방법
KR100459892B1 (ko) * 2001-12-13 2004-12-03 삼성전자주식회사 3차원 얼굴 모델을 위한 텍스쳐 생성 방법 및 장치
JP2003208601A (ja) 2002-01-15 2003-07-25 Nec Corp ３次元物体撮影装置、３次元形状モデル生成装置、３次元形状モデル生成方法、３次元形状モデル生成プログラム
KR100450210B1 (ko) 2002-10-25 2004-09-24 (주)엔룩스 3차원 스캔 얼굴모델의 합성 시스템과 그 방법 및 3차원스캔 얼굴모델의 합성기능을 갖춘 프로그램이 수록된기록매체
JP4216824B2 (ja) * 2005-03-07 2009-01-28 株式会社東芝 ３次元モデル生成装置、３次元モデル生成方法および３次元モデル生成プログラム
US7415152B2 (en) 2005-04-29 2008-08-19 Microsoft Corporation Method and system for constructing a 3D representation of a face from a 2D representation
WO2007069255A2 (fr) 2005-12-14 2007-06-21 Yeda Research And Development Co. Ltd. Exemple base sur la reconstruction 3d
US7856125B2 (en) * 2006-01-31 2010-12-21 University Of Southern California 3D face reconstruction from 2D images
KR100919104B1 (ko) 2008-01-03 2009-09-28 에스케이 텔레콤주식회사 발신자의 3차원 얼굴 표시장치 및 방법
US8260039B2 (en) 2009-02-25 2012-09-04 Seiko Epson Corporation Object model fitting using manifold constraints
US8204301B2 (en) * 2009-02-25 2012-06-19 Seiko Epson Corporation Iterative data reweighting for balanced model learning
SG181671A1 (en) * 2009-12-14 2012-07-30 Agency Science Tech & Res Method and system for single view image 3 d face synthesis
KR20110088361A (ko) 2010-01-26 2011-08-03 한국전자통신연구원 정면 얼굴 영상 생성 장치 및 방법
US8861800B2 (en) 2010-07-19 2014-10-14 Carnegie Mellon University Rapid 3D face reconstruction from a 2D image and methods using such rapid 3D face reconstruction
KR101514327B1 (ko) 2010-11-04 2015-04-22 한국전자통신연구원 얼굴 아바타 생성 장치 및 방법
KR101608253B1 (ko) 2011-08-09 2016-04-01 인텔 코포레이션 이미지 기반 멀티 뷰 3d 얼굴 생성
JP2013196294A (ja) 2012-03-19 2013-09-30 Toshiba Corp 人物画像処理装置、及び人物画像処理方法
US20130286161A1 (en) 2012-04-25 2013-10-31 Futurewei Technologies, Inc. Three-dimensional face recognition for mobile devices
KR20130137999A (ko) 2012-06-08 2013-12-18 주식회사 코어셀 시청자의 시청률을 측정하는 방법과 그 시스템

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050063582A1 (en) * 2003-08-29 2005-03-24 Samsung Electronics Co., Ltd. Method and apparatus for image-based photorealistic 3D face modeling
US20070091085A1 (en) * 2005-10-13 2007-04-26 Microsoft Corporation Automatic 3D Face-Modeling From Video
WO2012082077A2 (fr) * 2010-12-17 2012-06-21 Agency For Science, Technology And Research Reconstruction de visage 3d indépendamment de la pose à partir d'une image de visage 2d échantillonnée
WO2012126135A1 (fr) * 2011-03-21 2012-09-27 Intel Corporation Procédé de maquillage augmenté à modélisation de visage tridimensionnelle et alignement de points de repère
US20130287294A1 (en) * 2012-04-30 2013-10-31 Cywee Group Limited Methods for Generating Personalized 3D Models Using 2D Images and Generic 3D Models, and Related Personalized 3D Model Generating System

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"3D Imaging for Safety and Security", vol. 35, 1 January 2007, SPRINGER NETHERLANDS, Dordrecht, ISBN: 978-1-40-206181-3, article S. HUQ ET AL: "A Survey on 3D Modeling of Human Faces for Face Recognition", pages: 25 - 67, XP055184073, DOI: 10.1007/978-1-4020-6182-0_2 *
YUJIAN GAO ET AL: "Automatic construction of 3D animatable facial avatars", COMPUTER ANIMATION AND VIRTUAL WORLDS, 1 January 2010 (2010-01-01), pages n/a - n/a, XP055032197, ISSN: 1546-4261, DOI: 10.1002/cav.340 *

Cited By (60)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11869165B2 (en) 2010-04-07 2024-01-09 Apple Inc. Avatar editing environment
US11481988B2 (en) 2010-04-07 2022-10-25 Apple Inc. Avatar editing environment
US11641517B2 (en) 2016-06-12 2023-05-02 Apple Inc. User interface for camera effects
US11245837B2 (en) 2016-06-12 2022-02-08 Apple Inc. User interface for camera effects
US11165949B2 (en) 2016-06-12 2021-11-02 Apple Inc. User interface for capturing photos with different camera magnifications
US11962889B2 (en) 2016-06-12 2024-04-16 Apple Inc. User interface for camera effects
US10602053B2 (en) 2016-06-12 2020-03-24 Apple Inc. User interface for camera effects
US10362219B2 (en) 2016-09-23 2019-07-23 Apple Inc. Avatar creation and editing
US10572720B2 (en) 2017-03-01 2020-02-25 Sony Corporation Virtual reality-based apparatus and method to generate a three dimensional (3D) human face model using image and depth data
EP3370208A3 (fr) * 2017-03-01 2018-12-12 Sony Corporation Appareil de réalité virtuelle et procédé pour générer un modèle de visage humain tridimensionnel (3d) à l'aide d'images et de données de profondeur
CN108537110B (zh) * 2017-03-01 2022-06-14 索尼公司 生成三维人脸模型的基于虚拟现实的装置和方法
CN108537110A (zh) * 2017-03-01 2018-09-14 索尼公司 生成三维人脸模型的基于虚拟现实的装置和方法
US10528243B2 (en) 2017-06-04 2020-01-07 Apple Inc. User interface camera effects
US11204692B2 (en) 2017-06-04 2021-12-21 Apple Inc. User interface camera effects
US11687224B2 (en) 2017-06-04 2023-06-27 Apple Inc. User interface camera effects
GB2568475A (en) * 2017-11-15 2019-05-22 Cubic Motion Ltd A method of generating training data
US11178335B2 (en) 2018-05-07 2021-11-16 Apple Inc. Creative camera
US10523879B2 (en) 2018-05-07 2019-12-31 Apple Inc. Creative camera
US10325416B1 (en) 2018-05-07 2019-06-18 Apple Inc. Avatar creation user interface
US10410434B1 (en) 2018-05-07 2019-09-10 Apple Inc. Avatar creation user interface
US11722764B2 (en) 2018-05-07 2023-08-08 Apple Inc. Creative camera
US11682182B2 (en) 2018-05-07 2023-06-20 Apple Inc. Avatar creation user interface
US10861248B2 (en) 2018-05-07 2020-12-08 Apple Inc. Avatar creation user interface
US11380077B2 (en) 2018-05-07 2022-07-05 Apple Inc. Avatar creation user interface
US10375313B1 (en) 2018-05-07 2019-08-06 Apple Inc. Creative camera
US10270983B1 (en) 2018-05-07 2019-04-23 Apple Inc. Creative camera
US10580221B2 (en) 2018-05-07 2020-03-03 Apple Inc. Avatar creation user interface
US10325417B1 (en) 2018-05-07 2019-06-18 Apple Inc. Avatar creation user interface
US11468625B2 (en) 2018-09-11 2022-10-11 Apple Inc. User interfaces for simulated depth effects
US11128792B2 (en) 2018-09-28 2021-09-21 Apple Inc. Capturing and displaying images with multiple focal planes
US11895391B2 (en) 2018-09-28 2024-02-06 Apple Inc. Capturing and displaying images with multiple focal planes
US11321857B2 (en) 2018-09-28 2022-05-03 Apple Inc. Displaying and editing images with depth information
US11669985B2 (en) 2018-09-28 2023-06-06 Apple Inc. Displaying and editing images with depth information
CN109376698A (zh) * 2018-11-29 2019-02-22 北京市商汤科技开发有限公司 人脸建模方法和装置、电子设备、存储介质、产品
US11107261B2 (en) 2019-01-18 2021-08-31 Apple Inc. Virtual avatar animation based on facial feature movement
US10645294B1 (en) 2019-05-06 2020-05-05 Apple Inc. User interfaces for capturing and managing visual media
US10791273B1 (en) 2019-05-06 2020-09-29 Apple Inc. User interfaces for capturing and managing visual media
US10652470B1 (en) 2019-05-06 2020-05-12 Apple Inc. User interfaces for capturing and managing visual media
US10674072B1 (en) 2019-05-06 2020-06-02 Apple Inc. User interfaces for capturing and managing visual media
US10681282B1 (en) 2019-05-06 2020-06-09 Apple Inc. User interfaces for capturing and managing visual media
US11770601B2 (en) 2019-05-06 2023-09-26 Apple Inc. User interfaces for capturing and managing visual media
US11223771B2 (en) 2019-05-06 2022-01-11 Apple Inc. User interfaces for capturing and managing visual media
US10735643B1 (en) 2019-05-06 2020-08-04 Apple Inc. User interfaces for capturing and managing visual media
US11706521B2 (en) 2019-05-06 2023-07-18 Apple Inc. User interfaces for capturing and managing visual media
US10735642B1 (en) 2019-05-06 2020-08-04 Apple Inc. User interfaces for capturing and managing visual media
US11822778B2 (en) 2020-05-11 2023-11-21 Apple Inc. User interfaces related to time
US11061372B1 (en) 2020-05-11 2021-07-13 Apple Inc. User interfaces related to time
US11442414B2 (en) 2020-05-11 2022-09-13 Apple Inc. User interfaces related to time
US11921998B2 (en) 2020-05-11 2024-03-05 Apple Inc. Editing features of an avatar
US11054973B1 (en) 2020-06-01 2021-07-06 Apple Inc. User interfaces for managing media
US11617022B2 (en) 2020-06-01 2023-03-28 Apple Inc. User interfaces for managing media
US11330184B2 (en) 2020-06-01 2022-05-10 Apple Inc. User interfaces for managing media
US12008230B2 (en) 2020-09-24 2024-06-11 Apple Inc. User interfaces related to time with an editable background
US11212449B1 (en) 2020-09-25 2021-12-28 Apple Inc. User interfaces for media capture and management
US11416134B1 (en) 2021-04-30 2022-08-16 Apple Inc. User interfaces for altering visual media
US11350026B1 (en) 2021-04-30 2022-05-31 Apple Inc. User interfaces for altering visual media
US11418699B1 (en) 2021-04-30 2022-08-16 Apple Inc. User interfaces for altering visual media
US11778339B2 (en) 2021-04-30 2023-10-03 Apple Inc. User interfaces for altering visual media
US11539876B2 (en) 2021-04-30 2022-12-27 Apple Inc. User interfaces for altering visual media
US11776190B2 (en) 2021-06-04 2023-10-03 Apple Inc. Techniques for managing an avatar on a lock screen

Also Published As

Publication number Publication date
KR20160062572A (ko) 2016-06-02
US20180018819A1 (en) 2018-01-18
KR101997500B1 (ko) 2019-07-08
US9928647B2 (en) 2018-03-27
US20160148425A1 (en) 2016-05-26
EP3026636B1 (fr) 2020-01-15
US9799140B2 (en) 2017-10-24

Similar Documents

Publication Publication Date Title
US9928647B2 (en) Method and apparatus for generating personalized 3D face model
US10482656B2 (en) 3D face modeling methods and apparatuses
KR102523512B1 (ko) 얼굴 모델의 생성
EP2993614B1 (fr) Procédé et dispositif pour la reconnaissance faciale
US10268875B2 (en) Method and apparatus for registering face, and method and apparatus for recognizing face
EP3139303B1 (fr) Procédé et appareil pour le suivi de l'oeil
CN109829875B (zh) 用于估计视差的方法和装置
CN106547092B (zh) 用于补偿头戴式显示器的移动的方法和设备
EP3096263B1 (fr) Procédé de reconnaissance d'orientation du corps humain et système basé sur une caméra à deux lentilles
CN110853036A (zh) 用于训练对象检测模型的方法及设备
US9953247B2 (en) Method and apparatus for determining eye position information
CN112070782B (zh) 识别场景轮廓的方法、装置、计算机可读介质及电子设备
WO2012100225A1 (fr) Systèmes et procédés de génération de forme tridimensionnelle à partir d'images en couleur stéréoscopiques
US10582179B2 (en) Method and apparatus for processing binocular disparity image
US11189053B2 (en) Information processing apparatus, method of controlling information processing apparatus, and non-transitory computer-readable storage medium
JP2015184054A (ja) 同定装置、方法及びプログラム
KR102068489B1 (ko) 3차원 객체 생성 장치
KR101678455B1 (ko) 질감인지공간을 이용한 햅틱 속성 적용 장치 및 방법
CN115953464A (zh) 全局定位方法和装置
KR20230060029A (ko) 평면 검출 장치 및 방법
KR102269088B1 (ko) 동공 추적 장치 및 방법
KR20160107655A (ko) 패턴 이미지 기반 얼굴 마스크 제작 장치 및 방법
US11954899B2 (en) Systems and methods for training models to predict dense correspondences in images using geodesic distances
KR101521358B1 (ko) 다중 모델 정합 방법 및 그 장치
US20230005182A1 (en) Estimation method, estimation apparatus and program

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20161124

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20181002

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190828

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602015045535

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1225817

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200215

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200115

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200607

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200415

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200515

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200416

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200415

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602015045535

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1225817

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200115

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20201016

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201120

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20201130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200115

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230530

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231023

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231024

Year of fee payment: 9

Ref country code: DE

Payment date: 20231023

Year of fee payment: 9