WO2015192369A1 - Appareil et procédé de reconstruction de modèle de visage 3d - Google Patents

Appareil et procédé de reconstruction de modèle de visage 3d Download PDF

Info

Publication number
WO2015192369A1
WO2015192369A1 PCT/CN2014/080382 CN2014080382W WO2015192369A1 WO 2015192369 A1 WO2015192369 A1 WO 2015192369A1 CN 2014080382 W CN2014080382 W CN 2014080382W WO 2015192369 A1 WO2015192369 A1 WO 2015192369A1
Authority
WO
WIPO (PCT)
Prior art keywords
face
locations
model
landmarks
detected
Prior art date
Application number
PCT/CN2014/080382
Other languages
English (en)
Inventor
Minje Park
Olivier DUCHENNE
Yeongjae Cheon
Tae-Hoon Kim
Xiaolu Shen
Yangzhou Du
Wooju RYU
Myung-Ho Ju
Original Assignee
Intel Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intel Corporation filed Critical Intel Corporation
Priority to KR1020167032315A priority Critical patent/KR101828201B1/ko
Priority to CN201480078999.XA priority patent/CN106462995B/zh
Priority to PCT/CN2014/080382 priority patent/WO2015192369A1/fr
Priority to US14/443,337 priority patent/US9679412B2/en
Priority to EP14895424.1A priority patent/EP3158535A4/fr
Publication of WO2015192369A1 publication Critical patent/WO2015192369A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T17/205—Re-meshing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/653—Three-dimensional objects by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/167—Detection; Localisation; Normalisation using comparisons between temporally consecutive images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure relates to the field of data processing. More particularly, the present disclosure relates to three-dimensional (3D) face model reconstruction method and apparatus.
  • 3D face models have more expandability than two dimensional (2D) face models, and enable developers/designers of applications to provide users with richer experiences, e.g., in visual communication and entertainment.
  • 2D face models make a 3D face model from a single image is not an easy task.
  • tracking expressions and head pose of the 3D mesh in a single video stream is an even more challenging problem.
  • Prior art techniques typically require a complex registration step that may require 5-10 minutes, and they tend to be insufficiently robust for out-of-plane rotation.
  • prior art techniques tend to provide coverage for a limited to small sets of face shapes or expressions, and typically, the amount of computational resources required exceed or at least strain the capability of the more resource constrained devices, such as mobile devices like smartphones or computing tablets.
  • FIG. 1 is a block diagram for a computing device incorporated with the
  • 3D face modeler of the present disclosure according to the various embodiments.
  • Figure 2 illustrates an example facial landmark detector, according to the various embodiments.
  • Figure 3 illustrates an example operational flow of a model fitter, according to various embodiments.
  • Figure 4 illustrates an example operational flow of a model tracker, according to various embodiments
  • Figure 5 illustrates an example operational flow of a validator, according to various embodiments.
  • Figure 6 illustrates an example computer system suitable for use to practice various aspects of the present disclosure, according to various embodiments.
  • Figure 7 illustrates a storage medium having instructions for practicing methods described with references to Figures 2-5, according to disclosed embodiments.
  • an apparatus may include a facial landmark detector, a model fitter and a model tracker, coupled with each other.
  • the facial landmark detector may be configured to detect a plurality of landmarks of a face and their locations within each of a plurality of image frames.
  • the model fitter may be coupled to the facial landmark detector and configured to generate or regenerate a 3D model of the face from a 3D model of a neutral face, in view of detected landmarks of the face and their locations within one of the plurality of image frames.
  • the model tracker may be coupled to the facial landmark detector and the model fitter, and configured to adjust the 3D model generated by the model fitter to track the face in subsequent image frames. Model tracker may successively update the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames.
  • the apparatus may further include a validator coupled with the model tracker and configured to monitor the face tracking for excessive drifting, and cause the model fitter to regenerate the 3D model when excessive drifting is detected.
  • the facial landmark detector may include a face detector, an initial facial landmark detector, and one or more facial landmark detection linear regressors, coupled with each other.
  • the face detector may be configured to detect a face within an image frame
  • the initial facial detector may be coupled with the face detector and configured to detect a number of facial landmarks, including an inner eye corner, and their locations within a detected face.
  • the one or more facial landmark detection linear regressors may be coupled with the face detector and the initial facial detector and configured to iteratively refine the facial landmark locations with increased precision.
  • the facial landmark detector may also include a validator to reconstruct a face from the facial landmarks, and an image frame, and provide the reconstructed image frame to the face and initial facial landmark detectors to detect whether a face and one or more eyes are present within the reconstructed image frame.
  • the model fitter maybe configured to generate or regenerate a 3D model of the face from the neutral face by determining a 3D mesh for the face, employing a point based facial landmark model, an ellipsoidal head model to estimate rigid motion of the face, and non-rigid deformable mesh model for facial shapes and expression.
  • the model tracker may be similarly constituted as the model fitter, except the model tracker may be configured to maintain the 3D model of the face, through adjustment of the resulting 3D model of the face of the previous image frame.
  • these elements may enable 3D face model to be reconstructed for each image frame of a video stream more efficiently, requiring less computational resources, thus more suitable for resource constrained computing devices, such as mobile computing devices. Further, the reconstructed 3D face model may be more robust and able to track a wide range of facial expressions and head poses.
  • phrase “A and/or B” means (A), (B), or (A and B).
  • phrase “A, B, and/or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C).
  • module may refer to, be part of, or include an
  • ASIC Application Specific Integrated Circuit
  • an electronic circuit a processor (shared, dedicated, or group) and/or memory (shared, dedicated, or group) that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.
  • processor shared, dedicated, or group
  • memory shared, dedicated, or group
  • computing device 50 may include 3D face modeler 100.
  • 3D face modeler 100 may include facial landmark detector 102, model fitter 104 and model tracker 106, coupled with each other.
  • Facial landmark detector 102 may be configured to detect a plurality of landmarks of a face and their locations within each of a plurality of image frames 110.
  • Model fitter 104 may be coupled to the facial landmark detector 102 and configured to generate or regenerate a 3D model of the face from a 3D model of a neutral face, in view of detected landmarks of the face and their locations within one of the plurality of image frames 110.
  • Model tracker 106 may be coupled to facial landmark detector 102 and model fitter 104, and configured to adjust the 3D model generated by model fitter 104 to track the face in subsequent image frames. Model tracker 106 may successively update the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames 110.
  • 3D face modeler 100 may further include validator 108 coupled with model tracker 106 and model fitter 104, to monitor the face tracking for excessive drifting, and cause model fitter 104 to regenerate the 3D face model when excessive drifting is detected.
  • facial landmark detector 102 may be configured to detect a sparse number of landmark positions, e.g., 30 - 80, in a face detected within each of the plurality of image frames 110.
  • the sparse number of landmarks may be detected from a face rotated +/- 45 degrees to +/- 30 degrees.
  • the sparse number of landmark positions of the face in each of the plurality of image frames 110 may be provided to model fitter 104 to generate/regenerate a 3D model of the face from a neutral face, and to model tracker 106 to successively maintain the 3D model of the face generated by model fitter 104.
  • Example embodiments of facial landmark detector will be described later with references to Figure 2. Other embodiments, beside those later described with references to Figure 2 may also be employed instead.
  • model fitter 104 may be configured to generate or regenerate a 3D model of the face from the neutral face, by determining a 3D mesh for the face, employing a plurality of models, such as a point based facial landmark model, an ellipsoidal head model to estimate rigid motion of the face, and/or non-rigid deformable mesh model for facial shapes and expression. More specifically, in embodiments, model fitter 104 may be configured to determine a 3D for the face, through solving the following objective function:
  • E Ei an dmark + E e Ui psoi dal + Emesh (1)
  • Ei an dmark is the amount of error between a point based facial landmark model and the input facial landmark positions of an image frame;
  • Eeiiipsoidai is the projected pixel value differences between an ellipsoidal head model to estimate rigid motion of the face, and the image frame;
  • ⁇ mesh is the projected pixel value differences between a non-rigid deformable mesh model for facial shapes and expression, and the image frame;
  • E is the sum of the above three error components.
  • model fitter 104 may estimate all model parameters from neutral shape, expression, and head pose from scratch. Model fitter 104 may consider inter-person shape variations as well as intra-person expression variations.
  • model tracker 106 may be similarly constituted as model fitter 104, except model tracker 106 may be configured to maintain the 3D model of the face, through adjustment of the resulting 3D model of the face of the previous image frame. Further, model tracker 106 may fix inter-person shape variations, and consider only intra-person expression variations. In other words, to further improve computational efficiency, model tracker 106 assumes it is tracking the face of the same object or person. Example embodiments of the operational flow of model tracker 106 will be later described with references to Figure 4.
  • validator 108 may be configured to monitor the face tracking for excessive drifting through detection of whether inaccuracies (errors) of the 3D model successively adjusted by model tracker 106 has grown to exceed a threshold. While usage of previous frame's resulting 3D model as the starting 3D model of the current frame improves computational efficiency, tracking error can be accumulated as time proceeds. Although the error minimization term which is related to facial landmark detections can reduce the amount of drifting while tracking, the drifting artifact is unlikely to be completely prevented. Thus, validator 108 serves to ensure accuracy is not overly sacrificed for improvement in computational efficiency. Example embodiments of the operational flow of validator 108 will be later described with references to Figure 5.
  • 3D face modeler 100 may be implemented in hardware, software, or a combination thereof.
  • 3D face modeler 100 may be implemented, e.g., with application specific integrated circuits (ASIC), or programmable logic devices (PLD) programmed with the operational logic of 3D face modeler 100.
  • ASIC application specific integrated circuits
  • PLD programmable logic devices
  • computing device 50 may further include one or more processors, and memory, and one or more of facial landmark detector 102, model fitter 104, model tracker 106, and validator 108 may be implemented in assembler/machine instructions supported by the processor or any one of a number of programming languages compilable into the assembler/machine language supported by the processors.
  • computing device 50 may further include a number of peripheral devices such as display devices, input/output devices, or communication/networking devices.
  • computing device 50 may be any one of a number of computing devices known in the art, including but are not limited to smartphones, ultrabooks, computing tablets, laptops, desktop computers, servers, set-top boxes or game consoles.
  • resource constrained devices such as mobile devices like smartphones, and/or computing tablets.
  • example facial landmark detector 200 may include face detector 202, initial facial landmark detector 204, a number of facial landmarks detection linear regressors 206, and validator 208.
  • detector 202, facial landmark detector 204, facial landmarks detection linear regressors 206, and validator 208 may be configured to cooperate to take a coarse-to-fine approach.
  • a rough location of a face within an image frame may be estimated, and then a sparse number of facial landmarks may be localized based on fern-based cascaded regression. Then, several consecutive regularized linear regressors may be applied to more accurately identify the locations of the facial landmarks.
  • example facial landmark detector 200 may serve as facial landmark detector 102 of Figure 1.
  • face detector 202 may be configured to identify an area, e.g., a rectangular area, of an image frame as containing a face. Face detector 202 may be configured to search several different image scales to effectively find faces of different sizes for each given image frame. Statistics such as averages and standard deviations of distances between found rectangles and actual face bounding boxes may be pre-computed with a large number of training images, and later used to compensate the difference between face rectangle and actual face region.
  • Initial facial landmark detector 204 may be coupled with the face detector and configured to detect, within each area identified as having a face, a number of facial landmarks of the face, including in particular, inner eye corner, and their locations. In embodiments, to enhance computational efficiency, a sparse number of facial landmarks are detected. Experience has shown that inner eye corners are highly distinguishable and robust against facial expressions and poses in human faces. These facial landmarks and their initial locations may then be used to serve as initial facial landmarks to be refined by one or more linear regressors 206. Employment of linear regressors, as opposed to nonlinear regressors, enables high speed performance while avoiding data over-fitting, which in turn, enables handling of a wide range of face poses, expressions, and illumination variations.
  • the one or more facial landmark detection linear regressors 206 may be coupled with the face detector 202 and the initial facial landmark detector 204 and configured to iteratively refine and more accurately identify the facial landmark locations.
  • four (4) linear regresssors are employed.
  • the first linear regressor 206 may be trained with relatively large variations between input and humanly annotated ground truth data of facial landmarks in images (hereinafter simply, ground truth data) to cover initial errors of face and initial facial landmark detectors 202 and 204.
  • the remaining linear regressors 206 may be trained with decreasing variations between input and ground truth data, with each regression iteration refines the facial landmark positions with more precision.
  • each regressor 206 may be trained to minimize total distance between current landmark positions and ground truth annotations, as well as minimize the norm of the regressor 206. Resultantly, the signal to noise ratio of the output facial landmark positions may be improved.
  • the regressors 206 may be configured to work with gradient of an image frame, as opposed to intensity values of the image frame. Experience has shown working with image gradient may improve the robustness of the computation with respect to illumination changes.
  • validator 208 may be configured to reconstruct the face for the image frame from the one or more detected facial landmarks, and the image frame, and provide the reconstructed image frame to face and initial facial landmark detectors 202 and 204 again to check if face and initial facial landmark detectors 202 and 204 are able to detect whether a face and one or more eyes are present within the reconstructed image frame.
  • face and initial facial landmark detection may be skipped for the next frame. Instead, the facial landmark positions of the previous frame may be employed for the current frame. The process may then continue, until the validation fails. Then, the face detection and initial facial landmark detection may be performed again.
  • each of face detector 202, initial facial landmark detector 204, and regressors 206 may be pre-trained with a large number of training examples. In embodiments, each of face detector 202, initial facial landmark detector 204, and regressors 206 is trained with tens of thousands of facial images with a wide range of facial expressions and head poses.
  • example process 300 for generating an initial 3D face model from a neutral face model, based on facial landmarks identified for a face within an image frame may include operations performed in blocks 302-310.
  • the operations may be performed e.g., by the earlier describer model fitter 104 of Figure 1. While for ease of understanding, operations of blocks 302-310 will be described in sequence, it will be understood that in alternate embodiments, selected ones of the operations may be performed in different order.
  • facial landmark positions of a face within an image frame may be received.
  • an amount of error between a point based facial landmark model and the input facial landmark positions of an image frame may be calculated.
  • E e ui psoid ai an amount of projected pixel value differences between an ellipsoidal head model to estimate rigid motion of the face, and the image frame may also be calculate.
  • Zi mesh an amount of projected pixel value differences between a non-rigid deformable mesh model for facial shapes and expression, and the image frame may likewise be calculated.
  • the model parameters of a 3D mesh and head pose may be adjusted, and Ei andmark , E e u ipsoida i and/or E mesh re-calculated, in an attempt to reduce Eiandmark , E e mp S oidai and/or Emesh-
  • a determination may be made whether the improvement in reducing Ei andmark , E e u ips0lda i and/or £ m esh.is marginal, e.g., below a predetermined threshold; in other words, E, the total of Ei andmark , E e u ips0lda i and Emesh is effectively minimized.
  • process 300 may end, else process 300 may return to block 306 and continue therefrom as earlier described.
  • all model parameters from neutral shape, expression, and head pose may be estimated from scratch. Inter-person shape variations as well as intra-person expression variations may be considered.
  • example process 400 for adjusting the 3D face model to track facial expression and head pose may include operations performed in blocks 402-410.
  • the operations are similar to the operations of blocks 302-310, except the operations start with the 3D model of the last image frame, as opposed to from the neutral frame.
  • the operations may be performed e.g., by the earlier describer model tracker 106 of Figure 1. While for ease of understanding, operations of blocks 402-410 will be described in sequence, it will be understood that in alternate embodiments, selected ones of the operations may be performed in different order.
  • facial landmark positions of a face within an image frame may be received.
  • Ei an dmark an amount of error between a point based facial landmark model and the input facial landmark positions of an image frame may be calculated.
  • E e in psoi d a i an amount of projected pixel value differences between an ellipsoidal head model to estimate rigid motion of the face, and the image frame may also be calculated.
  • Emesh an amount of projected pixel value differences between a non-rigid deformable mesh model for facial shapes and expression, and the image frame may likewise be calculated.
  • the model parameters of the 3D mesh and the head pose may be adjusted and Ei andmark , E e u ips0lda i and/or Emesh re-calcualted in an attempt to reduce Ei andmark , E eUips0lda i and/or E mesh .
  • a determination may be made whether the improvement in reducing Ei andmark , E e u ips0lda i and/or Emesh-is marginal, e.g., below a predetermined threshold; in other words, E, the total of Ei and mark , Eeinpsoidai and Emesh is effectively minimized.
  • process 400 may end, else process 400 may return to block 406 and continue therefrom as earlier described.
  • inter- person shape variations may be fixed, with only intra-person expression variations considered.
  • example process 500 for monitoring and detecting for excessive drifting in tracking the 3D face model may include operations performed in blocks 502-508.
  • the operations may be performed e.g., by the earlier describer validator 108 of Figure 1. While for ease of understanding, operations of blocks 502-508 will be described in sequence, it will be understood that in alternate embodiments, selected ones of the operations may be performed in different order.
  • the latest amount of error (cumulative drift effect) of the 3D model tracked may be determined.
  • a determination may be made as to whether the latest amount of error (cumulative drifting effect) is in excess of a pre- configured threshold. If the latest amount of error (cumulative drifting effect) is not in excess of the pre-configured threshold, at block 506, the present 3D model may be allowed to be continually tracked and refined for the next image frame. Process 500 may continue thereafter from block 502 as earlier described. On the other hand, if the latest amount of error (cumulated drifting effect) is in excess of the pre-configured threshold, at block 508, a new 3D model may be caused to be re-generated from the neutral shape for the current or next image frame.
  • FIG. 6 illustrates an example computer system that may be suitable for use as a client device or a server to practice selected aspects of the present disclosure.
  • computer 800 may include one or more processors or processor cores 802, and system memory 804.
  • processors or processor cores may be considered synonymous, unless the context clearly requires otherwise.
  • computer 800 may include mass storage devices 806 (such as diskette, hard drive, compact disc read only memory (CD-ROM) and so forth), input/output devices 808 (such as display, keyboard, cursor control and so forth) and communication interfaces 810 (such as network interface cards, modems and so forth).
  • the elements may be coupled to each other via system bus 812, which may represent one or more buses. In the case of multiple buses, they may be bridged by one or more bus bridges (not shown).
  • system memory 804 and mass storage devices 806 may be employed to store a working copy and a permanent copy of the programming instructions
  • computational logic 822 implementing the operations associated with facial landmark detector 102, model fitter 104, model tracker 106, and validator 108, earlier described, collectively referred to as computational logic 822.
  • the various elements may be implemented by assembler instructions supported by processor(s) 802 or high-level languages, such as, for example, C, that can be compiled into such instructions.
  • the number, capability and/or capacity of these elements 810 - 812 may vary, depending on whether computer 800 is used as a client device or a server. When use as client device, the capability and/or capacity of these elements 810 - 812 may vary, depending on whether the client device is a stationary or mobile device, like a smartphone, computing tablet, ultrabook or laptop. Otherwise, the constitutions of elements 810-812 are known, and accordingly will not be further described.
  • the present disclosure may be embodied as methods or computer program products. Accordingly, the present disclosure, in addition to being embodied in hardware as earlier described, may take the form of an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to as a "circuit," "module” or “system.” Furthermore, the present disclosure may take the form of a computer program product embodied in any tangible or non-transitory medium of expression having computer-usable program code embodied in the medium.
  • Non-transitory computer-readable storage medium 902 may include a number of programming instructions 904.
  • Programming instructions 904 may be configured to enable a device, e.g., computer 800, in response to execution of the programming instructions, to perform, e.g., various operations associated with facial landmark detector 102, model fitter 104, model tracker 106, and validator 108.
  • programming instructions 904 may be disposed on multiple computer-readable non-transitory storage media 902 instead.
  • programming instructions 904 may be disposed on computer-readable transitory storage media 902, such as, signals.
  • the computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.
  • the computer-readable medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or Flash memory erasable programmable read-only memory
  • CD-ROM compact disc read-only memory
  • CD-ROM compact disc read-only memory
  • a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device.
  • the computer- usable or computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
  • a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
  • the computer- usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave.
  • the computer usable program code may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc.
  • Computer program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
  • LAN local area network
  • WAN wide area network
  • Internet Service Provider for example, AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
  • These computer program instructions may also be stored in a computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.
  • the computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • Embodiments may be implemented as a computer process, a computing system or as an article of manufacture such as a computer program product of computer readable media.
  • the computer program product may be a computer storage medium readable by a computer system and encoding a computer program instructions for executing a computer process.
  • processors 802 may be packaged together with memory having computational logic 822 (in lieu of storing on memory 804 and storage 806).
  • processors 802 may be packaged together with memory having computational logic 822 to form a System in Package (SiP).
  • SiP System in Package
  • processors 802 may be integrated on the same die with memory having computational logic 822.
  • processors 802 may be packaged together with memory having computational logic 822 to form a System on Chip (SoC).
  • SoC System on Chip
  • the SoC may be utilized in, e.g., but not limited to, a smartphone or computing tablet.
  • Example 1 may be an apparatus equipped to construct a three-dimensional, 3D, face model.
  • the apparatus may comprise a facial landmark detector, a model fitter, and a model tracker, coupled with each other.
  • the facial landmark detector may be configured to detect a plurality of landmarks of a face, and their locations, within each of a plurality of image frames.
  • the model fitter may be configured to generate a 3D model of the face from a 3D model of a neutral face, in view of detected landmarks of the face and their locations within a first one of the plurality of image frames.
  • the model tracker may be configured to adjust the 3D model to track the face in subsequent image frames, successively adjusting the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames.
  • Example 2 may be a facial landmark detector, e.g., the facial landmark detector of example 1.
  • the facial landmark detector may include a face detector, an initial facial landmark detector, and one or more facial landmark detection linear regressors, coupled with each other.
  • the face detector may be configured to detect a face within an image frame.
  • the initial facial landmark detector may be configured to detect one or more landmarks, including an inner eye corner, and their locations within a detected face.
  • the one or more facial landmark detection linear regressors may be configured to iteratively refine the facial landmark locations of the detected face with increased precision.
  • Example 4 may example 2 or 3, wherein the one or more face landmark detection linear regressors may comprise a first face landmark detection linear regressor to refine facial landmark locations to a first precision, and a second facial landmark detection linear regressor to refine the facial landmark locations from the first precision to a second precision, wherein the second precision is more precise than the first precision.
  • the one or more face landmark detection linear regressors may comprise a first face landmark detection linear regressor to refine facial landmark locations to a first precision, and a second facial landmark detection linear regressor to refine the facial landmark locations from the first precision to a second precision, wherein the second precision is more precise than the first precision.
  • Example 5 may be example 1 (optionally having a facial landmark detector of any one of examples 2-4,) wherein the model fitter may be configured to estimate an initial 3D mesh and head pose, in view of the detected landmarks of the face and their locations within the first image frame.
  • Example 6 may be example 5, wherein the model fitter may be configured to further modify the initial 3D mesh and head pose to reduce a first amount of differences between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected by the facial landmark detector.
  • Example 7 may be example 5 or 6, wherein the model fitter may be configured to further modify the initial 3D mesh and head pose to reduce a second amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected by the facial landmark detector.
  • Example 8 may be any one of examples 5-7, wherein the model fitter may be configured to further modify the initial 3D mesh and head pose to reduce a third amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected by the facial landmark detector.
  • Example 9 may be example 5, wherein the model fitter may be configured to further modify the initial 3D mesh and head pose to reduce a sum of a first, a second and a third amount of differences, wherein the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected, the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected, and the third amount of differences is an amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.
  • the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected
  • the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected
  • the third amount of differences is
  • Example 10 may be any one of examples 1 and 5-9 (optionally having a facial landmark detector of any one of examples 2-4), wherein the model tracker may be configured to fix inter-person shape variations, and consider only intra-person expression variation, when successively updating the 3D model in view of detected landmarks of the face and their locations within successive ones of the plurality of image frames.
  • Example 11 may be any one of examples 1 and 5-9 (optionally having a facial landmark detector of any one of examples 2-4), further comprising a validator coupled with the model fitter and the model tracker to monitor the adjustment of the 3D model to track the face to detect for excessive drifting, and to cause model fitter to regenerate another 3D model of the face when excessive drifting is detected.
  • Example 12 may be example 11, wherein the validator may be configured to detect for excessive drifting through detection of whether inaccuracies of the 3D model successively maintained by the model tracker has grown to exceed a threshold.
  • Example 13 may be example 11 or 12, wherein the model fitter may be configured to re-generate another 3D model of the face from the 3D model of neutral face, in view of detected landmarks of the face and their locations within a current or next one of the plurality of image frames.
  • Example 14 may be a method for constructing a three-dimensional, 3D, face model.
  • the method may comprise detecting, by a computing device, a plurality of landmarks of a face, and their locations, within each of a plurality of image frames; and generating, by the computing device, in view of detected landmarks of the face and their locations within a first one of the plurality of image frames.
  • the method may further comprise adjusting , by the computing device, the 3D model to track the face in subsequent image frames, successively adjusting the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames.
  • Example 15 may be a method for detecting a plurality of landmarks of a face.
  • the method may be used for the detecting operation for example 14.
  • the method may include detecting by a computing device a face within an image frame, and detecting by a computing device one or more landmarks, including an inner eye corner, and their locations within a detected face.
  • the method may further include iteratively refining by the computing device the facial landmark locations of the detected face with increased precision.
  • Example 16 may be the method of example 15, wherein detecting a plurality of landmarks of a face may further comprise reconstructing a face from the facial landmarks, and an image frame, and detecting whether a face and one or more eyes are present within the reconstructed image frame.
  • Example 17 may be the method of example 15 or 16, wherein iteratively refining comprises first refining facial landmark locations to a first precision, and second refining the facial landmark locations from the first precision to a second precision, wherein the second precision is more precise than the first precision.
  • Example 18 may be the method of example 14 (with the operation of detecting facial landmarks optionally employing any one of the methods of examples 15- 17), wherein generating may comprise estimating an initial 3D mesh and head pose, in view of the detected landmarks of the face and their locations within the first image frame.
  • Example 19 may be the method of example 18, wherein generating may further comprise modifying the initial 3D mesh and head pose to reduce a first amount of differences between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected.
  • Example 20 may be the method of example 18 or 19, wherein generating may further comprise modifying the initial 3D mesh and head pose to reduce a second amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected.
  • Example 21 may be the method of any one of examples 18-20, wherein generating may further comprise modifying the initial 3D mesh and head pose to reduce a third amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.
  • Example 22 may be the method of example 18, wherein generating may further comprise modifying the initial 3D mesh and head pose to reduce a sum of a first, a second and a third amount of differences, wherein the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected, the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected, and the third amount of differences is an amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.
  • the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected
  • the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected
  • the third amount of differences is
  • Example 23 may be the method of example 14 (with the operation of detecting facial landmarks optionally employing any one of the methods of examples 15- 17), wherein adjusting may comprise fixing inter-person shape variations, and considering only intra-person expression variation, when successively updating the 3D model in view of detected landmarks of the face and their locations within successive ones of the plurality of image frames.
  • Example 24 may be the method of example 14 or 23 (with the operation of detecting facial landmarks optionally employing any one of the methods of examples 15- 17), further comprising monitoring the adjustment of the 3D model to track the face to detect for excessive drifting, and re-generating another 3D model of the face when excessive drifting is detected.
  • Example 25 may be the method of example 24, wherein monitoring may comprise detecting whether inaccuracies of the 3D model successively maintained by the model tracker has grown to exceed a threshold.
  • Example 26 may be the method of example 24 or 25, wherein re-generating may comprise re-generating another 3D model of the face from the 3D model of neutral face, in view of detected landmarks of the face and their locations within a current or next one of the plurality of image frames.
  • Example 27 may be one or more computer-readable medium having instructions to cause a computing device, in response to execution of the instructions by the computing device, to perform any one of the methods of examples 14 - 26.
  • Example 28 may be an apparatus for constructing a three-dimensional, 3D, face model.
  • the apparatus may comprise means for detecting a plurality of landmarks of a face, and their locations, within each of a plurality of image frames; and means for generating, in view of detected landmarks of the face and their locations within a first one of the plurality of image frames.
  • the apparatus may further comprise means for adjusting the 3D model to track the face in subsequent image frames, successively adjusting the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames.
  • Example 29 may be an apparatus for detecting a plurality of landmarks of a face.
  • the apparatus may be used for the detecting operation for example 28.
  • the apparatus may include means for detecting a face within an image frame, and means for detecting one or more landmarks, including an inner eye corner, and their locations within a detected face.
  • the apparatus may further include means for iteratively refining the facial landmark locations of the detected face with increased precision.
  • Example 30 may be the apparatus of example 29, wherein means for detecting a plurality of landmarks of a face may further comprise means for reconstructing a face from the facial landmarks, and an image frame, and means for detecting whether a face and one or more eyes are present within the reconstructed image frame.
  • Example 31 may be the apparatus of example 29 or 30, wherein means for iteratively refining comprises first means for facial landmark locations to a first precision, and second means for refining the facial landmark locations from the first precision to a second precision, wherein the second precision is more precise than the first precision.
  • Example 32 may be the apparatus of example 28 (with means for detecting facial landmarks optionally employing any one of the apparatuses of examples 29-31), wherein means for generating may comprise means for estimating an initial 3D mesh and head pose, in view of the detected landmarks of the face and their locations within the first image frame.
  • Example 33 may be the apparatus of example 32, wherein means for generating may further comprise means for modifying the initial 3D mesh and head pose to reduce a first amount of differences between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected.
  • Example 35 may be the apparatus of any one of examples 32-34, wherein means for generating may further comprise means for modifying the initial 3D mesh and head pose to reduce a third amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.
  • Example 36 may be the apparatus of example 32, wherein means for generating may further comprise means for modifying the initial 3D mesh and head pose to reduce a sum of a first, a second and a third amount of differences, wherein the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected, the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected, and the third amount of differences is an amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.
  • the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected
  • the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected
  • Example 37 may be the apparatus of example 36 (with means for detecting facial landmarks optionally any one of examples 29-31), wherein means for adjusting may comprise means for fixing inter-person shape variations, and means for considering only intra-person expression variation, when successively updating the 3D model in view of detected landmarks of the face and their locations within successive ones of the plurality of image frames.
  • Example 38 may be the apparatus of example 28 or 37 (with means for detecting facial landmarks optionally employing means of examples 29-31), further comprising means for monitoring the adjustment of the 3D model to track the face to detect for excessive drifting, and means for re-generating another 3D model of the face when excessive drifting is detected.
  • Example 39 may be the apparatus of example 38, wherein means for monitoring may comprise detecting whether inaccuracies of the 3D model successively maintained by the model tracker has grown to exceed a threshold.
  • Example 40 may be the apparatus of example 38 or 39, wherein means for re-generating may comprise means for re-generating another 3D model of the face from the 3D model of neutral face, in view of detected landmarks of the face and their locations within a current or next one of the plurality of image frames.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne des appareils, des procédés et un support de stockage associés à la reconstruction d'un modèle de visage 3D. Dans des modes de réalisation, un appareil peut comprendre un détecteur de points d'intérêt du visage, un dispositif d'ajustement de modèle et un dispositif de suivi de modèle. Le détecteur de points d'intérêt du visage peut être configuré pour détecter une pluralité de points d'intérêt d'un visage et leurs emplacements dans chaque trame d'une pluralité de trames d'images. Le dispositif d'ajustement de modèle peut être configuré pour générer un modèle 3D du visage à partir d'un modèle 3D d'un visage neutre en tenant compte des points d'intérêt détectés du visage et de leurs emplacements dans une première trame de la pluralité de trames d'images. Le dispositif de suivi de modèle peut être configuré pour gérer le modèle 3D afin de suivre le visage dans les trames d'images suivantes par la mise à jour successive du modèle 3D en tenant compte des points d'intérêt détectés du visage et de leurs emplacements dans chaque trame successive de la pluralité de trames d'images. Dans des modes de réalisation, le détecteur de points d'intérêt du visage peut comprendre un détecteur de visage, un détecteur de points d'intérêt du visage initial, et un ou plusieurs régresseurs linéaires de détection de points d'intérêt du visage. D'autres modes de réalisation peuvent être décrits et/ou revendiqués.
PCT/CN2014/080382 2014-06-20 2014-06-20 Appareil et procédé de reconstruction de modèle de visage 3d WO2015192369A1 (fr)

Priority Applications (5)

Application Number Priority Date Filing Date Title
KR1020167032315A KR101828201B1 (ko) 2014-06-20 2014-06-20 3d 얼굴 모델 재구성 장치 및 방법
CN201480078999.XA CN106462995B (zh) 2014-06-20 2014-06-20 3d面部模型重建装置和方法
PCT/CN2014/080382 WO2015192369A1 (fr) 2014-06-20 2014-06-20 Appareil et procédé de reconstruction de modèle de visage 3d
US14/443,337 US9679412B2 (en) 2014-06-20 2014-06-20 3D face model reconstruction apparatus and method
EP14895424.1A EP3158535A4 (fr) 2014-06-20 2014-06-20 Appareil et procédé de reconstruction de modèle de visage 3d

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2014/080382 WO2015192369A1 (fr) 2014-06-20 2014-06-20 Appareil et procédé de reconstruction de modèle de visage 3d

Publications (1)

Publication Number Publication Date
WO2015192369A1 true WO2015192369A1 (fr) 2015-12-23

Family

ID=54934722

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/080382 WO2015192369A1 (fr) 2014-06-20 2014-06-20 Appareil et procédé de reconstruction de modèle de visage 3d

Country Status (5)

Country Link
US (1) US9679412B2 (fr)
EP (1) EP3158535A4 (fr)
KR (1) KR101828201B1 (fr)
CN (1) CN106462995B (fr)
WO (1) WO2015192369A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160210500A1 (en) * 2015-01-15 2016-07-21 Samsung Electronics Co., Ltd. Method and apparatus for adjusting face pose
WO2019145411A1 (fr) * 2018-01-26 2019-08-01 Iee International Electronics & Engineering S.A. Procédé et système d'estimation de pose de tête
EP3574446A4 (fr) * 2017-01-27 2020-09-30 Banuba Limited Systèmes informatiques et procédés mis en oeuvre par ordinateur, spécialisés dans le traitement des données d'image électronique

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9105014B2 (en) 2009-02-03 2015-08-11 International Business Machines Corporation Interactive avatar in messaging environment
US10155168B2 (en) 2012-05-08 2018-12-18 Snap Inc. System and method for adaptable avatars
US9928405B2 (en) * 2014-01-13 2018-03-27 Carnegie Mellon University System and method for detecting and tracking facial features in images
EP3186787A1 (fr) * 2014-08-29 2017-07-05 Thomson Licensing Procédé et dispositif pour enregistrer une image dans un modèle
KR102349059B1 (ko) * 2014-11-07 2022-01-10 삼성전자주식회사 영상의 관심영역으로부터 랜드마크를 결정하는 장치 및 방법
US20170280130A1 (en) * 2016-03-25 2017-09-28 Microsoft Technology Licensing, Llc 2d video analysis for 3d modeling
US10339365B2 (en) 2016-03-31 2019-07-02 Snap Inc. Automated avatar generation
US10360708B2 (en) * 2016-06-30 2019-07-23 Snap Inc. Avatar based ideogram generation
US10432559B2 (en) 2016-10-24 2019-10-01 Snap Inc. Generating and displaying customized avatars in electronic messages
US10454857B1 (en) 2017-01-23 2019-10-22 Snap Inc. Customized digital avatar accessories
US10635930B2 (en) * 2017-02-24 2020-04-28 Siemens Healthcare Gmbh Patient position control for scanning
CN111010882B (zh) 2017-04-27 2023-11-03 斯纳普公司 在基于地图的社交媒体平台上的位置隐私关联
US11893647B2 (en) 2017-04-27 2024-02-06 Snap Inc. Location-based virtual avatars
US10212541B1 (en) 2017-04-27 2019-02-19 Snap Inc. Selective location-based identity communication
CN107886568B (zh) * 2017-12-09 2020-03-03 东方梦幻文化产业投资有限公司 一种利用3D Avatar重建人脸表情的方法及系统
CN108256505A (zh) * 2018-02-12 2018-07-06 腾讯科技（深圳）有限公司 图像处理方法及装置
US11954905B2 (en) 2019-06-28 2024-04-09 Hewlett-Packard Development Company, L.P. Landmark temporal smoothing
KR102241153B1 (ko) * 2019-07-01 2021-04-19 주식회사 시어스랩 2차원 이미지로부터 3차원 아바타를 생성하는 방법, 장치 및 시스템
CN110569768B (zh) * 2019-08-29 2022-09-02 四川大学 人脸模型的构建方法、人脸识别方法、装置及设备
CN112017230A (zh) * 2020-09-07 2020-12-01 浙江光珀智能科技有限公司 一种三维人脸模型建模方法及系统
CN113160365A (zh) * 2021-02-04 2021-07-23 脸萌有限公司 图像处理方法、装置、设备和计算机可读存储介质
CN113537021A (zh) * 2021-07-08 2021-10-22 壹茹(上海)传媒科技有限公司 一种基于普通视频流的3d人脸模型表情自动生成方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1607551A (zh) * 2003-08-29 2005-04-20 三星电子株式会社 基于图像的超现实主义三维脸部建模的方法和设备
US20060188144A1 (en) * 2004-12-08 2006-08-24 Sony Corporation Method, apparatus, and computer program for processing image
WO2012126135A1 (fr) * 2011-03-21 2012-09-27 Intel Corporation Procédé de maquillage augmenté à modélisation de visage tridimensionnelle et alignement de points de repère
CN103210332A (zh) * 2011-04-18 2013-07-17 松下电器产业株式会社 摄像装置、摄像装置的对焦控制方法及集成电路
CN103593870A (zh) * 2013-11-12 2014-02-19 杭州摩图科技有限公司 一种基于人脸的图像处理装置及其方法

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020067362A1 (en) 1998-11-06 2002-06-06 Agostino Nocera Luciano Pasquale Method and system generating an avatar animation transform using a neutral face image
US7020305B2 (en) * 2000-12-06 2006-03-28 Microsoft Corporation System and method providing improved head motion estimations for animation
SE528068C2 (sv) 2004-08-19 2006-08-22 Jan Erik Solem Med Jsolutions Igenkänning av 3D föremål
WO2011079458A1 (fr) * 2009-12-31 2011-07-07 Nokia Corporation Procédé et dispositif pour un modèle binaire local basé sur la localisation d'une caractéristique faciale
CN103765480B (zh) * 2011-08-09 2017-06-09 英特尔公司 用于参数化3d脸部生成的方法和设备
WO2013020248A1 (fr) 2011-08-09 2013-02-14 Intel Corporation Génération de visages 3d multivue à base d'images
US8923392B2 (en) * 2011-09-09 2014-12-30 Adobe Systems Incorporated Methods and apparatus for face fitting and editing applications
US8933928B2 (en) 2011-11-22 2015-01-13 Seiko Epson Corporation Multiview face content creation
US9111134B1 (en) 2012-05-22 2015-08-18 Image Metrics Limited Building systems for tracking facial features across individuals and groups
US9314692B2 (en) 2012-09-21 2016-04-19 Luxand, Inc. Method of creating avatar from user submitted image
US9280854B2 (en) 2012-12-16 2016-03-08 Right Foot Llc System and method for customizing figurines with a subject's face
US20140185924A1 (en) 2012-12-27 2014-07-03 Microsoft Corporation Face Alignment by Explicit Shape Regression

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1607551A (zh) * 2003-08-29 2005-04-20 三星电子株式会社 基于图像的超现实主义三维脸部建模的方法和设备
US20060188144A1 (en) * 2004-12-08 2006-08-24 Sony Corporation Method, apparatus, and computer program for processing image
WO2012126135A1 (fr) * 2011-03-21 2012-09-27 Intel Corporation Procédé de maquillage augmenté à modélisation de visage tridimensionnelle et alignement de points de repère
CN103210332A (zh) * 2011-04-18 2013-07-17 松下电器产业株式会社 摄像装置、摄像装置的对焦控制方法及集成电路
CN103593870A (zh) * 2013-11-12 2014-02-19 杭州摩图科技有限公司 一种基于人脸的图像处理装置及其方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3158535A4 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160210500A1 (en) * 2015-01-15 2016-07-21 Samsung Electronics Co., Ltd. Method and apparatus for adjusting face pose
US10134177B2 (en) * 2015-01-15 2018-11-20 Samsung Electronics Co., Ltd. Method and apparatus for adjusting face pose
EP3574446A4 (fr) * 2017-01-27 2020-09-30 Banuba Limited Systèmes informatiques et procédés mis en oeuvre par ordinateur, spécialisés dans le traitement des données d'image électronique
WO2019145411A1 (fr) * 2018-01-26 2019-08-01 Iee International Electronics & Engineering S.A. Procédé et système d'estimation de pose de tête
LU100684B1 (en) * 2018-01-26 2019-08-21 Technische Univ Kaiserslautern Method and system for head pose estimation

Also Published As

Publication number Publication date
US20160275721A1 (en) 2016-09-22
CN106462995B (zh) 2020-04-28
US9679412B2 (en) 2017-06-13
KR20160148606A (ko) 2016-12-26
EP3158535A1 (fr) 2017-04-26
CN106462995A (zh) 2017-02-22
EP3158535A4 (fr) 2018-01-10
KR101828201B1 (ko) 2018-02-09

Similar Documents

Publication Publication Date Title
US9679412B2 (en) 3D face model reconstruction apparatus and method
US11557085B2 (en) Neural network processing for multi-object 3D modeling
US20210279894A1 (en) Depth and motion estimations in machine learning environments
US10621422B2 (en) Method and apparatus for generating facial expression and training method for generating facial expression
US9542621B2 (en) Spatial pyramid pooling networks for image processing
CN107251096B (zh) 图像捕获装置和方法
US20160042548A1 (en) Facial expression and/or interaction driven avatar apparatus and method
KR20150138374A (ko) 병렬 검출과 추적, 및/또는 그룹화된 피처 모션 시프트 추적을 이용한 얼굴 인식
US10991122B2 (en) Processing images to localize novel objects
CN108875519B (zh) 对象检测方法、装置和系统及存储介质
US11676361B2 (en) Computer-readable recording medium having stored therein training program, training method, and information processing apparatus
US11741568B2 (en) Systems and methods for low-power, real-time object detection
KR20230107224A (ko) 광학 흐름 추정을 위한 폐색 마스킹 및 지도 학습
JP6298521B2 (ja) プログラム、システム、装置、及び方法
US20160189339A1 (en) Adaptive 3d registration
Luvizon et al. Scene‐Aware 3D Multi‐Human Motion Capture from a Single Camera
US10304258B2 (en) Human feedback in 3D model fitting
US20190096073A1 (en) Histogram and entropy-based texture detection
KR101515845B1 (ko) 동작 인식 방법 및 동작 인식 장치
CN105493101B (zh) 包括在辅助对象定位中使用加速数据的图像帧处理
KR101909326B1 (ko) 얼굴 모션 변화에 따른 삼각 매쉬 모델을 활용하는 사용자 인터페이스 제어 방법 및 시스템
US10671881B2 (en) Image processing system with discriminative control
US20160180143A1 (en) Eye tracking with 3d eye position estimations and psf models
Chenglin et al. Edge-Enabled Real-time Railway Track Segmentation

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 14443337

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14895424

Country of ref document: EP

Kind code of ref document: A1

REEP Request for entry into the european phase

Ref document number: 2014895424

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2014895424

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 20167032315

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE