EP4145342A1 - Lernverfahren und trainingsvorrichtung für adaptives starres vorheriges modell sowie gesichtsverfolgungsverfahren und verfolgungsvorrichtung - Google Patents

Lernverfahren und trainingsvorrichtung für adaptives starres vorheriges modell sowie gesichtsverfolgungsverfahren und verfolgungsvorrichtung Download PDF

Info

Publication number
EP4145342A1
EP4145342A1 EP21796709.0A EP21796709A EP4145342A1 EP 4145342 A1 EP4145342 A1 EP 4145342A1 EP 21796709 A EP21796709 A EP 21796709A EP 4145342 A1 EP4145342 A1 EP 4145342A1
Authority
EP
European Patent Office
Prior art keywords
facial
model
face
parameters
faces
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP21796709.0A
Other languages
English (en)
French (fr)
Other versions
EP4145342A4 (de
Inventor
Wenyu Chen
Gengdai LIU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bigo Technology Pte Ltd
Original Assignee
Bigo Technology Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bigo Technology Pte Ltd filed Critical Bigo Technology Pte Ltd
Publication of EP4145342A1 publication Critical patent/EP4145342A1/de
Publication of EP4145342A4 publication Critical patent/EP4145342A4/de
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration using histogram techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/778—Active pattern-learning, e.g. online learning of image or video features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2021—Shape modification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the present disclosure relates to the field of processing images, and for example, relates to a method and apparatus for training an adaptive rigid prior model, and a method and apparatus for tracking faces.
  • Facial pose estimation is a significant topic in tracking faces in videos. Facial poses and facial expressions need to be tracked in real-time face tracking, and the real-time face tracking includes calculating a rigid transformation of a three-dimension face model, for example, rotation, translation, scaling, and the like. Stable facial pose estimation is applicable to achieving a vision enhancement effect, for example, trying on hats and glasses, adding beards and tattoos, and the like, and is also applicable to driving virtual characters to change expressions.
  • the facial pose is estimated based on a motion of the face in the video.
  • the motion of face in the video is not solely determined by the facial pose.
  • different expressions of the user may cause the motion of the facial pose in the video.
  • an effect of the facial expression should be efficiently reduced in optimizing the facial pose, that is, the rigidity is stabilized.
  • rigidity stabilization of face tracking is achieved using a prior in a real-time rigidity stabilization method.
  • Common priors are achieved by two relatively simpler methods: a heuristic prior for a specific facial region, and a dynamic rigid prior.
  • a heuristic prior for a specific facial region, but the heuristic prior is not applicable to all expressions.
  • the weight is dynamically estimated by estimating displacements of each facial region, and thus, the dynamic rigid prior is applicable to different expressions.
  • a trained model is merely applicable in a face at a specific size, and is not adaptive to changes of the facial sizes in the video.
  • the current rigid prior model fails to adapt all expressions and different facial sizes and estimate weights of a plurality of facial regions, such that the rigidity stabilization of facial pose estimation is not great, and the accuracy of tracking the faces is poor.
  • Embodiments of the present disclosure provide a method and apparatus for training an adaptive rigid prior model, and a method and apparatus for tracking faces, so as to solve the problem that a conventional rigid prior model fails to adapt all expressions and different facial sizes to estimate weights of a plurality of facial regions, such that the rigidity stabilization of facial pose estimation is not great, and the accuracy of tracking the faces is poor.
  • the embodiments of the present disclosure provide a method for training an adaptive rigid prior model.
  • the method includes:
  • the embodiments of the present disclosure provide a method for tracking faces.
  • the method includes:
  • the embodiments of the present disclosure provide an apparatus for training an adaptive rigid prior model.
  • the apparatus includes:
  • the embodiments of the present disclosure provide an apparatus for tracking faces.
  • the apparatus includes:
  • the embodiments of the present disclosure provide an electronic device.
  • the electronic device includes:
  • the embodiments of the present disclosure provide a computer-readable storage medium.
  • the computer-readable storage medium stores one or more computer programs, wherein the one or more programs, when loaded and run by a processor, causes the processor to perform the method for training the adaptive rigid prior model and/or the method for tracking the faces according to any one of the embodiments of the present disclosure.
  • FIG. 1 is a flowchart of steps of a method for training an adaptive rigid prior model according to some embodiments of the present disclosure.
  • the embodiments of the present disclosure are applicable to a case of training a rigid prior model to estimate weights of a plurality of facial regions of faces at different facial sizes.
  • the method is applicable to the apparatus for training the adaptive rigid prior model according to the embodiments of the present disclosure, and the apparatus for training the adaptive rigid prior model is implemented by a hardware or a software, and is integrated in the electronic device according to the embodiments of the present disclosure.
  • the method for training the adaptive rigid prior model includes the following steps.
  • model parameters of the adaptive rigid prior model are initialized, wherein the model parameters are functions with facial sizes as independent variables, and the adaptive rigid prior model is configured to output weights of a plurality of facial regions of faces at different facial sizes.
  • the face is divided into a plurality of facial regions.
  • the face evinces an expression
  • displacements of the plurality of facial regions of the faces relative to a neutral face are generated.
  • the neutral face is a face of the user without facial expression, and the neutral face is generated for each user.
  • a motion of each of the plurality of facial regions affects a face rigid transformation, for stable rigid transformation, it is desirable that the facial region with greater displacement has a less weight in the rigid transformation, and the facial region with less displacement has a greater weight in the rigid transformation in facial pose estimation.
  • the adaptive rigid prior model is configured to output the weights of the plurality of facial regions of the faces at different facial sizes, and the model parameters are the functions with the facial sizes as the independent variables.
  • a plurality of frames of facial data of a same face are acquired by face tracking on training video data using the adaptive rigid prior model.
  • the training video data is video data of a user.
  • the face tracking is a process of extracting facial key points of the same user from the training video data, establishing three-dimension faces, rigidly transforming the three-dimension faces, and acquiring optimal rigid transformation parameters and optimal facial expression parameters.
  • the facial key points Q f and the facial expression parameters Q f of the faces in the plurality of frames of training video data are extracted.
  • the facial key points Q f are two-dimensional points of face key sites.
  • the facial key points extracted by a pre-trained facial key point detection model, the facial sizes of the faces are determined based on the facial key points of the faces, and the displacements of the plurality of facial regions of the faces relative to the neutral face of the faces are calculated by multiplying the facial expression parameters Q f by a predetermined blend shape deformer B exp .
  • the predetermined blend shape deformer B exp is a tool for producing facial expression animations, and is capable of achieving a smooth and high precision deformation effect of basic objects by target shape objects.
  • the blend shape deformer B exp is predetermined for a plurality of users, and the blend shape deformer predetermined for a user is acquired in the case that the video data of the user is determined as the training video data.
  • the facial sizes of the faces in the plurality of frames of the video data are acquired, and the displacements of the plurality of facial regions relative to the neutral face are acquired by optimizing rigid transformation parameters and facial expression parameters and determined as the facial data.
  • the model parameters are acquired based on the plurality of frames of facial data.
  • the facial data includes facial sizes and the displacements of the plurality of facial regions of faces relative to the neutral face, the faces at different facial sizes extracted from the training video data are organized to a plurality of face containers based on the facial sizes.
  • an average value of the facial sizes of the faces in the face container is calculated, and in the case that the facial sizes are the average value, data points of the model parameters of the adaptive rigid prior model are acquired by analyzing the displacements of the plurality of facial regions of the faces in the plurality of face containers.
  • the data points of the model parameters corresponding to different average values are interpolated, so as to acquire the model parameters and take the model parameters as new model parameters of the adaptive rigid prior model.
  • the model parameters are functions of the facial sizes, that is, the model parameters are a spline curve. In the spline curve, the facial size is used as an independent variable, and the model parameter is used as a dependent variable.
  • a condition of stopping updating the model parameters is that difference values of the model parameters acquired by two adjacent updates are less than a first predetermined threshold, or the number of updates of the model parameters reaches to a preset number.
  • S 105 is performed in the case that the difference values of the model parameters acquired by two adjacent updates are less than a second predetermined threshold.
  • the process is returned to S102 of acquiring the plurality of frames of facial data of the same face by face tracking on training video data using the adaptive rigid prior model with updated model parameters, and the model parameters are re-updated based on the facial data.
  • S 105 is performed in the case that the number of updates of the model parameters reaches to the preset number.
  • the process is returned to S102 of acquiring the plurality of frames of facial data of the same face by face tracking on training video data using the adaptive rigid prior model with updated model parameters, and the model parameters are re-updated based on the facial data.
  • the adaptive rigid prior model with finally updated model parameters is a trained model, and the trained adaptive rigid prior model is deployed on a client, a server, or the like.
  • the adaptive rigid prior model is capable of assigning different weights to different regions based on real time facial sizes of the faces and the displacements of the plurality of facial regions of the faces.
  • the model parameters of the adaptive rigid prior model are functions with the facial sizes as independent variables, and the adaptive rigid prior model is configured to output the weights of the plurality of facial regions of the faces at different facial sizes.
  • the model parameters are initialized, the plurality of frames of facial data of the same face are acquired by face tracking on the training video data using the adaptive rigid prior model, the model parameters are updated based on the plurality of frames of facial data, the model parameters of the adaptive rigid prior model is stopped updating in response to the condition of stopping updating the model parameters being satisfied, and the final adaptive rigid prior model is acquired.
  • the adaptive rigid prior model is capable of assigning different weights to different regions based on real time facial sizes of the faces and the displacements of the plurality of facial regions of the faces, such that rigidity stabilization of face tracking is improved, and a robust result of face tracking is achieved.
  • FIG. 2A is a flowchart of steps of another method for training an adaptive rigid prior model according to some embodiments of the present disclosure.
  • the embodiments of the present disclosure are refined on the basis of above embodiment.
  • the method for training the adaptive rigid prior model according to the embodiments of the present disclosure includes the following steps.
  • model parameters of the adaptive rigid prior model are initialized, wherein the model parameters are functions with facial sizes as independent variables, and the adaptive rigid prior model is configured to output weights of a plurality of facial regions of faces at different facial sizes.
  • FIG. 2B is a schematic diagram of a division of a plurality of facial regions of a face according to some embodiments of the present disclosure.
  • the face is divided to seven regions, that is, k is a natural number from 1 to 7.
  • Weights of the plurality of face facial regions of the faces at all facial sizes are initialized to be 1 in initializing the model parameters of the adaptive rigid prior model.
  • facial key points and facial expression parameters of faces in a plurality of frames of the training video data are extracted by face tracking on the training video data.
  • a face in each frame of video data f is determined based on following parameters: P f ⁇ f Q f .
  • P f represents a rigid transformation parameter
  • ⁇ f represents a facial expression parameter
  • Q f represents a facial key point.
  • the rigid transformation parameter P f and the facial expression parameter ⁇ f are unknown parameters, and facial key point Q f is a known parameter and is acquired by detecting the facial key point in face tracking.
  • the facial sizes of the faces are determined based on the facial key points.
  • a face frame is determined based on the facial key points, and a length of a diagonal line of the face frame is used as the facial size of the face in each frame of video data f .
  • displacements of the plurality of facial regions of the faces relative to a neutral face of the faces are calculated based on the adaptive rigid prior model, the facial expression parameters, and a predetermined blend shape deformer.
  • three-dimension faces are constructed based on the neutral face, the facial expression parameters, and the predetermined blend shape deformer, and the three-dimension face models is rigidly transformed.
  • a rigid transformation optimization function is constructed based on the weights calculated based on the rigid transformation, the facial key points, and the adaptive rigid prior model.
  • optimal rigid transformation parameters are acquired by solving the rigid transformation optimization function
  • optimal facial expression parameters are acquired based on the optimal rigid transformation parameters
  • the displacements of the plurality of facial regions of the faces are acquired based on the optimal facial expression parameters.
  • B user represents the neutral face of the user without expression, and the face of the user without expression is pre-acquired, that is, the neutral face.
  • B exp represents a blend shape deformer of the user.
  • the facial displacement B exp ⁇ f of the face in the video data f, that is, the three-dimension face is acquired by adding a plurality of expressions to the neutral face of the user.
  • P f ⁇ f argmin ⁇ i ⁇ ⁇ P f B user + B exp ⁇ f i ⁇ Q f i ⁇ 2 .
  • P f ( ⁇ ) represents the rigid transformation on the three-dimension face F f
  • ⁇ ( ⁇ ) represents a two-dimensional projection (an orthogonal projection or a perspective projection) on the three-dimension face.
  • the function (2) is the rigid transformation function
  • w ki represents a weight of the facial region k of the facial key point i of the face in the training video data f
  • j represents an iteration number.
  • the expression optimization function (3) is iteratively solved.
  • the optimal facial expression parameter ⁇ f is acquired in the case that the expression optimization function (3) converges, and the optimal facial expression parameter ⁇ f is the expression parameter of last iteration.
  • the facial displacement B exp ⁇ f of the face in the video data f is calculated, and the displacements of the plurality of facial regions relative to the neutral face are acquired based on the division of the plurality of facial regions shown in FIG. 2B .
  • the faces are organized to a plurality of face containers based on the facial sizes.
  • the following facial data is extracted from each frame of video data f ⁇ F : s f d f k k ⁇ K .
  • s f represents the facial size of the face in the video data f
  • d f k k ⁇ K represents the displacement of the k th facial region of the face in the video data f relative to the neutral face.
  • a histogram analysis is performed on all facial sizes ⁇ s f ⁇ f ⁇ F of the faces, and the faces extracted from the video data f are organized to n face containers C i .
  • a maximum facial size and a minimum facial size are first determined, a size range from the maximum facial size to the minimum facial size is evenly divided into a plurality of size ranges, the plurality of size ranges are used as facial size ranges of the plurality of face containers.
  • Target facial size ranges of the facial sizes of the faces are determined, and the faces are organized to face containers within the target facial size ranges.
  • the faces are organized to n face containers C i based on following formula: s min + i ⁇ 1 n s max ⁇ s min , s min + i n s max ⁇ s min .
  • s min min f ⁇ F ⁇ s f ⁇ , that is, the minimum facial size.
  • s max max f ⁇ F ⁇ s f ⁇ , that is, the maximum facial size.
  • the face container is discarded.
  • all faces F are a union of faces F i in the plurality of face containers C i , and an intersection of faces F i and faces F j in any two face containers C i and C j is null.
  • s i mean f ⁇ Fi ⁇ s f ⁇ , that is, the average values s i of the facial sizes of the faces in the plurality of face containers C i are calculated.
  • S207 includes following sub-steps.
  • maximum displacements of the plurality of facial regions are determined based on facial image data of the faces for the plurality of facial regions of the faces in the plurality of face containers.
  • each face includes a plurality of facial regions.
  • F i is the face in the face containers C i
  • ⁇ i k is the maximum displacement of the facial region k in the face containers C i .
  • the displacements of the plurality of facial regions are organized to a plurality of displacement containers.
  • a histogram analysis is performed on the displacements of the facial region, and the displacements of the facial region are organized to m displacement containers.
  • S205 of performing the histogram analysis on facial sizes may be made to S205 of performing the histogram analysis on facial sizes.
  • a displacement container containing a greatest number of displacements in the plurality of displacement containers is determined as a maximum displacement container.
  • numbers of the displacements in the plurality of displacement containers are counted, and the displacement container containing a greatest number of displacements is determined as the maximum displacement container.
  • a middle displacement of the plurality of displacements in the maximum displacement container is determined as a data point of a first model parameter.
  • the plurality of displacements d f k in the maximum displacement container are ranked in descending order or in ascending order, the displacement ⁇ i k in the middle is determined as the data point of the first model parameter with the face size being the average value, one data point s i ⁇ i k i ⁇ I of the first model parameter is acquired.
  • I ⁇ 1,2,..., n ⁇ , and n is the number of the face containers.
  • a data point of a second model parameter and a data point of a third model parameter are calculated based on the maximum displacements of the plurality of facial regions, the middle displacement, a predetermined maximum weight, a predetermined minimum weight, and the data point of the first model parameter.
  • the data points of the model parameters of the adaptive rigid prior model include: the data point of the first model parameter, the data point of the second model parameter, and the data point of the third model parameter.
  • a desired weight range is freely set for each facial region, for example, seven facial regions shown in FIG. 2B .
  • the facial region 1 and the facial region 2 are symmetric, and thus the facial region 1 and the facial region 2 are combined to be trained.
  • the facial region 4 is the nose of the face, the displacement of the nose is least, and the nose is most stable in the case that the user evinces an expression, a greater weight is assigned to the facial region 4.
  • the facial region 3 and the facial region 5 include profile points of the face, and is relatively stable, and thus, a greater weight is assigned to the facial region 3 and the facial region 5.
  • the facial region 1 and the facial region 2 are eye portions
  • the facial region 6 and the facial region 7 are mouth portions
  • the displacements of the eye portions and the mouth portions are greater in the case that the user evinces an expression, which may cause instable rigid transformation.
  • a less weight is assigned to the facial region 1, the facial region 2, the facial region 6, and the facial region 7.
  • the weight ranges of the plurality of facial regions are set as the following table: Region k 1 and 2 3 and 5 4 6 7 Maximum weight w max k 1 1.5 2 1 1 1 Minimum weight w min k 0.0001 1 1 0.0001 0.0001
  • ⁇ i k represents the data point of the second model parameter with the facial sizes being the average values
  • ⁇ i k represents the data point of the third model parameter with the facial sizes being the average values
  • w max k represents the maximum weight of the facial region k
  • w min k represents the minimum weight of the facial region k.
  • the model parameters are acquired by interpolating the data points of the model parameters corresponding to different average values, and the model parameters are determined as new model parameters of the adaptive rigid prior model, wherein the model parameters are functions of the facial sizes.
  • the first model parameter, the second model parameter, and the third model parameter are acquired by spline interpolating the data point of the first model parameter, the data point of the second model parameter, and the data point of the third model parameter in the case that the facial sizes are the average values, and the first model parameter, the second model parameter, and the third model parameter are determined as the new model parameters of the adaptive rigid prior model.
  • the first model parameter, the second model parameter, and the third model parameter are the functions of the facial sizes.
  • one spline interpolation is performed on the data points of the plurality of model parameters at different facial sizes, that is, the data points of the plurality of model parameters are connected by straight segments, so as to acquire the functions of the plurality of model parameters and the facial sizes.
  • the data points ⁇ i k , ⁇ i k , and ⁇ i k are determined in the case that the facial sizes are s i for each face container C i , the following data points are determined for each face container C i : s i ⁇ i k i ⁇ I ′ s i ⁇ i k i ⁇ I ′ s i ⁇ i k i ⁇ I .
  • the model parameters ⁇ k ( s ), ⁇ k ( s ), and ⁇ k ( s ) of the adaptive rigid prior model w s k ( ⁇ ) are acquired by interpolating the plurality of data points of the plurality of face containers C i .
  • the model parameters ⁇ k ( s ) , ⁇ k ( s ), and ⁇ k ( s ) are functions of the facial sizes s, that is, the facial sizes s is the independent variable.
  • the model parameters ⁇ k ( s ) , ⁇ k ( s ), and ⁇ k ( s ) of the adaptive rigid prior model w s k ⁇ are determined, and the weights of the plurality of facial regions are determined by the adaptive rigid prior model w s k ⁇ .
  • difference values between model parameters acquired by two updates are calculated, whether the difference values are less than a first predetermined threshold is determined, and whether the condition of stopping updating the model parameters is satisfied is determined. That is, whether the difference values between model parameters acquired by two updates are less than the first predetermined threshold is determined, and S201 is performed in the case that the condition of stopping updating the model parameters is satisfied.
  • the face tracking is further performed on the training video data by the adaptive rigid prior model w s k ⁇ with updated model parameters, and the model parameters are further updated based on the data acquired by face tracking.
  • the number of updates of the model parameters is counted, and whether the number of updates is greater than a second predetermined threshold is determined. In the case that the number of updates is greater than the second predetermined threshold, the condition of stopping updating the model parameters is satisfied, and thus S210 is performed. In the case that the number of updates is less than or equal to the second predetermined threshold, the process returns to S202.
  • updating the model parameters of the adaptive rigid prior model is stopped updating in response to the condition of stopping updating the model parameters being satisfied, and a final adaptive rigid prior model is acquired.
  • the adaptive rigid prior model with finally updated model parameters is a trained model, and the trained adaptive rigid prior model is deployed on a client, a server, or the like.
  • the adaptive rigid prior model is capable of assigning different weights to different regions based on real time facial sizes of the faces and the displacements of the plurality of facial regions of the faces.
  • the flowchart of training the adaptive rigid prior model in the example is as follows: S0, the adaptive rigid prior model w s k ⁇ is initialized to be 1.
  • model parameters of the adaptive rigid prior model are initialized, such that the weights of the plurality of facial regions of the faces at different facial sizes are 1.
  • ⁇ i k ⁇ i k ⁇ ⁇ i k ln w max k w min k
  • ⁇ i k w max k exp ⁇ i k ⁇ i k
  • S10 whether to stop training is determined, S11 is performed in response to a result of stopping training, and S12 is performed in response to a result of not stopping training.
  • the face tracking is performed using updated w s k ⁇ .
  • the facial key points and the facial expression parameters of the faces in the plurality of frames of the training video data are extracted by face tracking on the training video data, the facial sizes of the faces are determined based on the facial key points, and the displacements of the plurality of facial regions of the faces relative to the neutral face of the faces are calculated using the adaptive rigid prior model, the facial expression parameters, and the predetermined blend shape deformer.
  • the faces are organized to a plurality of face containers based on the facial sizes, and the average values of the facial sizes of the faces are calculated.
  • the data points of the model parameters of the adaptive rigid prior model are acquired by analyzing the displacements of the plurality of facial regions of the faces in the plurality of face containers.
  • the data points of the model parameters corresponding to different average values are interpolated, so as to acquire the model parameters and determine the model parameters as new model parameters of the adaptive rigid prior model. Updating of the model parameters is stopped in response to the condition of stopping updating being satisfied. Otherwise, the model parameters are further updated by face tracking on the training video data.
  • the adaptive rigid prior model is capable of assigning different weights to different regions based on real-time facial sizes of the faces and the displacements of the plurality of facial regions of the faces, such that rigidity stabilization of face tracking is improved, and the a result of face tracking is achieved.
  • the adaptive rigid prior model is a piecewise function, such that a greater weight caused by less displacement of facial region can be avoided, and the stability of face tracking is ensured.
  • FIG. 3 is a flowchart of steps of a method for tracking faces according to some embodiments of the present disclosure.
  • the embodiments of the present disclosure are applicable to a case of tracking faces in videos.
  • the method is performed by the apparatus for tracking the faces according to the embodiments of the present disclosure, and the apparatus for tracking the faces is implemented by a hardware or a software, and is integrated in the electronic device according to the embodiments of the present disclosure.
  • the method for training the adaptive rigid prior model according to the embodiments of the present disclosure includes the following steps.
  • a plurality of facial key points of a same face, initial facial expression parameters, and initial rigid transformation parameters in a plurality of frames of video data are extracted by face tracking on the video data.
  • the face tracking is performed on the video data to acquire the face in each frame of video data f : P f ⁇ f Q f .
  • P f represents a rigid transformation parameter
  • ⁇ f represents a facial expression parameter
  • Q f represents a facial key point.
  • the rigid transformation parameter P f and the facial expression parameter ⁇ f are unknown parameters, and facial key point Q f is a known parameter and is acquired by detecting the facial key point in face tracking.
  • the face is tracked to acquire optimal result of the facial expression parameter ⁇ f .
  • facial sizes of faces in the plurality of frames of video data are determined based on the plurality of facial key points.
  • a face frame is determined based on the facial key points, and a length of a diagonal line of the face frame is determined as the facial size of the face in each frame of video data f .
  • weights of a plurality of facial regions of the faces are acquired by inputting the facial sizes to a pre-trained adaptive rigid prior model.
  • the adaptive rigid prior model is configured to output the weight of the plurality of facial regions of the faces at different facial sizes, and the adaptive rigid prior model is trained by the method for training the adaptive rigid prior model according to the above embodiments.
  • the adaptive rigid prior model is shown as the formula (1) in the above embodiments.
  • optimal facial expression parameters are acquired based on the weights of the plurality of facial regions, the plurality of facial key points, the initial rigid transformation parameters, and the initial facial expression parameters, and the optimal facial expression parameters are determined as results of face tracking on the plurality of frames of video data.
  • neutral face data of the faces is acquired, three-dimension face models of the faces in the plurality of frames video data are generated based on the neutral face data, the initial facial expression parameters, and a predetermined blend shape deformer, and the three-dimension face models are rigidly transformed.
  • Two-dimensional facial data is acquired by performing, based on the initial rigid transformation parameters, two-dimensional projection on the three-dimension face models, and the two-dimensional facial data includes a plurality of projected two-dimensional key points in one-to-one correspondence to the plurality of facial key points.
  • Optimal rigid transformation parameters are acquired by optimizing the rigid transformation optimization function
  • optimal facial expression parameters are acquired by optimizing, based on the optimal rigid transformation parameters, the facial expression optimization function
  • the optimal facial expression parameters are determined as the results of face tracking on the plurality of frames of video data.
  • the weight w ki is calculated by the adaptive rigid prior model, the rigid transformation optimization function (2) and the facial expression optimization function (3) are constructed, and the optimal facial expression parameter ⁇ f is acquired by sequentially optimizing the function (2) and function (3).
  • the optimal facial expression parameter ⁇ f is acquired by sequentially optimizing the function (2) and function (3).
  • a face vision enhancement special effect is achieved based on the facial expression parameter ⁇ f , for example, adding hats, glasses, beards, tattoos, and the like to the face.
  • the weights of the plurality of facial regions are acquired by the adaptive rigid prior model
  • the optimal results of the facial expression parameters are acquired by constructing rigid transformation based on the weights
  • the optimal results of the facial expression parameters are determined as the results of face tracking.
  • the adaptive rigid prior model is capable of assigning different weights to different regions based on real time facial sizes of the faces and the displacements of the plurality of facial regions of the faces, such that rigidity stabilization of face tracking is improved, and a robust result of face tracking is achieved.
  • FIG. 4 is a block diagram of an apparatus for training an adaptive rigid prior model according to a fourth embodiment of the present disclosure. As shown in FIG. 4 , the apparatus for training the adaptive rigid prior model includes:
  • the apparatus for training the adaptive rigid prior model is capable of performing the method for training the adaptive rigid prior model according to the above embodiments, and includes corresponding function modules and achieves the beneficial effects of performing the method.
  • FIG. 5 is a block diagram of an apparatus for tracking faces according to some embodiments of the present disclosure. As shown in FIG. 5 , the apparatus for tracking faces includes:
  • the apparatus for tracking the faces is capable of performing the method for tracking the faces in the third embodiment, and has corresponding function module and the beneficial effects of performing the method.
  • FIG. 6 shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
  • the electronic device includes: a processor 601, a memory 602, a display screen 603 with a touch function, an input device 604, an output device 605, and a communication device 606.
  • the number of processors 601 in the electronic device is one or more, and FIG. 6 shows the electronic device by taking one processor 601 as an example.
  • the processor 601, the memory 602, the display screen 603, the input device 604, the output device 605, and the communication device 606 in the electronic device are connected via a bus or other ways, and FIG. 6 shows the electronic device by taking the bus as an example.
  • the electronic device is configured to perform the method for training the adaptive rigid prior model and/or the method for tracking the faces according to any one of the embodiments of the present disclosure.
  • a computer-readable storage medium is provided in the embodiments of the present disclosure.
  • One or more instructions in the storage medium when loaded and executed by a processor of an electronic device, cause the electronic device to perform the method for training the adaptive rigid prior model and/or the method for tracking the faces according to the above embodiments of the present disclosure.
  • the terms “one embodiment,” “some embodiments,” “an example,” “some examples” and the like indicates that the features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure.
  • the illustrative description of above terms does not necessarily refer to the same embodiment or example.
  • the described features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Medical Informatics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Graphics (AREA)
  • Architecture (AREA)
  • Computer Hardware Design (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)
EP21796709.0A 2020-04-29 2021-04-15 Lernverfahren und trainingsvorrichtung für adaptives starres vorheriges modell sowie gesichtsverfolgungsverfahren und verfolgungsvorrichtung Pending EP4145342A4 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010355935.7A CN111507304B (zh) 2020-04-29 2020-04-29 自适应刚性先验模型训练方法、人脸跟踪方法及相关装置
PCT/CN2021/087576 WO2021218650A1 (zh) 2020-04-29 2021-04-15 自适应刚性先验模型训练方法、人脸跟踪方法、训练装置、及跟踪装置

Publications (2)

Publication Number Publication Date
EP4145342A1 true EP4145342A1 (de) 2023-03-08
EP4145342A4 EP4145342A4 (de) 2023-11-01

Family

ID=71874880

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21796709.0A Pending EP4145342A4 (de) 2020-04-29 2021-04-15 Lernverfahren und trainingsvorrichtung für adaptives starres vorheriges modell sowie gesichtsverfolgungsverfahren und verfolgungsvorrichtung

Country Status (4)

Country Link
US (1) US12400339B2 (de)
EP (1) EP4145342A4 (de)
CN (1) CN111507304B (de)
WO (1) WO2021218650A1 (de)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111507304B (zh) * 2020-04-29 2023-06-27 广州市百果园信息技术有限公司 自适应刚性先验模型训练方法、人脸跟踪方法及相关装置
CN114925116A (zh) * 2022-06-01 2022-08-19 中国西安卫星测控中心 一种航天器遥测数据预测方法

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070071288A1 (en) 2005-09-29 2007-03-29 Quen-Zong Wu Facial features based human face recognition method
JP4655235B2 (ja) * 2008-03-14 2011-03-23 ソニー株式会社 情報処理装置および方法、並びにプログラム
JP5127686B2 (ja) * 2008-12-11 2013-01-23 キヤノン株式会社 画像処理装置および画像処理方法、ならびに、撮像装置
CN102479388A (zh) * 2010-11-22 2012-05-30 北京盛开互动科技有限公司 基于人脸跟踪和分析的表情互动方法
CN103150561A (zh) 2013-03-19 2013-06-12 华为技术有限公司 人脸识别方法和设备
CN103310204B (zh) * 2013-06-28 2016-08-10 中国科学院自动化研究所 基于增量主成分分析的特征与模型互匹配人脸跟踪方法
KR20150099129A (ko) 2014-02-21 2015-08-31 한국전자통신연구원 국소 특징 기반 적응형 결정 트리를 이용한 얼굴 표정 인식 방법 및 장치
CN107341784A (zh) 2016-04-29 2017-11-10 掌赢信息科技（上海）有限公司 一种表情迁移方法及电子设备
US10949648B1 (en) * 2018-01-23 2021-03-16 Snap Inc. Region-based stabilized face tracking
CN110096925B (zh) * 2018-01-30 2021-05-14 普天信息技术有限公司 人脸表情图像的增强方法、获取方法和装置
CN108647668A (zh) * 2018-05-21 2018-10-12 北京亮亮视野科技有限公司 多尺度轻量级人脸检测模型的构建方法及基于该模型的人脸检测方法
CN109508678B (zh) * 2018-11-16 2021-03-30 广州市百果园信息技术有限公司 人脸检测模型的训练方法、人脸关键点的检测方法和装置
CN110135361A (zh) 2019-05-19 2019-08-16 北京深醒科技有限公司 一种基于红外摄像头下的多姿态人脸识别方法
CN110675385B (zh) * 2019-09-25 2024-05-14 腾讯科技（深圳）有限公司 一种图像处理方法、装置、计算机设备以及存储介质
CN111507304B (zh) 2020-04-29 2023-06-27 广州市百果园信息技术有限公司 自适应刚性先验模型训练方法、人脸跟踪方法及相关装置

Also Published As

Publication number Publication date
WO2021218650A1 (zh) 2021-11-04
CN111507304B (zh) 2023-06-27
EP4145342A4 (de) 2023-11-01
CN111507304A (zh) 2020-08-07
US20230177702A1 (en) 2023-06-08
US12400339B2 (en) 2025-08-26

Similar Documents

Publication Publication Date Title
EP4160271B1 (de) Verfahren und vorrichtung zur verarbeitung von daten für ein autonomes fahrzeug, elektronische vorrichtung und speichermedium
US11301719B2 (en) Semantic segmentation model training methods and apparatuses, electronic devices, and storage media
US11450349B2 (en) Real time video processing for changing proportions of an object in the video
EP3920146A1 (de) Verfahren zum trainieren eines smpl-parametervorhersagemodells, computervorrichtung und speichermedium
EP3953859B1 (de) Verfahren und system zur schätzung von augenbezogenen geometrischen parametern eines benutzers
US10445910B2 (en) Generating apparatus, generating method, and non-transitory computer readable storage medium
EP4145342A1 (de) Lernverfahren und trainingsvorrichtung für adaptives starres vorheriges modell sowie gesichtsverfolgungsverfahren und verfolgungsvorrichtung
US10860755B2 (en) Age modelling method
WO2021208767A1 (zh) 人脸轮廓修正方法、装置、设备及存储介质
EP4276681A1 (de) Gesichtsverfolgungsverfahren und -vorrichtung sowie elektronische vorrichtung und speichermedium
CN112651389A (zh) 非正视虹膜图像的矫正模型训练、矫正、识别方法及装置
CN113657396B (zh) 训练方法、译文展示方法、装置、电子设备以及存储介质
CN112929562B (zh) 视频抖动的处理方法、装置、设备以及存储介质
WO2023029418A1 (zh) 图像超分辨率模型训练方法、装置和计算机可读存储介质
CN110472588B (zh) 锚点框确定方法、装置、计算机设备和存储介质
US20220327757A1 (en) Method and apparatus for generating dynamic video of character, electronic device and storage medium
CN109934058A (zh) 人脸图像处理方法、装置、电子设备、存储介质及程序
Mkhadri et al. A group VISA algorithm for variable selection
CN118967512A (zh) 一种基于变分扩散的图像盲复原方法
CN114327346B (zh) 显示方法、显示装置、电子设备以及存储介质
US11830236B2 (en) Method and device for generating avatar, electronic equipment, medium and product
CN117290627A (zh) 一种服务器端页面组件渲染方法、系统及介质
US9077963B2 (en) Systems and methods for generating a depth map and converting two-dimensional data to stereoscopic data
CN115063299B (zh) 一种图像预处理方法、装置、电子设备以及存储介质
CN113610992A (zh) 骨骼驱动系数确定方法、装置、电子设备及可读存储介质

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20221026

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06K0009000000

Ipc: G06T0005400000

A4 Supplementary search report drawn up and despatched

Effective date: 20230929

RIC1 Information provided on ipc code assigned before grant

Ipc: G06V 40/16 20220101ALI20230925BHEP

Ipc: G06V 10/778 20220101ALI20230925BHEP

Ipc: G06V 10/62 20220101ALI20230925BHEP

Ipc: G06V 10/50 20220101ALI20230925BHEP

Ipc: G06T 7/20 20170101ALI20230925BHEP

Ipc: G06T 7/70 20170101ALI20230925BHEP

Ipc: G06T 5/40 20060101AFI20230925BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20250722