CN111652918A - Method, apparatus, medium, and electronic device for determining 3D face model - Google Patents

Method, apparatus, medium, and electronic device for determining 3D face model Download PDF

Info

Publication number
CN111652918A
CN111652918A CN202010498525.8A CN202010498525A CN111652918A CN 111652918 A CN111652918 A CN 111652918A CN 202010498525 A CN202010498525 A CN 202010498525A CN 111652918 A CN111652918 A CN 111652918A
Authority
CN
China
Prior art keywords
coordinates
face
key point
model
registered
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010498525.8A
Other languages
Chinese (zh)
Other versions
CN111652918B (en
Inventor
张家馨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Horizon Robotics Science and Technology Co Ltd
Original Assignee
Shenzhen Horizon Robotics Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Horizon Robotics Science and Technology Co Ltd filed Critical Shenzhen Horizon Robotics Science and Technology Co Ltd
Priority to CN202010498525.8A priority Critical patent/CN111652918B/en
Publication of CN111652918A publication Critical patent/CN111652918A/en
Application granted granted Critical
Publication of CN111652918B publication Critical patent/CN111652918B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

Disclosed are a method, apparatus, medium, and electronic device for determining a 3D face model, wherein the method comprises: acquiring 3D coordinates of key points of the face of a detected object in a plurality of images to obtain a plurality of groups of 3D coordinates; wherein the 3D coordinates of the facial keypoints comprise: 3D coordinates of the key points of the face under a 3D coordinate system of the camera device; determining rotation and translation matrixes between the key point model of the face to be registered of the detected object and the multiple groups of 3D coordinates respectively according to the posterior probabilities corresponding to the multiple groups of 3D coordinates respectively; and if the rotation and translation matrix meets a preset convergence condition, taking the face key point model to be registered as the 3D face model of the detected object. The technical scheme provided by the disclosure is beneficial to conveniently and accurately establishing the 3D face model of the detected object.

Description

Method, apparatus, medium, and electronic device for determining 3D face model
Technical Field
The present disclosure relates to computer vision technology, and more particularly, to a method of determining a 3D face model, an apparatus for determining a 3D face model, a storage medium, and an electronic device.
Background
In some computer vision applications, it is often desirable to build a 3D face model for a target object. For example, in a DMS (Driver Monitor System), a 3D face model needs to be created for the Driver in advance, and then the face orientation of the Driver and the like can be determined using the 3D face model.
The accuracy of the established 3D face model may have an impact on subsequent processing, for example, the accuracy of the subsequently determined face orientation of the driver. How to quickly and accurately establish a 3D face model of a target object is a technical problem of great concern.
Disclosure of Invention
The present disclosure is proposed to solve the above technical problems. Embodiments of the present disclosure provide a method, an apparatus, a storage medium, and an electronic device for determining a 3D face model.
According to an aspect of an embodiment of the present disclosure, there is provided a method of determining a 3D face model, the method including: acquiring 3D coordinates of key points of the face of a detected object in a plurality of images to obtain a plurality of groups of 3D coordinates; wherein the 3D coordinates of the facial keypoints comprise: 3D coordinates of the key points of the face under a 3D coordinate system of the camera device; determining a rotation and translation matrix between the key point model of the face to be registered of the detected object and the plurality of groups of 3D coordinates according to the posterior probability corresponding to the plurality of groups of 3D coordinates respectively; and if the rotation and translation matrix meets a preset convergence condition, taking the face key point model to be registered as the 3D face model of the detected object.
According to still another aspect of the embodiments of the present disclosure, there is provided an apparatus for determining a 3D face model, including: the key point coordinate acquisition module is used for acquiring 3D coordinates of key points of the face of the detected object in the images to obtain a plurality of groups of 3D coordinates; wherein the 3D coordinates of the facial keypoints comprise: 3D coordinates of the key points of the face under a 3D coordinate system of the camera device; the acquisition and rotation translation matrix module is used for determining rotation translation matrixes between the key point model of the face to be registered of the detected object and the multiple groups of 3D coordinates according to the posterior probabilities corresponding to the multiple groups of 3D coordinates acquired by the key point coordinate acquisition module; and the model registration module is used for taking the face key point model to be registered as the 3D face model of the detected object if the rotation and translation matrix obtained by the rotation and translation matrix obtaining module meets a preset convergence condition.
According to still another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for implementing the above method.
According to still another aspect of an embodiment of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; the processor is used for reading the executable instructions from the memory and executing the instructions to realize the method.
Based on the method and the device for determining the 3D face model provided by the embodiment of the disclosure, in the process of carrying out registration processing on the to-be-registered face key point model of the detected object, the 3D coordinates of the face key points of the detected object are used, so that the data volume used in the registration processing process is small, the calculated amount is small, and the registration efficiency of the 3D face model is improved. Therefore, the technical scheme provided by the disclosure is beneficial to conveniently and accurately establishing the 3D face model for the detected object.
The technical solution of the present disclosure is further described in detail by the accompanying drawings and examples.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent by describing in more detail embodiments of the present disclosure with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure, and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. In the drawings, like reference numbers generally represent like parts or steps.
FIG. 1 is a schematic diagram of a scenario in which the present disclosure is applicable;
FIG. 2 is a flow diagram of one embodiment of a method of determining a 3D face model of the present disclosure;
FIG. 3 is a flow diagram of one embodiment of updating a posterior probability of the present disclosure;
FIG. 4 is a flow diagram of another embodiment of a method of determining a 3D face model of the present disclosure;
FIG. 5 is a schematic structural diagram of an embodiment of the apparatus for determining a 3D face model according to the present disclosure;
fig. 6 is a block diagram of an electronic device provided in an exemplary embodiment of the present application.
Detailed Description
Example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of the embodiments of the present disclosure and not all embodiments of the present disclosure, with the understanding that the present disclosure is not limited to the example embodiments described herein.
It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
It will be understood by those of skill in the art that the terms "first," "second," and the like in the embodiments of the present disclosure are used merely to distinguish one element from another, and are not intended to imply any particular technical meaning, nor is the necessary logical order between them.
It is also understood that in embodiments of the present disclosure, "a plurality" may refer to two or more and "at least one" may refer to one, two or more.
It is also to be understood that any reference to any component, data, or structure in the embodiments of the disclosure, may be generally understood as one or more, unless explicitly defined otherwise or stated otherwise.
In addition, the term "and/or" in the present disclosure is only one kind of association relationship describing the associated object, and means that there may be three kinds of relationships, such as a and/or B, and may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the former and latter associated objects are in an "or" relationship.
It should also be understood that the description of the various embodiments of the present disclosure emphasizes the differences between the various embodiments, and the same or similar parts may be referred to each other, so that the descriptions thereof are omitted for brevity.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Embodiments of the present disclosure may be implemented in electronic devices such as terminal devices, computer systems, servers, etc., which are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with an electronic device, such as a terminal device, computer system, or server, include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set top boxes, programmable consumer electronics, network pcs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.
Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks may be performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.
Summary of the disclosure
In the process of implementing the present disclosure, the inventors found that, in the process of establishing a 3D face model for a target object, a depth camera may be used to shoot the target object to obtain a 3D image of the face of the target object, that is, to obtain 3D point cloud data of the face of the target object, so that the 3D face model of the target object may be obtained by using the 3D point cloud data obtained by shooting. Considering that a group of 3D point cloud data often has phenomena such as noise and point cloud data loss, it can be considered that a plurality of groups of 3D point cloud data are obtained by shooting, and registration processing is performed on the plurality of groups of 3D point cloud data to reduce the influence of the phenomena such as noise and point cloud data loss on the 3D face model of the target object, so that the 3D face model of the target object obtained after registration processing is more accurate. However, when the registration processing is performed on a plurality of sets of 3D point cloud data, since the 3D point cloud data is dense point cloud data and the 3D face model is also a dense point 3D face model, the amount of calculation of the registration processing is large, and the time for building the 3D face model is long.
Brief description of the drawings
The technique of determining a 3D face model provided by the present disclosure may be applied in a DMS. One example is shown in figure 1.
In fig. 1, a vehicle 100 is provided with an image pickup device 101 and a DMS. The camera 101 may be a depth camera and the viewing range of the camera 101 may include the area where the driver is located.
In the case that the DMS determines that it is necessary to establish a 3D face model for the driver (e.g., when the driver drives the vehicle 100 for the first time), the DMS prompts the driver to face the camera 101 in at least two postures, and controls the camera 101 to capture the drivers in different postures, so as to obtain multiple sets of 3D point cloud data of the face of the driver. The DMS can establish a 3D face model for the driver in real time by using a plurality of groups of 3D point cloud data obtained by shooting, and store the 3D face model.
While the driver drives the vehicle 100, the DMS may detect the driving state of the driver in real time based on the 2D image captured by the camera 101 and the stored 3D face model, and when it is detected that the driver is in a dangerous driving state such as a distracted driving state (e.g., the driver's face is pointing out of the window or in a front-end driving direction for a long time) or a fatigue driving state (e.g., the driver's face is pointing in a ground direction for a long time), the driver may be prompted in a voice or light manner, so as to ensure the driving safety of the vehicle 100.
Exemplary method
FIG. 2 is a flow chart of one embodiment of a method of determining a 3D face model of the present disclosure. The method shown in fig. 2 comprises: s200, S201, and S202. The following describes each step.
S200, acquiring 3D coordinates of key points of the face of the detected object in the multiple images to obtain multiple groups of 3D coordinates.
The multiple images in the present disclosure are typically captured by the same imaging device. For example, the plurality of images may be a plurality of images captured by an imaging device fixed in a vehicle, with respect to a driver. The face of the same detected object in the multiple images can be determined through face recognition and the like. The 3D coordinates of the face key points in the present disclosure may refer to 3D coordinates of the face key points in a 3D coordinate system of the image pickup device. That is, the 3D coordinates of the face key points in the present disclosure may be 3D coordinates based on a 3D coordinate system of the image pickup device. The Z axis of the 3D coordinate system of the image pickup device is usually the optical axis direction of the image pickup device, and if the optical axis direction of the image pickup device is assumed to be the front, the X axis of the 3D coordinate system of the image pickup device may be the horizontal rightward direction, and the Y axis of the 3D coordinate system of the image pickup device may be the vertical downward direction.
The facial key points of the detected object in the present disclosure may refer to facial key points that do not substantially change in position with changes in facial expression. That is, the facial key points of the detected object in the present disclosure may refer to facial key points whose key point positions change little when the facial expression of the detected object changes (including blinking). For example, facial keypoints in the present disclosure may include, but are not limited to: the key points of the left eye inner canthus, the left eye outer canthus, the right eye inner canthus, the right eye outer canthus, the nose tip, the eyebrow, the lip crest, etc.
The 3D coordinates of all facial keypoints of a detected object in one image in the present disclosure form a set of 3D coordinates.
A set of 3D coordinates in this disclosure may be represented as a matrix with dimensions (3, K), where K represents the number of all facial keypoints of the detected object in one image. Assuming that the width of the matrix is 3 and the height of the matrix is K, one row in the matrix represents the 3D coordinates of one face key point, and the matrix includes the number of rows as the number of face key points. Accordingly, the sets of 3D coordinates obtained from the plurality of images by the present disclosure may be represented in the form of (J, 3, K), where J represents the number of the plurality of images, that is, there are J matrices with dimension (3, K) in the present disclosure, and each matrix corresponds to one image.
S201, according to the posterior probabilities corresponding to the multiple groups of 3D coordinates, rotation and translation matrixes between the key point model of the face to be registered of the detected object and the multiple groups of 3D coordinates are determined.
The posterior probability corresponding to a set of 3D coordinates in the present disclosure may refer to the possibility that each point in the set of 3D coordinates is respectively taken as a face key point in the 3D face model of the detected object. The posterior probability corresponding to each group of 3D coordinates in this step may be: respectively setting initial posterior probability for each group of 3D coordinates in an initialization mode; the following steps can be also included: the obtained posterior probabilities are updated each time in an iterative manner.
The to-be-registered face key point model of the detected object in the present disclosure may refer to a 3D face model that currently needs to be adjusted. The face keypoint model to be registered in the present disclosure may include a plurality of face keypoints, i.e., the plurality of face keypoints form a stereoscopic face model. The to-be-registered face key point model of the detected object in this step may be: the initial model matrix set for the detected object in the initialization mode may also be: and updating the obtained face key point model to be registered each time in an iterative mode.
The rotation and translation matrix in the present disclosure may indicate the position changes of each face key point in the to-be-registered face key point model of the detected object and the corresponding face key point in each set of 3D coordinates, X, Y in the 3D coordinate system of the camera device and the Z axis.
S202, if the rotation translation matrix meets a preset convergence condition, taking the face key point model to be registered as a 3D face model of the detected object.
The preset convergence condition in the present disclosure may be a condition set for a change between the rotational-translational matrix obtained this time and the rotational-translational matrix obtained in the previous iteration process. For example, the preset convergence condition may be: the obtained rotation and translation matrix is basically the same as the rotation and translation matrix obtained in the previous iteration process. That is, the rotation angle represented by the rotation matrix obtained this time is substantially the same as the rotation angle represented by the rotation matrix obtained in the previous iteration process, and the translation position represented by the translation matrix obtained this time is substantially the same as the translation position represented by the translation matrix obtained in the previous iteration process. The substantial identity may be considered as a difference between the two being less than a predetermined difference.
Since the 3D face model in the present disclosure includes a plurality of facial key points (e.g., a plurality of facial key points that do not change in position with changes in facial expression), the 3D coordinates of each facial key point can be represented as (x)k,yk,zk) Thus, the 3D face model in this disclosure may take the form of a matrix having dimensions of 3 × K, where K is the number of face keypoints and 3 is the dimension of the 3D coordinates of the face keypoints.
According to the method and the device, the 3D coordinates of the face key points of the detected object are used in the registration process of the to-be-registered face key point model of the detected object, so that the data volume used in the registration process is small, the calculated amount is small, and the registration efficiency of the 3D face model is improved. Therefore, the technical scheme provided by the disclosure is beneficial to conveniently and accurately establishing the 3D face model for the detected object.
In an alternative example, if the rotational-translational matrix obtained in S201 does not satisfy the preset convergence condition, the present disclosure may update the currently obtained sets of 3D coordinates according to the currently obtained rotational-translational matrix (e.g., perform a position change process on the currently obtained sets of 3D coordinates using the rotational-translational matrix, etc.) for use in the next iteration process. In addition, the face key point model to be registered and the posterior probability can be updated according to the updated groups of 3D coordinates, so that the rotation and translation matrixes between the face key point model to be registered of the detected object and the groups of 3D coordinates are determined again by utilizing the updated face key point model to be registered and the posterior probability of the iteration, and whether the rotation and translation matrixes obtained in the iteration process meet the preset convergence condition is determined again.
According to the method, under the condition that the rotation translation matrix does not meet the preset convergence condition, the multiple groups of 3D coordinates, the face key point model to be registered, the posterior probability and the rotation translation matrix are continuously updated in an iterative mode, so that the rotation translation matrix can be gradually converged in the iterative process, and the registration process of the face key point model to be registered is conveniently and rapidly realized.
In an optional example, the to-be-registered face keypoint model used by the method of the present disclosure for the first time is usually an initialized face keypoint model, that is, the present disclosure may perform initialization processing on the face keypoint model before obtaining the rotation and translation matrix by using the to-be-registered face keypoint model for the first time, so as to obtain the to-be-registered face keypoint model.
Optionally, after obtaining multiple sets of 3D coordinates of the detected object from multiple (e.g., J being an integer greater than 1) images, the present disclosure may perform initialization processing on the face key point model by using the currently obtained multiple sets (e.g., J sets) of 3D coordinates, so as to obtain the face key point model to be registered. Under the condition that J groups of 3D coordinates are expressed in a (J, 3, K) form, for the K-th facial key point, J K-th facial key points exist in the present disclosure, the present disclosure may perform mean calculation on X coordinate values of the J K-th facial key points, perform mean calculation on Y coordinate values of the J K-th facial key points, perform mean calculation on Z coordinate values of the J K-th facial key points, and use an X coordinate mean of the K-th facial key point, a Y coordinate mean of the K-th facial key point, and a Z coordinate mean of the K-th facial key point obtained by the mean calculation as X coordinate values, Y coordinate values, and Z coordinate values of the K-th facial key point in the facial key point model. And after the 3D coordinates of K face key points in the face key point model are all subjected to assignment, obtaining the initialized face key point model to be registered. In addition, the face key point model can be initialized by one group of 3D coordinates in the plurality of groups of 3D coordinates, and the face key point model can be initialized by the X-coordinate mean value, the Y-coordinate mean value and the Z-coordinate mean value of the partial group of 3D coordinates. And will not be described one by one here.
Optionally, the present disclosure may initialize the face keypoint model according to a preset constant value, so as to obtain the face keypoint model to be registered. For example, the present disclosure may set all coordinate values in the face keypoint model to the same preset constant value. For another example, the present disclosure may set all coordinate values in the face keypoint model to different preset constant values. That is, the preset constant values of the coordinate values of different face key points in the face key point model of the present disclosure may be the same or different; and the preset constant value of the X-axis coordinate value, the preset constant value of the Y-axis coordinate value and the preset constant value of the Z-axis coordinate value of the same face key point can be the same or different. The preset constant value may be set empirically.
According to the method and the device, the face key point model is initialized by utilizing the multiple groups of 3D coordinates, so that the initial value of each coordinate in the face key point model to be registered is set more reasonably, and the rotation and translation matrix is converged as soon as possible.
In an alternative example, the present disclosure may obtain rotation and translation matrices between the to-be-registered face keypoint model of the detected object and the sets of 3D coordinates respectively by using an SVD (Singular Value Decomposition) algorithm. For example, for any group (for example, the jth group) of 3D coordinates in one iteration process, the optimal rotation matrix R and translation matrix T between the to-be-registered face key point model of the detected object and the group of 3D coordinates in the current iteration process may be calculated by using the SVD algorithm according to the posterior probability corresponding to the group of 3D coordinates, so that the present disclosure may obtain a plurality of (for example, J) rotation matrices and a plurality of (for example, J) translation matrices. The specific process of obtaining the rotational-translational matrix by using the SVD algorithm is not described in detail herein. The method and the device are beneficial to quickly and accurately obtaining the rotation and translation matrix by utilizing the SVD algorithm.
In an alternative example, the process of the present disclosure for updating the sets of 3D coordinates using the rotational-translation matrix may be: for a 3D coordinate (e.g., (j, 3, k) of any face key point in any group of 3D coordinates obtained in the current iteration process, that is, a 3D coordinate of a k-th face key point in a jth group of 3D coordinates), the present disclosure may perform rotation transformation and translation transformation processing on the 3D coordinate of the face key point (e.g., a 3D coordinate of a k-th face key point in the jth group of 3D coordinates) by using a rotation and translation matrix corresponding to the group of 3D coordinates (e.g., the jth group of 3D coordinates), to obtain a rotation and translation transformed 3D coordinate (e.g., a 3D coordinate of the k-th face key point after rotation and translation transformation), and the present disclosure may use the rotation and translation transformed 3D coordinate as an updated 3D coordinate of the face key point (e.g., the k-th face key point). After the 3D coordinates of all face key points (e.g., K face key points) in the set of 3D coordinates are subjected to the rotational translation transformation processing by using the rotational translation matrix corresponding to the set of 3D coordinates (e.g., the jth set of 3D coordinates), the updating process of the set of 3D coordinates (e.g., the jth set of 3D coordinates) is ended. According to the method, after the 3D coordinates of all face key points (such as K face key points) in the corresponding group of 3D coordinates are subjected to rotational translation transformation processing by using the rotational translation matrix corresponding to each group of 3D coordinates, the multiple groups of 3D coordinates in the iteration process are updated. Wherein J can range from 1 to J, and K can range from 1 to K. According to the method, the multiple groups of 3D coordinates are respectively updated by utilizing the rotation and translation matrixes respectively corresponding to the groups of 3D coordinates in each iteration process, so that the difference between the groups of 3D coordinates and the face key point model is gradually reduced, and the registration efficiency of the face key point model is improved.
In an optional example, the present disclosure may calculate all 3D coordinates or partial 3D coordinates of the same face keypoint in the currently obtained multiple sets of 3D coordinates, and update the face keypoint model to be registered using the calculation result. One example of the present disclosure that updates the to-be-registered face keypoint model using multiple sets of 3D coordinates may be: for any face key point (e.g., the kth face key point) in the face key point model to be registered, the present disclosure may perform mean calculation on X-axis coordinate values, Y-axis coordinate values, and Z-axis coordinate values of all 3D coordinates of the face key point (e.g., the kth face key point) in a plurality of sets of 3D coordinates (e.g., J sets of 3D coordinates) obtained in the current iteration process, and update the X-axis coordinate values, Y-axis coordinate values, and Z-axis coordinate values of the 3D coordinates of the face key point (e.g., the kth face key point) in the face key point model to be registered using a result of the mean calculation.
Optionally, the present disclosure may obtain the X-axis coordinate values, the Y-axis coordinate values, and the Z-axis coordinate values of all facial key points in the facial key point model to be registered by using the following formula (1).
Figure BDA0002523869060000091
Figure BDA0002523869060000092
Figure BDA0002523869060000101
In the above-mentioned formula (1),Xkrepresenting the X-axis coordinate value of the kth face key point in the face key point model to be registered; xk_jThe X-axis coordinate value of the kth face key point in the jth group of 3D coordinates in the iteration process is represented; y isiThe Y-axis coordinate value of the kth face key point in the face key point model to be registered is represented; y isk_jA Y-axis coordinate value representing a kth face key point in a jth group of 3D coordinates in the current iteration process; ziThe Z-axis coordinate value of the kth face key point in the face key point model to be registered is represented; zk_jA Z-axis coordinate value representing a kth face key point in a jth group of 3D coordinates in the iteration process; j denotes the number of sets of the sets of 3D coordinates, i.e., the number of the plurality of images.
According to the face key point model updating method and device, in each iteration process, the face key point model is updated by means of the mean value calculation result of all X-axis coordinate values, the mean value calculation result of Y-axis coordinate values and the mean value calculation result of Z-axis coordinate values of the corresponding face key points in each set of currently obtained 3D coordinates, the face key point model is enabled to be gradually attached to the face of the detected object, and therefore the accuracy of the face key point model is improved.
In one optional example, the present disclosure may update the posterior probability using the currently obtained sets of 3D coordinates, the currently updated model of the facial keypoints to be registered, and a preset allowable error constant. One example of updating a posterior probability according to the present disclosure may be a flow as shown in fig. 3. The method shown in fig. 3 comprises: s300 and S301.
S300, determining a first matrix, namely the current first matrix, according to the multiple groups of currently obtained 3D coordinates and the currently updated key point model of the face to be registered.
Optionally, the process shown in fig. 3 may be applied in each iteration of the present disclosure, and in each iteration, the currently obtained sets of 3D coordinates in S300 may be: and updating the plurality of groups of 3D coordinates in the last iteration process. Similarly, in an iterative process, the current updated face keypoint model to be registered in the present disclosure may be: and updating the face key point model to be registered in the last iteration process.
Optionally, the current first matrix in the present disclosure is used to characterize a position error variation range of the current face keypoint model to be registered, and the current first matrix may be a matrix with a dimension of (1, K), that is, the current first matrix may be a vector including K elements, where K is the number of face keypoints. One element in the current first matrix corresponds to one face key point in the current face key point model to be registered. That is, one element in the current first matrix is used to characterize the position error variation range of one face keypoint in the current face keypoint model to be registered. The current first matrix may also be referred to as a current variance matrix.
Optionally, in the present disclosure, a specific process of obtaining the current first matrix by using the currently obtained multiple sets of 3D coordinates and the currently updated key point model of the face to be registered may be:
for a kth face key point in a to-be-registered face key point model, calculating a variance between a 3D coordinate of the kth face key point in currently-obtained multiple sets (e.g., J sets) of 3D coordinates and a 3D coordinate of the kth face key point in the currently-updated to-be-registered face key point model, and obtaining variances (e.g., J variances) corresponding to the face key points in each set of 3D coordinates.
Optionally, under the condition that the currently obtained first matrix is denoted as σ, the currently obtained multiple groups of 3D coordinates are denoted as V, and the currently obtained face key point model to be registered is denoted as μ, the method may calculate and obtain a value of an element corresponding to a kth face key point in the current first matrix by using the following formula (2):
Figure BDA0002523869060000111
in the above formula (2), σkRepresenting the maximum value of the position error change of the k-th face key point in the current first matrix sigma; mu.sk_jRepresenting the vector length of the kth face key point in the current updated face key point model to be registered; vk_jRepresenting the vector length of the kth face key point in the jth group of currently obtained 3D coordinates; j denotes the number of sets of the currently obtained sets of 3D coordinates.
The method and the device have the advantages that the values of the corresponding elements in the current first matrix are set by utilizing the variance mean value obtained through calculation, a reasonable position error change range is favorably set for each face key point in the current face key point model to be registered, and therefore the posterior probability is favorably converged gradually when the posterior probability is updated.
S301, updating the posterior probability according to the multiple groups of currently obtained 3D coordinates, the current first matrix and a preset allowable error constant.
Optionally, the present disclosure may update the posterior probability using the following steps a and B:
and step A, determining a position error matrix of the current face key point model to be registered relative to each current 3D coordinate, namely a current position error matrix, according to the multiple groups of currently obtained 3D coordinates and the current face key point model to be registered.
Optionally, the current position error matrix in the present disclosure is used to represent differences between a plurality of sets of currently obtained 3D coordinates and the current face key point model to be registered, respectively. The current position error matrix in the present disclosure may be an X-axis, Y-axis, and Z-axis isotropy based position error matrix. That is, the current position error matrix of the present disclosure may be a matrix of dimensions (J, K). Where J represents the number of sets of 3D coordinates and where K represents the number of facial keypoints.
Optionally, in a case that the current position error matrix is a matrix of J rows and K columns, for a K-th face key point in a J-th group of 3D coordinates that are currently obtained, the present disclosure may calculate a distance (e.g., a euclidean distance) between a 3D coordinate of the K-th face key point in the J-th group of 3D coordinates that are currently obtained and a 3D coordinate of the K-th face key point in the face key point model to be currently registered, and use the distance as a value of an element of the J-th row and the K-th column in the current position error matrix.
Step B, the posterior probability can be updated according to the current position error matrix, the current first matrix and a preset allowable error constant.
Optionally, the present disclosure may calculate the current position error matrix, the current first matrix, and a preset allowable error constant, and update the posterior probability by using the calculation result. The allowable error constant in the present disclosure may be set according to an actually used imaging device, for example, the allowable error constant may be set according to a depth measurement error parameter (e.g., 5mm or the like) of an actually used depth imaging device. For example, in a case where an error of x millimeters (e.g., 5mm) is generally present when a depth imaging device actually used performs a depth measurement, the present disclosure may set the allowable error constant to x millimeters.
According to the method and the device, the allowable error constant is considered in the process of updating the posterior probability, and particularly the current position error matrix and the allowable error constant are considered in the process of updating the posterior probability, so that the convergence efficiency of the posterior probability is improved, and the registration efficiency of the face key point model is improved.
Optionally, one example of the present disclosure for updating the posterior probability may include the following steps a and b:
step a, obtaining a norm of a 3D coordinate of each face key point in the current position error matrix, and obtaining a current position error matrix based on the norm.
Optionally, the present disclosure may obtain an L2 norm of the 3D coordinates of each facial keypoint in the current position error matrix. For example, for the kth face keypoint, since the current position error matrix is a position error matrix based on isotropy of the X axis, the Y axis, and the Z axis, when calculating the L2 norm of the 3D coordinate of the kth face keypoint, the present disclosure takes the value of one element in the current position error matrix corresponding to the kth face keypoint as the position error value of the X axis, the Y axis, and the Z axis at the same time, thereby implementing L2 norm calculation.
And b, updating the posterior probability according to the current position error matrix based on the norm, the first matrix sigma and a preset allowable error constant.
Alternatively, the present disclosure may increase an allowable error constant for each element of the norm-based current position error matrix and update the a posteriori probabilities with the current position error matrix and the first matrix incremented by the allowable error constant. For example, the present disclosure may update the posterior probability β using the following equation (3):
βjk＝exp(-errorjk/(2×σk))/σk 1.5formula (3)
In the above equation (3), βjkRepresenting the probability value of the kth face key point in the posterior probability corresponding to the jth group of 3D coordinates; errorjkRepresenting an element corresponding to a kth face keypoint in a jth set of 3D coordinates in a current position error matrix incremented by an allowable error constant; sigmakRepresenting the elements corresponding to the kth face keypoint in the first matrix.
According to the method, in the process of updating the posterior probability, the allowable error constant is added to the norm-based current position error matrix, so that the influence of the measurement error of the camera device on the update of the posterior probability is favorably avoided, the convergence efficiency of the posterior probability is favorably improved, and the registration efficiency of the face key point model is favorably improved.
FIG. 4 is a flow chart of one embodiment of a method of determining a 3D face model of the present disclosure. The method shown in fig. 4 includes: S400-S411. The following describes each step.
S400, the method for determining the 3D face model of the present disclosure is started.
S401, shooting is carried out by the image shooting device, and a plurality of images are obtained. For example, by photographing with a depth camera, a plurality of 2D images and a plurality of 3D images (i.e., depth maps such as point cloud data and the like) may be obtained.
Alternatively, the present disclosure may obtain J2D images and J3D images.
S402, acquiring 3D coordinates of key points of the face of the detected object in the multiple images, and acquiring multiple groups of 3D coordinates.
Optionally, in the present disclosure, a face recognition technology and a face key point detection technology may be utilized to perform face recognition processing and face key point detection processing on each 2D image, so that 2D coordinates of the face key point of the detected object in each 2D image may be obtained. The 2D coordinates of the face key points of the detected object in each 2D image obtained by the present disclosure generally include: 2D coordinates of facial keypoints that are substantially invariant to changes in facial expression. Thereafter, the present disclosure may obtain 3D coordinates of each face keypoint in each 2D image using the 2D coordinates of the face keypoint, a projection matrix of the camera, and the corresponding 3D image, and the 3D coordinates of all face keypoints in each 2D image form a set of 3D coordinates.
Optionally, J sets of 3D coordinates may be obtained for the J2D images and J3D images, and each set of 3D coordinates may include 3D coordinates of K face keypoints. That is, the number of key points of the face of the detected object obtained from each 2D image is K.
And S403, initializing the face key point model, taking the initialized face key point model as a current face key point model to be registered, and respectively initializing posterior probabilities corresponding to each group of currently obtained 3D coordinates to obtain current posterior probabilities corresponding to each group of 3D coordinates.
Optionally, the facial keypoint model in this disclosure is a 3 × K matrix. The present disclosure may initialize the face keypoint model with a preset constant value. The present disclosure may also initialize the face keypoint model using the currently obtained J sets of 3D coordinates. In one example, for the 1 st facial key point of the K facial key points, the present disclosure may perform mean calculation on the X-coordinate value, the Y-coordinate value, and the Z-coordinate value of the 1 st facial key point of the J groups of 3D coordinates, and initialize the X-coordinate value, the Y-coordinate value, and the Z-coordinate value of the 1 st facial key point in the current to-be-registered facial key point model using the X-coordinate mean, the Y-coordinate mean, and the Z-coordinate mean of the 1 st facial key point obtained by the mean calculation. According to the method, after the X coordinate value, the Y coordinate value and the Z coordinate value of the Kth face key point in the current face key point model to be registered are all initialized, the face key point model is initialized successfully, and the current face key point model to be registered is obtained.
Optionally, in the present disclosure, when the number of groups of the multiple groups of 3D coordinates is J, J current posterior probabilities may be obtained, where one current posterior probability corresponds to one group of 3D coordinates, and one current posterior probability includes K probability values. One probability value in a current posterior probability corresponds to one face key point. All the current posterior probabilities obtained in S403 may be expressed in the form of (J, K), i.e., all the current posterior probabilities may be a matrix of J × K.
Alternatively, the present disclosure may initialize all probability values in the current posterior probability corresponding to each group of 3D coordinates to a constant, for example, initialize all probability values to 1.
S404, determining rotation and translation matrixes between the current to-be-registered face key point model of the detected object and the currently obtained groups of 3D coordinates respectively by utilizing SVD according to the current posterior probability corresponding to the currently obtained groups of 3D coordinates respectively, and obtaining the current rotation and translation matrix.
Optionally, each set of 3D coordinates in the present disclosure corresponds to one current rotational-translational matrix, and different sets of 3D coordinates correspond to different current rotational-translational matrices. That is, the present disclosure may obtain J rotational-translation matrices.
S405, judging whether the current rotation translation matrix meets a preset convergence condition, if so, going to S410, and if not, going to S406.
Alternatively, the predetermined convergence condition in the present disclosure is generally set for the respective corresponding rotational-translation matrices of all sets of 3D coordinates. For example, the difference between each rotational-translational matrix and the corresponding rotational-translational matrix obtained from the previous iteration should meet a predetermined difference, and so on.
And S406, respectively performing rotation transformation and translation transformation on the current multiple groups of 3D coordinates according to the rotation translation matrixes corresponding to the multiple groups of 3D coordinates, so as to update the current multiple groups of 3D coordinates.
And S407, updating the current face key point model to be registered according to the plurality of groups of currently obtained 3D coordinates.
Optionally, the present disclosure may update the 3D coordinates of the corresponding face key points in the current to-be-registered face key point model by using the 3D coordinate mean of the corresponding face key points in the currently obtained J groups of 3D coordinates. For example, for the 1 st facial key point of the K facial key points, the present disclosure may perform mean calculation on the X-coordinate value, the Y-coordinate value, and the Z-coordinate value of the J1 st facial key points in the J groups of 3D coordinates obtained currently, and update the X-coordinate value, the Y-coordinate value, and the Z-coordinate value of the 1 st facial key point in the current to-be-registered facial key point model using the X-coordinate mean, the Y-coordinate mean, and the Z-coordinate mean of the 1 st facial key point obtained by the mean calculation. According to the method, after the X coordinate value, the Y coordinate value and the Z coordinate value of the Kth face key point in the current to-be-registered face key point model are updated, the current to-be-registered face key point model is successfully updated.
S408, updating the first matrix according to the multiple groups of currently obtained 3D coordinates and the current face key point model to be registered.
Alternatively, for the 1 st facial keypoint in the K facial keypoints, the present disclosure may calculate a variance of the 3D coordinate of the 1 st facial keypoint in the currently obtained 1 st set of 3D coordinates and the 3D coordinate of the 1 st facial keypoint in the currently to-be-registered facial keypoint model (e.g., a variance between magnitudes of vectors represented by two 3D coordinate points, etc.), calculate a variance of the 3D coordinate of the 1 st facial keypoint in the currently obtained 2 nd set of 3D coordinates and the 3D coordinate of the 1 st facial keypoint in the currently to-be-registered facial keypoint model, … …, calculate a variance of the 3D coordinate of the 1 st facial keypoint in the currently obtained J set of 3D coordinates and the 3D coordinate of the 1 st facial keypoint in the currently to-be-registered facial keypoint model. By analogy, the method can obtain J variances of the 1 st face key point, the method can calculate the mean value of the J variances, and the mean value is used as the value corresponding to the 1 st face key point in the current first matrix. According to the mode, the method can obtain the mean value of J variances of the 2 nd face key point, the mean value of J variances of the 3 rd face key point, … … and the mean value of J variances of the Kth face key point, and the mean value of J variances of each face key point is used for setting the value corresponding to the corresponding face key point in the current first matrix, so that the first matrix is updated, and the current first matrix is obtained.
And S409, updating the current posterior probability according to the currently obtained multiple groups of 3D coordinates, the first matrix and the allowable error constant, returning to S404, and starting the process of obtaining the rotation and translation matrix and judging the rotation and translation matrix next time. The process of updating the current posterior probability can be referred to the description of the above embodiments, and is not described in detail here.
And S410, taking the current face key point model to be registered as the 3D face model of the detected object.
S411, the method for determining the 3D face model is ended.
Exemplary devices
FIG. 5 is a schematic structural diagram of an embodiment of the apparatus for determining a 3D face model according to the present disclosure. The device of the embodiment can be used for realizing the corresponding method embodiment of the disclosure. The apparatus shown in fig. 5 mainly comprises: a key point coordinates obtaining module 500, a rotational translation matrix obtaining module 501, and a model registration module 502. The apparatus may optionally further comprise: an update module 503 and an initialization module 504.
The key point coordinate acquiring module 500 is configured to acquire 3D coordinates of key points of the face of the detected object in the multiple images, so as to obtain multiple sets of 3D coordinates. The 3D coordinates of the face key points comprise: the 3D coordinates of the face key points in the 3D coordinate system of the camera device.
The module 501 for obtaining a rotational translation matrix is configured to determine rotational translation matrices between the key point model of the face to be registered of the detected object and the multiple sets of 3D coordinates according to the posterior probabilities corresponding to the multiple sets of 3D coordinates obtained by the module 500 for obtaining key point coordinates. For example, the module 501 for obtaining a rotation and translation matrix may determine, according to posterior probabilities corresponding to the multiple groups of 3D coordinates, rotation and translation matrices between the to-be-registered face key point model of the detected object and the multiple groups of 3D coordinates by using a singular value decomposition algorithm.
The model registration module 502 is configured to, if the rotational translation matrix obtained by the rotational translation matrix obtaining module 501 meets a preset convergence condition, take the face key point model to be registered as the 3D face model of the detected object.
The updating module 503 is configured to update the multiple groups of 3D coordinates according to the rotational translation matrix if the rotational translation matrix obtained by the rotational translation matrix obtaining module 501 does not meet the preset convergence condition, and update the face key point model to be registered and the posterior probability according to the updated multiple groups of 3D coordinates. And the updated key point model of the face to be registered and the posterior probability are used for determining a rotation and translation matrix between the key point model of the face to be registered of the detected object and the plurality of groups of 3D coordinates respectively.
Optionally, the updating module 503 may include: and the coordinate updating submodule is used for performing rotation transformation and translation transformation on the 3D coordinates of any face key point in any group of 3D coordinates according to the rotation translation matrix corresponding to the group of 3D coordinates, and obtaining the updated 3D coordinates of the face key point.
Optionally, the updating module 503 may further include: and the model updating submodule is used for carrying out mean value calculation on the 3D coordinates of the face key points in the multiple groups of 3D coordinates aiming at any face key point in the face key point model to be registered, and updating the 3D coordinates of the face key points in the face key point model to be registered by using the result of the mean value calculation.
Optionally, the updating module 503 may further include: and the posterior probability updating submodule is used for determining a first matrix for representing the position error change range of the key point model of the face to be registered according to the plurality of groups of 3D coordinates and the updated key point model of the face to be registered, and updating the posterior probability according to the plurality of groups of 3D coordinates, the first matrix and a preset allowable error constant.
Optionally, for any face key point in the face key point model to be registered, the update posterior probability submodule may calculate a variance between a 3D coordinate of the face key point in the multiple sets of 3D coordinates and the updated 3D coordinate of the face key point in the face key point model to be registered, to obtain a variance corresponding to the face key point in each set of 3D coordinates; the update posterior probability sub-module may use the mean of the variances as the values corresponding to the key points of the face in the first matrix.
Optionally, the update posterior probability submodule may determine a position error matrix of the to-be-registered face key point model relative to each group of 3D coordinates according to the plurality of groups of 3D coordinates and the to-be-registered face key point model; then, the update posterior probability submodule may update the posterior probability according to the position error matrix, the first matrix, and a preset allowable error constant. For example, the update posterior probability submodule may first obtain a norm of a 3D coordinate of each facial key point in the position error matrix, to obtain a norm-based position error matrix; the update posterior probability submodule may then update the posterior probability based on the norm-based position error matrix, the first matrix, and a preset allowable error constant.
The initialization module 504 is configured to initialize the face key point model according to the plurality of sets of 3D coordinates, and obtain a face key point model to be registered. The initialization module 504 may also be configured to initialize the face keypoint model according to a preset constant value, so as to obtain the face keypoint model to be registered.
The operations specifically executed by the modules and the sub-modules included in the modules may be referred to the description of the above method embodiment, and are not described in detail here.
Exemplary electronic device
An electronic device according to an embodiment of the present disclosure is described below with reference to fig. 6. FIG. 6 shows a block diagram of an electronic device in accordance with an embodiment of the disclosure. As shown in fig. 6, the electronic device 61 includes one or more processors 611 and a memory 612.
The processor 611 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 61 to perform desired functions.
The memory 612 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory, for example, may include: random Access Memory (RAM) and/or cache memory (cache), etc. The nonvolatile memory, for example, may include: read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium and executed by processor 611 to implement the methods of determining a 3D face model of the various embodiments of the present disclosure described above and/or other desired functionality. Various contents such as an input signal, a signal component, a noise component, etc. may also be stored in the computer-readable storage medium.
In one example, the electronic device 61 may further include: an input device 613, an output device 614, etc., which are interconnected by a bus system and/or other form of connection mechanism (not shown). The input device 613 may also include, for example, a keyboard, a mouse, and the like. The output device 614 can output various information to the outside. The output devices 614 may include, for example, a display, speakers, printer, and communication network and remote output devices connected thereto, among others.
Of course, for simplicity, only some of the components of the electronic device 61 relevant to the present disclosure are shown in fig. 6, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device 61 may include any other suitable components, depending on the particular application.
Exemplary computer program product and computer-readable storage Medium
In addition to the above-described methods and apparatus, embodiments of the present disclosure may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the method of determining a 3D face model according to various embodiments of the present disclosure described in the "exemplary methods" section of this specification above.
The computer program product may write program code for carrying out operations for embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in a method of determining a 3D face model according to various embodiments of the present disclosure described in the "exemplary methods" section above of this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium may include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure is not intended to be limited to the specific details so described.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts in the embodiments are referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The block diagrams of devices, apparatuses, systems referred to in this disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, and systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," comprising, "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
It is also noted that in the devices, apparatuses, and methods of the present disclosure, each component or step can be decomposed and/or recombined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, and the like, will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims (10)

1. A method of determining a 3D face model, comprising:
acquiring 3D coordinates of key points of the face of a detected object in a plurality of images to obtain a plurality of groups of 3D coordinates; wherein the 3D coordinates of the facial keypoints comprise: 3D coordinates of the key points of the face under a 3D coordinate system of the camera device;
determining a rotation and translation matrix between the key point model of the face to be registered of the detected object and the plurality of groups of 3D coordinates according to the posterior probability corresponding to the plurality of groups of 3D coordinates respectively;
and if the rotation and translation matrix meets a preset convergence condition, taking the face key point model to be registered as the 3D face model of the detected object.
2. The method of claim 1, wherein the method further comprises:
if the rotation and translation matrix does not meet the preset convergence condition, updating the multiple groups of 3D coordinates according to the rotation and translation matrix, and updating the face key point model to be registered and the posterior probability according to the updated multiple groups of 3D coordinates;
and the updated key point model of the face to be registered and the posterior probability are used for determining a rotation and translation matrix between the key point model of the face to be registered of the detected object and the plurality of groups of 3D coordinates respectively.
3. The method according to claim 2, wherein the method further comprises, before the step of determining a rotation-translation matrix between the to-be-registered face keypoint model of the detected object and the sets of 3D coordinates respectively according to the posterior probabilities corresponding to the sets of 3D coordinates respectively:
initializing a face key point model according to the multiple groups of 3D coordinates to obtain a face key point model to be registered; or
And initializing the face key point model according to a preset constant value to obtain the face key point model to be registered.
4. The method according to any one of claims 2 to 3, wherein the determining, according to the posterior probabilities corresponding to the sets of 3D coordinates, the rotation and translation matrices between the to-be-registered face keypoint model of the detected object and the sets of 3D coordinates respectively comprises:
and determining rotation and translation matrixes between the key point model of the face to be registered of the detected object and the multiple groups of 3D coordinates respectively by utilizing a singular value decomposition algorithm according to the posterior probabilities corresponding to the multiple groups of 3D coordinates respectively.
5. The method of claim 4, wherein said updating the plurality of sets of 3D coordinates according to the rotational-translation matrix comprises:
and aiming at the 3D coordinate of any face key point in any group of 3D coordinates, performing rotation transformation and translation transformation on the 3D coordinate of the key point according to the rotation translation matrix corresponding to the group of 3D coordinates, and obtaining the updated 3D coordinate of the face key point.
6. The method according to any one of claims 2 to 5, wherein said updating the facial keypoint model to be registered and the posterior probability according to the updated sets of 3D coordinates comprises:
and performing mean calculation on the 3D coordinates of the face key points in the multiple groups of 3D coordinates aiming at any face key point in the face key point model to be registered, and updating the 3D coordinates of the face key points in the face key point model to be registered by using the result of the mean calculation.
7. The method according to any one of claims 2 to 6, wherein said updating the facial keypoint model to be registered and the posterior probability according to the updated sets of 3D coordinates comprises:
determining a first matrix for representing the position error variation range of the face key point model to be registered according to the plurality of groups of 3D coordinates and the updated face key point model to be registered;
and updating the posterior probability according to the multiple groups of 3D coordinates, the first matrix and a preset allowable error constant.
8. An apparatus for determining a 3D face model, comprising:
the key point coordinate acquisition module is used for acquiring 3D coordinates of key points of the face of the detected object in the images to obtain a plurality of groups of 3D coordinates; wherein the 3D coordinates of the facial keypoints comprise: 3D coordinates of the key points of the face under a 3D coordinate system of the camera device;
the acquisition and rotation translation matrix module is used for determining rotation translation matrixes between the key point model of the face to be registered of the detected object and the multiple groups of 3D coordinates according to the posterior probabilities corresponding to the multiple groups of 3D coordinates acquired by the key point coordinate acquisition module;
and the model registration module is used for taking the face key point model to be registered as the 3D face model of the detected object if the rotation and translation matrix obtained by the rotation and translation matrix obtaining module meets a preset convergence condition.
9. A computer-readable storage medium, the storage medium storing a computer program for performing the method of any of the preceding claims 1-7.
10. An electronic device, the electronic device comprising:
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-7.
CN202010498525.8A 2020-06-04 2020-06-04 Method, device, medium and electronic equipment for determining 3D face model Active CN111652918B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010498525.8A CN111652918B (en) 2020-06-04 2020-06-04 Method, device, medium and electronic equipment for determining 3D face model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010498525.8A CN111652918B (en) 2020-06-04 2020-06-04 Method, device, medium and electronic equipment for determining 3D face model

Publications (2)

Publication Number Publication Date
CN111652918A true CN111652918A (en) 2020-09-11
CN111652918B CN111652918B (en) 2023-08-18

Family

ID=72344070

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010498525.8A Active CN111652918B (en) 2020-06-04 2020-06-04 Method, device, medium and electronic equipment for determining 3D face model

Country Status (1)

Country Link
CN (1) CN111652918B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010250420A (en) * 2009-04-13 2010-11-04 Seiko Epson Corp Image processing apparatus for detecting coordinate position of characteristic part of face
CN106503671A (en) * 2016-11-03 2017-03-15 厦门中控生物识别信息技术有限公司 The method and apparatus for determining human face posture
CN107945219A (en) * 2017-11-23 2018-04-20 翔创科技（北京）有限公司 Face image alignment schemes, computer program, storage medium and electronic equipment
WO2018176958A1 (en) * 2017-03-28 2018-10-04 武汉斗鱼网络科技有限公司 Adaptive mapping method and system depending on movement of key points in image
CN109034095A (en) * 2018-08-10 2018-12-18 杭州登虹科技有限公司 A kind of face alignment detection method, apparatus and storage medium
CN110459301A (en) * 2019-07-29 2019-11-15 清华大学 Brain neuroblastoma surgical navigation method for registering based on thermodynamic chart and facial key point

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010250420A (en) * 2009-04-13 2010-11-04 Seiko Epson Corp Image processing apparatus for detecting coordinate position of characteristic part of face
CN106503671A (en) * 2016-11-03 2017-03-15 厦门中控生物识别信息技术有限公司 The method and apparatus for determining human face posture
WO2018176958A1 (en) * 2017-03-28 2018-10-04 武汉斗鱼网络科技有限公司 Adaptive mapping method and system depending on movement of key points in image
CN107945219A (en) * 2017-11-23 2018-04-20 翔创科技（北京）有限公司 Face image alignment schemes, computer program, storage medium and electronic equipment
CN109034095A (en) * 2018-08-10 2018-12-18 杭州登虹科技有限公司 A kind of face alignment detection method, apparatus and storage medium
CN110459301A (en) * 2019-07-29 2019-11-15 清华大学 Brain neuroblastoma surgical navigation method for registering based on thermodynamic chart and facial key point

Also Published As

Publication number Publication date
CN111652918B (en) 2023-08-18

Similar Documents

Publication Publication Date Title
US20130286161A1 (en) Three-dimensional face recognition for mobile devices
US11816865B2 (en) Extrinsic camera parameter calibration method, extrinsic camera parameter calibration apparatus, and extrinsic camera parameter calibration system
CN108381549B (en) Binocular vision guide robot rapid grabbing method and device and storage medium
CN111612852B (en) Method and apparatus for verifying camera parameters
KR20210074333A (en) Biometric detection method and device, storage medium
JP2005327076A (en) Parameter estimation method, parameter estimation device and collation method
KR102386444B1 (en) Image depth determining method and living body identification method, circuit, device, and medium
CN112597872A (en) Gaze angle estimation method and device, storage medium, and electronic device
US20230020965A1 (en) Method and apparatus for updating object recognition model
CN111307163B (en) Positioning method and positioning device of movable equipment and electronic equipment
CN112446917A (en) Attitude determination method and device
CN111723716B (en) Method, device, system, medium and electronic equipment for determining target object orientation
CN112489114A (en) Image conversion method and device, computer readable storage medium and electronic equipment
CN110348359B (en) Hand gesture tracking method, device and system
CN114882465A (en) Visual perception method and device, storage medium and electronic equipment
CN113689508A (en) Point cloud marking method and device, storage medium and electronic equipment
CN111784842B (en) Three-dimensional reconstruction method, device, equipment and readable storage medium
CN111652918B (en) Method, device, medium and electronic equipment for determining 3D face model
KR20220026423A (en) Method and apparatus for three dimesiontal reconstruction of planes perpendicular to ground
CN112668596A (en) Three-dimensional object recognition method and device and recognition model training method and device
CN110827337B (en) Method and device for determining posture of vehicle-mounted camera and electronic equipment
CN115272487A (en) Camera pose estimation method and device, storage medium and electronic equipment
CN112489111B (en) Camera external parameter calibration method and device and camera external parameter calibration system
CN111337010A (en) Positioning method and positioning device of movable equipment and electronic equipment
CN111985384A (en) Method and device for acquiring 3D coordinates of face key points and 3D face model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant